Calculating confidence intervals for Cohen’s d and eta-squared using SPSS, R, and Stata

[Now with update for STATA by my colleague +Chris Snijders]
[Now with update about using the MBESS package for within-subject designs]
[Now with an update on using ESCI]

Confidence intervals are confusing intervals. I have nightmares where my students will ask me what they are, and then I try to define them, and I mumble something nonsensical, and they all point at me and laugh.

Luckily, I have had extensive training in reporting statistics I don’t understand completely when I studied psychology, so when it comes to simply reporting confidence intervals, I’m fine. Although these calculations are really easy to do, for some reason I end up getting a lot of e-mails about them, and it seems people don’t know what to do to calculate confidence intervals for effect sizes. So I thought I’d write one clear explanation, and save myself some time in the future. I’ll explain how to do it in 2 ways, the first using SPSS, the second using R, the third by my colleague Chris Snijders on using Stata, and some brief comments about using ESCI.

CI for eta-squared in SPSS

First, download CI-R2-SPSS.zip from the website of Karl L Wuensch. His website is extremely useful (the man deserves an award for it) especially for the step-by-step guides he has created. The explanations he has written to accompany the files are truly excellent and if this blog post is useful, credit goes to Karl Wuensch.

This example focusses on designs where all factors in your ANOVA are fixed (e.g., manipulated), not random (e.g., measured), in which case you need to go here. All you need to do is open NoncF.sav (which refers to the non-central F-distribution, for an introduction, see the OSC blog), fill in some numbers in SPSS, and run a script. You’ll see an empty row of numbers, except .95 in the conf column (which happens to be a value you probably don’t want to use, see the end of this post).

Let’s say you have the following in your results section: F(1,198) = 5.72. You want to report partial η² and a confidence interval around it. You type in the F-value 5.72 in the first column, and then the degrees of freedom (1 in the second column, 198 in the third), and you change .95 into .90 (see below for the reason). Then, you just open NoncF3.sps, run the script, and you get the output in the remaining columns of your SPSS file:

We are only interested in the last three columns. From the r2 column, we get r² or η², = .028, with the lower (lr2) and upper (ur2) values of the confidence interval to the right that give us 90% CI [.003; .076]. Easy peasy.

CI for eta-squared in R (or R Studio)

I’m still not very good with R. I use it as a free superpowered calculator, which means I rarely use it to analyze my data (for which I use SPSS) but I use it for stuff SPSS cannot do (that easily). To calculate confidence intervals, you need to install the MBESS package (installing R, Rstudio and MBESS might take less time than starting up SPSS, at least on my computer).

To get the confidence interval for the proportion of variance (r², or η², or partial η²) in a fixed factor analysis of variance we need the ci.pvaf function. MBESS has loads more options, but you just need to copy paste and run:

ci.pvaf(F.value=5.72, df.1=1, df.2=198, N=200, conf.level=.90)

This specifies the F-value, degrees of freedom, and the sample size (which is not needed in SPSS), and the confidence level (again .90, and not .95, see below). You’ll get the following output:

Here we see the by now familiar lower limit and upper limit (.003 and .076). Regrettably, MBESS doesn’t give partial η², so you need to request it in SPSS (or you can use my effect size spreadsheet).

UPDATE

I found out that for within designs, the MBESS package returns an error. For example:

Error in ci.pvaf(F.value = 25.73, df.1 = 2, df.2 = 28, N = 18, conf.level = 0.9) : N must be larger than df.1+df.2

This error is correct in between-subjects designs (where the sample size is larger than the degrees of freedom) but this is not true in within-designs (where the sample size is smaller than the degrees of freedom for many of the tests). Thankfully, Ken Kelley (who made the MBESS package) helped me out in an e-mail by pointing out you could just use the R Code within the ci.pvaf function and adapt it. The code below will give you the same (at least to 4 digits after the decimal) values as the Smithson script in SPSS. Just change the F-value, confidence level, and the df.1 and df.2.

CI for Cohen’s d in SPSS

Karl Wuensch adapted the files by Smithson (2001) and created a zip file to compute effect sizes around Cohen’s d which works in almost the same way as the calculation for confidence intervals around eta-squared (except for a dependent t-test, in which case you can read more here or here). Open the file NoncT.sav. You’ll again see an almost empty row where you only need to fill in the t-value and the degrees of freedom. Note that (as explained in Wuensch’s help file) there’s a problem with the SPSS files if you fill in a negative t-value, so fill in a positive t-value, and reverse the signs of the upper and lower CI if needed.

If you have a t-test that yielded t(198) = 2.39, then you fill in 2.39 in the first column, and 198 in the second column. For a one-sample t-test this would be enough, in a two-sample t-test you need to fill in the sample sizes n1 (100 participants) and n2 (100 participants). Open T-D-2samples.sps and run it. In the last three columns, we get Cohen’s d (0.33) and the upper and lower limits, 95% CI [0.06, 0.62].

CI for Cohen’s d in R

In MBESS, you can calculate the 95% confidence interval using:

ci.smd(ncp=2.39, n.1=100, n.2=100, conf.level=0.95)

The ncp (or non-centrality parameter) sounds really scary, but it’s just the t-value (in our example, 2.39). n.1 and n.2 are the samples sizes in both groups. You’ll get the following output:

Yes, that’s really all there is to it. The step-by step guides by Wuensch, and the help files in the MBESS package should be able to help you if you run into some problems.

Why should you report 90% CI for eta-squared?

Again, Karl Wuensch has gone out of his way to explain this in a very clear document, including examples, which you can find here. If you don’t want to read it, you should know that while Cohen’s d can be both positive and negative, r² or η² are squared, and can therefore only be positive. This is related to the fact that F-tests are always one-sided (so no, don’t even think about dividing that p = .08 you get from an F-test by two and reporting p = .04, one-sided). If you calculate 95% CI, you can get situations where the confidence interval includes 0, but the test reveals a statistical difference with a p < .05. For a paper by Steiger (2004) that addresses this topic, click here. This means that a 95% CI around Cohen's d equals a 90% CI around η² for exactly the same test. Furthermore, because eta-squared cannot be smaller than zero, a confidence interval for an effect that is not statistically different from 0 (and thus that would normally 'exclude zero') necessarily has to start at 0. You report such a CI as 90% CI [.00; .XX]  where the XX is the upper limit of the CI. Confidence intervals are confusing intervals. 

@Stata afterburner - a confidence interval around eta-squared

by Chris Snijders

In the fall semester I will be teaching a statistics class with Daniel. I have sleepless nights already about what that will do to my evaluations – who wants to appear old and retarded next to such youthful brilliance? [Note that Chris has little to worry about - he was chosen as the best teacher at Eindhoven University of Technology a few years ago - DL]. As Daniel is SPSS and R, and I am just Stata, this also implies some extra work: trying to translate Daniel’s useful stuff to Stata - our students have had their basic course in Stata and I want them to be able to keep up. Luckily, in the case of (confidence intervals around) eta-squared I have an easy way out: Stata 13 includes several effect size measures by default.

The below example is heavily “inspired” by the Stata manual. It shows how to get eta-squared and its confidence interval in Stata. First, read in your data. I use the standard apple data set from Stata:

use http://www.stata-press.com/data/r13/apple

The details of the data do not matter here. Running an Anova goes like this:

anova weight treatment           (output omitted)

To get eta-squared and its confidence interval, type

estat esize

to get

Effect sizes for linear models

-------------------------------------------------------------------

             Source |   Eta-Squared     df     [95% Conf. Interval]

--------------------+----------------------------------------------

              Model |   .9147383         3     .4496586    .9419436

                    |

          treatment |   .9147383         3     .4496586    .9419436

-------------------------------------------------------------------

If you insist on 90% confidence interval, as does Daniel, type:

estat esize, level(90)

Voila!

Stata does several additional things too, such as calculating bootstrapped confidence intervals if you prefer, or calculate effect sizes directly.

Useful additional info can be found at

http://blog.stata.com/tag/eta-squared/,

http://www.stata.com/stata13/effect-sizes/, or try the YouTube intro at https://www.youtube.com/watch?v=rfn5FY96BMc (using the ugly windowed interface).

ESCI update

For people who prefer to use ESCI software by Cumming, please note that ESCI also has an option to provide 95% CI around Cohen's d, both for independent as for dependent t-tests. However, the option is slightly hidden - you need to scroll to the right, where you can check a box which is placed out of view. I don't know why such an important option is hidden away like that, but I've been getting a lot of e-mails about this, so I've added a screenshot below to point out where you can find it. After clicking the check box, a new section appears on the left that allows you to calculate 95% CI around Cohen's d (see second screenshot below)

Conclusion

I think you should report confidence intervals because calculating them is pretty easy and because they tell you something about the accuracy with which you have measured what you were interested in. Personally, I don’t feel them in the way I feel p-values, but that might change. Psychological Science is pushing confidence intervals as ‘new’ statistics, and I’ve seen some bright young minds being brainwashed to stop reporting p-values altogether and only report confidence intervals. I’m pretty skeptical about the likelihood this will end up being an improvement. Also, I don’t like brainwashing. More seriously, I think we need to stop kidding ourselves that statistical inferences based on a single statistic will ever be enough. Confidence intervals, effect sizes, and p-values (all of which can be calculated from the test statistics and degrees of freedom) present answers to different but related questions. Personally, I think it is desirable to report as many statistics that are relevant to your research question as possible. 

Smithson, M. (2001). Correct confidence intervals for various regression effect sizes and parameters: The importance of noncentral distributions in computing intervals. Educational And Psychological Measurement, 61(4), 605-632. doi:10.1177/00131640121971392