Psychologists are gradually coming round to the view that it is a good idea to present interval estimates alongside point estimates of statistics. The most common statistic reported in psychology research is almost certainly the mean (strictly the arithmetic mean). Presenting an interval estimate for the mean of a single sample is usually quite simple. This is usually done as 95% confidence interval about the mean – and most researchers in psychology are able to calculate this by hand or get their statistical software to calculate and graph it for them.


Extending this to more than one mean introduces an additional layer of complexity. This is because the difference between two means is a different quantity, and its CI (although related to those of the individual means) is different in width from the CIs of the individual means. This creates a problem when plotting the CI because a researcher might be interested in the CI for an individual mean, the CI for their difference (or both).

The complexity increases further if the aim is to plot a set of means (e.g., from an ANOVA design). In this case, plotting all the possible differences (as is commonly done) obscures patterns in the individual means (e.g., linear or quadratic trends).  Last, but not least, if the means are not from independent samples, there are further difficulties. This happens in within-subjects or repeated measures designs.

In these designs the variation around each mean is correlated with the variation around the other means. This correlation arises from individual differences. Statistical procedures such as ANOVA can capitalize on these individual differences to produce more sensitive statistical inferences (i.e., to increase statistical power or obtain narrower CIs). This is done by estimating the variation due to individual differences, and removing it from the error variance (the estimate of statistical noise in the data set).

This is a problem for graphical presentation of means because the precision of individual means is influenced by individual differences, whereas the precision of differences between means is not (because the estimate of individual differences is common to repeated samples from the same people and thus can be removed). Further complications arise when the sphericity assumption of repeated measures ANOVA is violated.

Several solutions to these problems have been proposed in the literature. The best known of these in psychology is that of Loftus and Masson (1994). Another well-known solution is that of Goldstein and Healy (1995), extended to correlated samples by Afshartous and Preston (2010).

Despite a large literature on the problems of graphing a set of correlated means, many people avoid the problems altogether by not reporting (or graphing) CIs or report CIs that are misleading in some way. Researchers are often unaware of the problems or find the solutions hard to understand and implement.

I recently reviewed the main approaches in the literature, describe how to obtain suitable intervals for individual means and differences between means and provide R code to calculate and plot the intervals.

The main highlights are that:

i) for inferences about individual means the standard approach works fairly well for between-subject (independent measures) designs, but there is a case to use CIs from a multilevel model for within-subject (repeated measures) designs

ii) an approach proposed by Cousineau (2005) with a correction by Morey (2008) offers advantages over the Loftus and Masson (1994) approach for within-subject ANOVA designs. It simplifies the calculations and does not assume sphericity. The Loftus-Masson approach will however usually be superior when n is small.

iii) if you are interested in differences between means then you should probably plot a version of the Cousineau-Morey (or Loftus-Masson) interval that is adjusted so that overlap of the CIs around two individual means corresponds to overlap of the CI for their difference. This can be done by incorporating a multiplier to the width of the individual CIs. This multiplier is equal to (2^0.5)/2.

iv) if you are interested in both precision of individual means and their differences you can use a two-tiered error bar to display both quantities (Cleveland, 1985).

v) the intervals (and graphical presentation of means) are useful for informal inference about a set of means. For formal inference it is better to set up precise hypotheses and test these via an a priori of contrast. This could be a traditional null hypothesis significance test, but other approaches are available. These include confidence intervals, Bayes factors, likelihood ratios and so forth (Baguley, in press; Dienes, 2008).


The paper is available here, the R code here and the data sets here.

Update: R functions now available for the simpler between-subjects (independent measures) ANOVA
case (at the Serious stats blog).

References

Afshartous D., & Preston R. A. (2010). Confidence intervals for dependent data: equating nonoverlap with statistical significance. Computational Statistics and Data
Analysis. 54, 2296-2305.


Baguley, T. (2012, in press). Serious Stats: A guide to advanced statistics for the behavioral sciences. Basingstoke: Palgrave.


Cleveland. W. S. (1985). Elements Of Graphing Data. New York, NY: Chapman & Hall.

Cousineau, D. (2005). Confidence intervals in within-subject designs: A simpler solution to Loftus and Masson’s method. Tutorials in Quantitative Methods for Psychology, 1, 42-45.


Goldstein, H., & Healy, M. J. R. (1995). The graphical presentation of a collection of means. Journal of the Royal Statistical Society. Series A (Statistics in Society), 158, 175-177.

Loftus, G. R., & Masson, M. E. J. (1994). Using confidence intervals in within-subject designs. Psychonomic Bulletin & Review, 1, 476-490.

Morey, R. D. (2008). Confidence intervals from normalized data: A correction to Cousineau (2005). Tutorials in Quantitative Methods for Psychology, 4, 61-64.

0

Add a comment

I have been thinking to write a paper about MANOVA (and in particular why it should be avoided) for some time, but never got round to it. However, I recently discovered an excellent article by Francis Huang that pretty much sums up most of what I'd cover. In this blog post I'll just run through the main issues and refer you to Francis' paper for a more in-depth critique or the section on MANOVA in Serious Stats (Baguley, 2012).
2

I wrote a brief introduction to logistic regression aimed at psychology students. You can take a look at the pdf here:  

A more comprehensive introduction in terms of the generalised linear model can be found in my book:

Baguley, T. (2012). Serious stats: a guide to advanced statistics for the behavioral sciences. Palgrave Macmillan.

I wrote a short blog (with R Code) on how to calculate corrected CIs for rho and tau using the Fisher z transformation.

I have written a short article on Type II versus Type III SS in ANOVA-like models on my Serious Stats blog:

https://seriousstats.wordpress.com/2020/05/13/type-ii-and-type-iii-sums-of-squares-what-should-i-choose/

I have just published a short blog on the Egon Pearson correction for the chi-square test. This includes links to an R function to run the corrected test (and also provides residual analyses for contingency tables).

The blog is here and the R function here.

Bayesian Data Analysis in the Social Sciences Curriculum

Supported by the ESRC’s Advanced Training Initiative

Venue:           Bowden Room Nottingham Conference Centre

Burton Street, Nottingham, NG1 4BU

Booking information online

Provisional schedule:

Organizers:

Thom Baguley   twitter: @seriousstats

Mark Andrews  twitter: @xmjandrews

The third and (possibly) final round of the workshops of our introductory workshops was overbooked in April, but we have managed to arrange some additional dates in June.

There are still places left on these. More details at: http://www.priorexposure.org.uk/

As with the last round we are planning a free R workshop before hand (reccomended if you need a refresher or have never used R before).

In my Serious Stats blog I have a new post on providing CIs for a difference between independent R square coefficients.

You can find the post there or go direct to the function hosted on RPubs. I have been experimenting with knitr  but can't yet get the html from R Markdown to work with my blogger or wordpress blogs.
1
Links
Blog Archive
Subscribe
Subscribe
About Me
About Me
Loading
Dynamic Views theme. Powered by Blogger. Report Abuse.