Every time I teach an introductory statistics course I'm struck by how difficult it is to run a simple 2 by 2 chi-square test of independence. SPSS is one of the worst culprits (but even in R it almost seems more trouble than it is worth to use a computer). One solution is calculate it by hand (and that's what I do in my introductory classes). This leaves me with a practical problem - how can I quickly and easily calculate solutions to practical exercises, check student work or calculate chi-square statistics for my own work?
I've mostly assumed that my solution is so obvious that it is what most statistics teachers do. I just set up an Excel spreadsheet that calculates the chi-square statistic, degrees of freedom and p value for a 2 by 2 table. Thus all I have to do is type in four numbers.
Over the years I've included a few other features - it displays the intermediate calculations (useful for checking student work), calculates standardized residuals and various common 'effect size' estimates. These are phi, phi-squared and the odds ratio. For instance, I also added the Haldane estimator of the odds ratio (which is the odds ratio estimated after adding 0.5 to each cell). This is a useful estimator when observed counts are zero or very small.
My favourite application of the spreadsheet is when editing or reviewing journal submissions reporting chi-square. Often they will not consider the odds ratio. The spreadsheet means I can check calculations for accuracy (if they look dubious) and I can easily include odds ratios in my review or decision (all in under a minute). The odds ratio is useful because it strips out the base rates when comparing effects from different conditions.
Using the absolute difference in rates for chi-squares with different base rates can be misleading. For instance, the difference in solution rates in an easy problem solving task might be 70% - 50% = 20%. For a very hard problem it might be 20% - 10% = 10%. Comparing the absolute differences in rates (often called the ARR or absolute risk reduction in medical settings) would be misleading. The 10% difference is probably more impressive for the very hard problem (it represents a doubling or halving of solution rates). The odds ratio allows a comparison of the probability of solution relative to the probability of non-solution (i.e., the odds) and thus strips out the base rate impact (by scaling in terms of the probability of non-solution). The odds ratio also has other nice mathematical properties (which I won't go into here).
Not everyone loves odds ratios. Medics often dislike them ... because they strip out base rates! For interpreting and communicating medical risks the base rates are important. It matters whether a disease has a base rate of 1 in 100 or 1 in 10. So stripping out the base rate might be misleading in this context. Even so, medics seem unreasonably biased against odds ratios. If the base rates in your sample are dodgy (i.e., don't reflect the population you are interested in) the odds ratio is probably a safer bet. If you know the odds ratio and the correct base rates you can estimate the true risks by combining the two.
(This is one reason why I think that the quest for the perfect effect size statistic is flawed. Different statistics are required for different jobs. Odds ratios, however, get a (largely undeserved) bad press. Most psychologists will probably be better off using them than other measures for 2 by 2 tables, because we rarely have samples that accurately sample the 'true' base rate. This is because our samples are either unrepresentative or, even more importantly, there may be no 'true' base rate. The problem solving example illustrates this. A problem doesn't really have a 'true' difficulty level that we are trying to generalize to or make decisions about.)
If you want to use the spreadsheet you can download it from here. Alternatively, just set up your own (it is quite a useful exercise for understanding how chi-square, odds ratios and so on work). On the other hand my spreadsheet does a few of the slightly awkward calculations for you (e.g., CIs for the odds ratios). The first sheet has very basic instructions. The second sheet has the 2 by 2 table calculations (and effect sizes). The other sheets provide basic statistics for 2 by 3 and 3 by 3 tables. No effect size metrics are included for the latter cases because I think they are not very meaningful for effects with df > 1 (and I've never needed them for real research).
Note: for best results (to preserve formatting etc.) download the document as an Excel file.
No comments:
Post a Comment