Significance & Mauchly’s W: I don’t think that word means what you think it means
One of my favorite movie lines ever comes from The Princess Bride,
“You keep using that word. I do not think it means what you think it means.”
Sometimes I want to say that to people who want me to give an explanation for every result that is “significant”.
Perhaps you would like to test for sphericity. There is a nice (and pretty!) definition of sphericity and compound symmetry given on the graphpad.com site . Compound symmetry and sphericity are often discussed as if they are the exact same thing. They are not exactly, but the two are related. In case you were dying to know, compound symmetry means the variances AND the covariances are equal.
Sphericity means that the variances of the differences between all pairs are the same. Let’s say you test people once a year for three years. So, the variance of the differences between say, tests given one year apart and two tests given two years apart would be the same if you met this assumption.
Any self-respecting statistics package will include a test for sphericity when doing a MANOVA. In my case, I am doing a repeated measures MANOVA and I strongly suspected that the test for sphericity would be non-significant having engaged in the highly sophisticated statistical technique of looking at my correlation matrix and noting that the correlations among the variables were almost all the same. This very rarely happens in real life. I also looked at the covariances and these were pretty much identical also.
You can get the covariances in SAS by doing this:
proc corr data = mydata cov ;
var varname1 - varname6 ;
So, how do I get my test of sphericity?
In SAS, I do this:
Proc glm data = mydata ;
class site ;
model mon tues weds thurs = site ;
repeated quiz 4 / printe ;
manova _h_ = all ;
and guess what? My Mauchly’s test of sphericity was significant. What the hell?
One problem, that should have been obvious if I had thought of it, is that I have a huge sample size for this study. Mauchly’s test is sensitive to sample size, that means, among other things, it will be significant with large sample sizes even when you have very minor departures from sphericity.
So, I did what any self-respecting statistician does when confronted with a trivially small meaningless but statistically significant result. I ignored it and went on with my work.
We find that obvious (I hope) when talking about correlations of .03 or mean differences that amount to .4 points on a scale of 0 to 100. Yet, when it is a test with which a person is not very familiar, all of a sudden a significant result is scary.
“OH MY GOD! We have violated the sphericity assumption! Whatever shall we do? Release the flying monkeys.”
because they’re not really sure what an important departure from sphericity looks like or even means. This applies to any case, from t-test (for some people) to differences in nested model chi-square values.
Significant is not a synonym for important. Not ever.
P.S. (For a nice explanation of compound symmetry and sphericity , check out A bluffer’s guide to sphericity , by Andy Field. In fact, you should check out the whole Statistics Hell site. It’s pretty funny. )