{"id":2606,"date":"2012-08-27T01:56:53","date_gmt":"2012-08-27T06:56:53","guid":{"rendered":"http:\/\/www.thejuliagroup.com\/blog\/?p=2606"},"modified":"2017-04-26T13:55:00","modified_gmt":"2017-04-26T18:55:00","slug":"before-you-even-think-about-propensity-score-matching","status":"publish","type":"post","link":"https:\/\/www.thejuliagroup.com\/blog\/before-you-even-think-about-propensity-score-matching\/","title":{"rendered":"Before you even THINK about propensity score matching"},"content":{"rendered":"

Propensity score matching has had a huge rise in popularity over the past few years. That isn’t a terrible thing, but in my not so humble opinion, many people are jumping on the bandwagon without thinking through if this is what they really need to do.<\/p>\n

The idea is quite simple – you have two groups which are non-equivalent, say, people who attend a support group to quit being douchebags and people who don’t. At the end of the group term, you want to test for a decline in douchebaggery.<\/p>\n

However, you believe that that people who don’t attend the groups are likely different from those who do in the first place, bigger douchebags, younger, and, it goes without saying, more likely to be male.<\/p>\n

The very, very important key phrase in that sentence is YOU BELIEVE.<\/p>\n

Before you ever do a propensity score matching program you should test that belief and see if your groups really ARE different. If not, you can stop right now. You’d think doing a few ANOVAs, t-tests or cross-tabs in advance would be common sense. Let me tell you something, common sense suffers from false advertising. It’s not common at all.<\/p>\n

Even if there are differences between the groups, it may not matter unless it is related to your dependent variable, in this case, the Unreliable Measure of Douchebaggedness.<\/p>\n

Say, for example, that you find that your subjects in the support group are more likely to eat grapefruits for breakfast, live on even-numbered streets and own a parrot. Even though I’d be a little suspicious of anyone who gets up early enough to eat breakfast, if it turns out that none of those variables are related to how big of \u00a0douchebag you are, there is no point in doing a propensity score match.<\/p>\n

Finally, and perhaps most obvious and most frequently overlooked, if your dependent variable is not measured reliably, no amount of statistical hocus-pocus is going to make anything predict it. (Short explanation – an unreliable measure is one that has a large proportion of error variance. Error variance is, by definition, random. Random error is not going to be related to anything. Imagine that every student just colored in the bubbles in the test at random. Now imagine trying to predict the test scores with any variable. Not happening. I think all students SHOULD color in the test sheets at random. I did once. The school psychologist told me I was mentally retarded. She was wrong.)<\/p>\n

and AFTER you do propensity score matching (or anything else) …<\/p>\n

Even after all of this, sometimes it still doesn’t work. A few years ago, I had a client who had a really logical theory, well-designed study and when we ran the analyses every which way, none of the data supported their hypotheses.<\/p>\n

At the end of it all, the client asked me what else we could do, and I said<\/p>\n

“There isn’t anything else we can do that I would recommend. You know, sometimes the theory is just wrong.”<\/p><\/blockquote>\n

It reminds me of the title of a good presentation I went to at the Joint Statistical Meetings earlier this month,<\/p>\n

“Bayesian statistics are powerful but they’re not magical”<\/p><\/blockquote>\n

I think that could be applied to just about any kind of statistical technique. I wish I had said it first.<\/p>\n

Support my day job AND get smarter. Buy Fish Lake for Mac or Windows. Brush up on math skills and canoe the rapids.<\/a><\/p>\n

\"girl<\/a><\/p>\n

For random advice from me and my lovely children, subscribe to our youtube channel 7GenGames TV<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

Propensity score matching has had a huge rise in popularity over the past few years. That isn’t a terrible thing, but in my not so humble opinion, many people are jumping on the bandwagon without thinking through if this is what they really need to do. The idea is quite simple – you have two…<\/p>\n","protected":false},"author":5,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"_kad_post_classname":"","_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[11],"tags":[],"class_list":["post-2606","post","type-post","status-publish","format-standard","hentry","category-statistics"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.thejuliagroup.com\/blog\/wp-json\/wp\/v2\/posts\/2606","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.thejuliagroup.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.thejuliagroup.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.thejuliagroup.com\/blog\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/www.thejuliagroup.com\/blog\/wp-json\/wp\/v2\/comments?post=2606"}],"version-history":[{"count":3,"href":"https:\/\/www.thejuliagroup.com\/blog\/wp-json\/wp\/v2\/posts\/2606\/revisions"}],"predecessor-version":[{"id":5396,"href":"https:\/\/www.thejuliagroup.com\/blog\/wp-json\/wp\/v2\/posts\/2606\/revisions\/5396"}],"wp:attachment":[{"href":"https:\/\/www.thejuliagroup.com\/blog\/wp-json\/wp\/v2\/media?parent=2606"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.thejuliagroup.com\/blog\/wp-json\/wp\/v2\/categories?post=2606"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.thejuliagroup.com\/blog\/wp-json\/wp\/v2\/tags?post=2606"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}