Nov

20

Last post I wrote a little about local norms versus national norms and gave the example of how the best-performing student in the area can still be below grade level.

Today, I want to talk a little about tests. As I mentioned previously, when we conducted the pretest prior to student playing our game, Spirit Lake,  the average student scored 37% on a test of mathematics standards for grades 2-5. These were questions that required them to say, subtract one three-digit number from another or  multiply two one-digit numbers.

Originally, we had written our tests to model the state standardized tests which, at the time, were multiple choice. This ended up presenting quite a problem. Here is a bit of test theory for you. A test score is made up two parts – true score variance and error variance.

True score variance exists when Bob gets an answer right and Fred gets it wrong because Bob really knows more math (and the correct answer) compared to Fred.

Error variance occurs when, for some reason, Bob gets the answer right and Fred gets it wrong even though there really is no difference between the two. That is, the variance between Fred and Bob is an error.  (If you want to be picky about it, you would say it was actually the variance from the mean was an error, but just hush.)

How could this happen? Well, the most likely explanation is that Bob guessed and happened to get lucky. (It could happen for other reasons – Fred really knew the answer but misread the question, etc.)

If very little guessing occurs on  a test, or if guesses have very little chance of being correct, then you don’t have to worry too much.

However, the test we used initially had four multiple-choice items for each question. The odds of guessing correctly were 1 in 4, that is, 25%. Because students turned out to be substantially further below grade level than we had anticipated, they did a LOT of guessing. In fact, for several of the items, the percentage of correct responses was close to the 25% students would get from randomly guessing.

When we computed the internal consistency reliability coefficient (Cronbach alpha) which measures the degree to which items in a test correlate with one another, it was a measly .57. In case you are wondering, no, this is not good. It shows a relatively high degree of error variance. So, we were sad.

SAS CODE FOR COMPUTING ALPHA

PROC CORR DATA = mydataset NOCORR ALPHA ;

VAR item1 – item24 ;

 

The very simple code above will give you coefficient alpha as well as the descriptive statistics for each item. Since we very wisely scored our items 0 = wrong, 1= right a mean of say, .22 would indicate that only 22% of students answered an item correctly.

To find out how we fixed this, read the next post.

To buy our games or donate one to a school, click here. Evaluated and developed based on actual data. How about that? Learn fractions, multiplication , statistics – take your pick!

steam_icon-300x142Fish lake woman

Nov

19

I hate the concept of those books with titles like “something or other for dummies”  or “idiot’s guide to whatever” because of the implication that if you don’t know microbiology or how to create a bonsai tree of take out your own appendix you must be a moron. I once had a student ask me if there was a structural equation modeling for dummies book. I told her that if you are doing structural equation modeling you’re no dummy. I’m assuming you’re no dummy and I felt like doing some posts on standardized testing without the jargon.

I haven’t been blogging about data analysis and programming lately because I have been doing so much of it. One project I completed recently was analysis of data from a multi-year pilot of our game, Spirit Lake. 

Buffalo

Before playing the game, students took a test to assess their mathematics achievement. Initially, we created a test that modeled the state standardized tests administered during the previous year, which were multiple choice. We knew that students in the schools were performing below grade level but how far below surprised both us and the school personnel. A sample of 93 students in grades 4 and 5 took a test that measured math standard for grades 2 through 5. The mean score was 37%. The highest score was 63%.

Think about this for a minute in terms of local and national norms. The student , let’s call him Bob, who received a 63% was the highest among students from two different schools across multiple classes. (These were small, rural schools.) So, Bob would be the ‘smartest’ kid in the area. With a standard deviation of 13%, Bob scored two standard deviations above the mean.

Let’s look at it from a different perspective, though. Bob, a fifth-grader, took a test where three-fourths of the questions were at least a year, if not, two or three, below his current grade level, and barely achieved a passing score. Compared to his local norm, Bob is a frigging genius. Compared to national norms, he’s none too bright. I actually met Bob and he is a very intelligent boy, but when most of his class still doesn’t know their multiplication tables, it’s hard for the teacher to get time to teach Bob decimals, and really, why should she worry, he’s acing every test. Of course, the class tests are a year below what should be his grade level.

One advantage of standardized testing, is that if every student in your school or district is performing below grade level it allows you to recognize the severity of the problem and not think “Oh, Bob is doing great.”

He wouldn’t be the first student I knew who went from a ‘gifted’ program in one community to a remedial program when he moved to a new, more affluent school.

—

Get Fish Lake here (yes, that’s another game) before it is released on Steam next month! Learn fractions, canoe rapids, spear fish. Buy for yourself or donate to a school for under ten bucks!

steam_icon-300x142

Nov

2

When I was in my twenties, nearing the end of my competitive years, Dr. James Wooley dropped by the club to visit. If you aren’t into judo, you probably don’t recognize his name as a two-time Olympian. By the time I was competing on the international scene, he had retired from competition, married and was in private practice in Orange County.

I asked whether he missed competition and he shook his head,

“Oh, lord, no!”

(Did I mention he was from Texas?)

“It was great but now I’m finally finished with school, seeing patients, I have a wife and we’re looking to start a family. It was great but I don’t miss it at all.”

From the wisdom of my twenty-something years, I did not believe him for one second. At the time, winning was the most important thing in my life. I thought about it the second my eyes opened in the morning, as I dropped to the floor and did 50 push-ups and 50 sit-ups to start the day. I dreamed about winning. I thought Jimmy was just putting a good face on being old and depressed.

Fast forward a decade or so, the first time it was the end of April and I had not even realized the national championships were happening until they were over. That used to be part of the calendar of my life – start training in January for the Nationals, win those in April. Take a break. Win whatever was the summer event – U.S. Open, Panamerican Games. Take a break.

I retired from competition, married, had more kids, earned a Ph.D., started businesses. Jimmy was right – I didn’t miss it and life did not suck.

Now the kids are adults. I have to send the absentee ballot for the youngest express mail to Boston so she can vote. I’m on the fourth business. Life is good.

I’m closer to 60 than 50 now and if you had asked me to imagine that when I was in my thirties, I’m sure I would have thought it would be depressing.

I still teach judo but after several surgeries on my knees and one on my hand, I don’t do it nearly as well as I once did. I have wrinkles, grey hair and investors who don’t want to talk to me because we all know that innovative ideas are the monopoly of young people.

Let me tell you some of the things that DON’T suck about being old.
1. I don’t have to worry about whether I will have saved enough for retirement, gotten an education, been reasonably successful in my career, raised children who were decent people. The answer is, “All of the above”. Much of the anxiety I had as a younger person is gone because those questions have been answered.

2. I wear what’s comfortable and I don’t give a damn what anyone thinks. My feet don’t hurt from wearing high heels. I don’t walk around cold because wearing a sweater would cover up my girlish figure. Both of my daughters, when they got married, felt the need to tell me that jeans and a hoodie were not acceptable wedding attire.

3. I know a lot of things – from how to make banana bread from scratch to how to code a game in JavaScript to how to interpret results for ordinal logistic regression. All that ‘fake it until you make it’ bullshit is in the past. I’m not faking it. I really do know what it means when my estimates fail to converge or when the knife doesn’t come out clean. I don’t worry about anyone finding me out because there’s nothing to find.

4. I like my husband and he likes me. Yes, we’re both old and wrinkled and grey. He’s lost 50 pounds in the last year, which shows a pretty damn impressive display of will power. He’s brilliant, a great father and makes a good martini. He can help The Spoiled One with her calculus homework and our junior developers with their C# code. He’s not a jerk (women in tech realize having a brilliant guy who is not a jerk is worth something in a lot of ways).

5. Life is easier. One of the advantages of being around a long time is that people get to know you. When you are young, you need to submit proposals to speak at conferences, submit articles to journals, apply for jobs. As you get older, people ask you to work/ write/ speak for them because they know from your previous work that you probably aren’t going to suck. You don’t have to prove yourself because you already did. (Except in the Opposite World of Silicon Valley where education and experience aren’t valued – but that’s a post for another day.)

Mom and Aunt Sylvia

Mom and Aunt Sylvia

Sometimes, I look at my mom, or older friends of mine, and wonder what it is like to be retired, to not have your calendar filled six months, or even 6 days, in advance. I wonder whether it sucks to have nothing you have to do in the day, to have not only your kids but your grandkids safely launched .

I’m guessing that it’s probably just fine.
I’m not just sitting around getting older. I’m also making games. You can buy them here.

characters traveling on map

Or,

you can download a free app, Making Camp, for your iPad here

Wigwam

Blogroll

WP Themes