Bar graph showing percentage correct by item grade level

It never ceases to amaze me that intelligent people will spend huge amounts of time doing a literature review, designing elaborate theories, generating elegant hypotheses, selecting a three-stage stratified random sample, performing multivariate analyses, and their measures on which this brilliant study rests are some questions they made up with their three best friends over Chardonnay during happy hour one Friday night. This is also known as the “panel of experts” method and it has the added benefit that it allows you to deduct the wine on your taxes. (Not actual tax advice. Consult your accountant. Of course, if you are doing your 1040 based on reading this blog, you are probably beyond help.)

We did not go with this approach. Our original idea was to use released items from the state standards test from North Dakota but, unfortunately, that is one of the states that never releases items. What we did was find standards that were the same, verbatim, as other states and then found items from those states that had been released. For example,

” Compute a given percent of a whole number”

and the problem would be

“What is 40% of 250?”

with the same four multiple choice options that had been used on the state test.

As someone pointed out, even if the same test had not been previously, since we pulled only the items that tested exactly what we included in the game, the individual items had been validated. So, we had content validity.

One bit of evidence for construct validity came from the item difficulty levels. Here is one of several charts. This shows what percentage of the fourth-grade students answered each item correctly. The items are broken down by grade level. It is also important to know that the state tests showed the majority of students at this school to be low-performing in mathematics. What we see is that as students go from second-grade level items, all of which the majority of the students answered correctly, to fifth-grade items, the percentage correct declines. We see that for the fifth-grade items, only one of them did the students exceed the 25% that would be answered correct by random guessing (remember, there were four multiple-choice options).

Since the state’s test have shown these students to be performing poorly, we should see that they generally are not at grade level, that is, they do not answer many of the fourth-grade items correctly at a rate exceeding chance. That, as you can see from the chart, is the exact situation.

Of course, we did more than this, beginning with replicating this identical chart with fifth-graders, who showed pretty much the same pattern but, as would be expected, answered a higher proportion correctly at each grade level than did the fourth-graders.

That’s the sort of thing that too many studies take for granted and never test. This isn’t the exciting part of creating a game, the part where you make an attack scene and the kid gets  to shoot flaming arrows. So, what good does this do us? Well, the combination of the different analyses of the measure confirms that the measure we used for students to test whether or not their mathematics achievement increased is, in fact, a valid measure of mathematics achievement.

Also, this method has the advantage of not being required to share any of the wine with our best friend/ expert panel so we get to drink it all ourselves.

You can read more about our games at the 7 Generation Games site here. 




On twitter, where I probably spend too much of my time, Dave Winer suggested that start-ups write about themselves, thus cutting out Tech Crunch as the middle man. Since Dave wrote the original blogging software and the first blog on the Internet (really!), his advice is worth considering. So, this week is all about our progress.

Graph of results

The most important thing to know is – we finished our pilot test and it works.

As you can see from the graph, both intervention groups increased substantially more from the pre-test to the post-test than did the control group.

Here is what we did:

The reservation had exactly two schools. One was our control group which simply got tested on specific math concepts and then tested again eight weeks later. The other group played our game over the eight week period. Not continuously because there was a break for Thanksgiving and a mid-way break we had planned to do updates. So, after about six actual weeks of hands-on, the kids math scores went up.

We did a repeated measures Analysis of Variance and the time by group interaction effect was significant – meaning there was a statistically significant improvement for our group that played the game. You might wonder (if you were a statistician) if there was an interaction with grade. There are a couple of possible reasons why the fourth-grade did better. One is that the whole class used the game, in their classroom (we provided laptops). The fifth-graders were pulled out of their classroom and only five from each class participated. Also, the fourth-grade teachers, on their own initiative, got much more involved and used the supplemental materials we provided them, increased their own time spent teaching math.

Looking at teacher effects and the effect of whole-class versus pull-out instruction are two things we will be doing next.




Stowe, Vt ski slopeI’m working on the latest two proposals for federal funding for our computer games while sitting by the fire in a ski resort in Vermont, which is funny because I have never skied in my life and don’t intend to start now. I’m meeting with two people who we would like to have begin working with us this year. In fact, I think one will start as our new marketing and social media person next week. THEY like to snowboard so here we are meeting in Vermont.

I have a meeting in Boston on Saturday with another group of people before heading home.

In both the proposals I’m writing and the meetings I’ve been having, I’ve been telling people over and over –

“We’re going to make millions on this.”

Sometimes they look at me a little funny –  this isn’t the sort of things people in education normally say, and half the purpose of our games is to be educational. The other half is to be fun games or no one will get to the educational part.

It reminds me a lot of when I was competing in judo and told people I was going to win the world championships. They looked at me funny, then, too, because no American had ever done it.

On the other hand, other people have made millions of dollars writing games. Yes, lots of people have not made much money, but one reason lots of them haven’t is the great majority of apps being cranked out are crap. Someone told me it makes me sound arrogant when I say that and who do I think I am that ours will be better  – which makes me think of judo again, because I heard the exact same thing when I said I was going to win a gold medal. It really is true, though, that there is a HUGE amount in the educational game space – apps, online games and desktop games – that is just plain garbage. The world is full of people hoping to get rich quick, who use some drag and drop program or learn a very little bit of coding and then upload some piece of schlock waiting for the money to roll in. I learned from judo that the old “let’s all pretend we are equal and get along” bit didn’t cut it. I don’t want to be equal. I want to be better.

That’s another thing I learned from judo – hard work pays off and harder work pays off more. Between Christmas and January 5th, I’ll have had four meetings in three states with potential customers, employees and collaborators. While I’m working on proposals, the Rocket Scientist is hitting the list of fixes, revisions, and improvements.

One more judo lesson – sometimes you have to put in a lot of work before you see the pay off. I can’t exactly say we are betting the farm on this – come on, I’m blogging from a ski resort – but we will certainly be taking a cut in income as we put almost all of our hours that could be billable into the games we are writing instead.

The main lesson I learned from competing for 14 years is that if you keep focused on your goal and keep hacking* away, you eventually reach it. A lot of people say they want to win a gold medal ,because it sounds good. Not too many people are willing to put in the work for the long haul and I understand that. They have other goals. They want a more balanced life.

For me, I want to make these games. There are a few reasons. Making a lot of money is not the primary one, but having started out with very little, I can tell you money is nothing to sneeze at – it makes life easier, more convenient, more comfortable. Wanting to do it is about half because I am just plain interested for the technical challenge of making something that really teaches math to kids who aren’t the type who usually learn it very well. The other half is that it can make a difference in people’s lives. If you do well enough in school to graduate high school, go on to college, or even just know enough math skills to stay continuously employed – all of that makes a huge difference compared to dropping out of high school, not graduating from college or not being able to get any kind of job.

Why not just do it and give the game away? That’s really not practical. We need artwork, music , editing, testing, documentation, marketing – people aren’t willing to work for no money and we can only do so much ourselves. That is why so much stuff available is garbage – it takes a lot of time and money to make a good game. If you are talking about one to teach math, on top of time, money, talented artists, excellent developers and everything else that goes into a game, you also need people who understand not just math but how to teach it.

I’m not sure if this counts as a New Year’s resolution or more of a prediction, but it’s going to be a great year for math games.

* The Rocket Scientist told me I shouldn’t use the words “hacking” or “programmer” because they are low-level. Consequently, I use them both as much as possible to show that he is not the boss of me and word chooser of the house.

P.S. If you’re going to be at the workshop with JMM next week on Using Games to teach Statistics, say hey.


« go back


WP Themes