I’ve hardly blogged, answered emails or talked to anyone these past few months. We just received funding for two games and I have been working night and day, crisscrossing the country recording voices, demonstrating prototypes, contracting for art and animation, reviewing designs and fixing bugs.
Ah, the bugs. We have arrived at the level of complexity in our games where improving one part almost inevitably breaks some other part. After months of revision, Fish Lake will be released on Steam TODAY (not a Steam member ? You can get it on our website, runs on Mac or Windows).
Is it finished?
Well, as a wise man once told me, games are never finished, they’re just abandoned. There are a thousand enhancements we’re still planning, but we have fixed all of the bugs, tested it a hundred times, passed review, for sale on Amazon .
Last night, I read Michael Raethel’s book “It only hurts when I hit enter”. It’s a humor book about life as a programmer. I can guarantee that if you have spent years as a programmer you will find it funny – and familiar. I had to read aloud to my husband the prank on the guy who always claimed credit for solutions from the consultants. Maybe you wouldn’t find it funny if you hadn’t been there, but we had both been there. Everyone has worked with a Henry.
Reading the book made me realize how LONG I have been at this whole program endeavor. When he mentioned COBOL, I smiled and said to myself, “I remember that”.
The description of the green and white lined computer paper stacked up everywhere was another memory that took me way, way back.
It’s been a good run. I’m inside, dry and warm in my nice place by the beach while it pours down rain. If you have always had a house with heat and a roof that doesn’t leak then you probably don’t appreciate that in the same way. We have plenty to eat and more toys than we need.
Still, there are times when it feels as if I have done all of this before. When I look in a filing cabinet or on my computer for a contract for consulting services, I’ll find a dozen more contracts, going back a dozen years. Looking for a review of a grant, I’ll come across four others, three of them funded, one of them not.
Don’t get me wrong, I like programming and I like research. It’s pretty amazing that I can get paid to sit and type numbers into a computer and even more amazing that those numbers can turn into a game that kids play and it raises their math scores.
And yet … In our nice building by the beach, there is a white-haired man with an office on the 12th floor, overlooking the ocean, filled with very nice furniture. He’s a very successful attorney, I hear, and he is in the office every day unless he’s in court. If he isn’t 90 years old, he’s damn close.
Now, some people say,
“That’s just what I want when I’m 90 years old – to be in full possession of my faculties, still be in demand, productive.”
That is NOT what I want. Well, I want the possession of my faculties part, I mean, I don’t want to be drooling on myself. However, I don’t want to drop dead at my desk after having killed one last bug.
I’m not sure what I do want but I’m sure I DON’T want to keep running like a hamster on a wheel.
An angel investor once asked me what I planned to do once I got 7 Generation Games where I wanted it to be, played by millions of people. I said I might just sell the company and go on to whatever the next chapter in my life might be. He didn’t like that answer. He said they were looking for people who were going to go off and start another company after that. He said they were looking for “serial entrepreneurs”. That’s a whole ‘nother post , like, if you made a billion dollars the first time, you wouldn’t need to be a serial entrepreneur.
Frankly, I thought mine was a perfectly fine answer. The thing about a series is that at some point it ends. Yes, Dennis, even The Simpsons will some day end.
One person, whose picture I have replaced with the mother from our game, Spirit Lake, so she can remain anonymous, said to me:
But there is nothing we can do about it, right?I mean, how can you stop kids from guessing?
This was the wrong question. What we know about the measure could be summarized as this:
- Students in many low-performing schools were even further below grade level than we or the staff in their districts had anticipated. This is known as new and useful knowledge, because it helps to develop appropriate educational technology for these students. (Thanks to USDA Small Business Innovation Research funds for enabling this research.)
- Because students did not know many of the answers, they often guessed at the correct answer.
- Because the questions were multiple choice, usually A-D, the students had a 25% probability of getting the correct answer just by chance, interjecting a significant amount of error when nearly all of the students were just guessing on the more difficult items.
- Three-fourths of the test items were below the fifth-grade level. In other words, if you had only gotten correct the answers three years below your grade level, the average seventh-grader should have scored 75% – generally, a C.
There are actually two ways to address this and we did both of them. The first is to give the test to students who are more likely to know the answers so less guessing occurs. We did this, administering the test to an additional 376 students in low-performing schools in grades four through eight. While the test scores were significantly higher (Mean of 53% as opposed to mean of 37% for the younger students) they were still low. The larger sample had a much higher reliability of 87. Hopefully, you remember from your basic statistics that restriction of range attenuates the correlation. By increasing the range of scores, we increased our reliability.
The second thing we did was remove the probability of guessing correctly by changing almost all of the multiple choice questions into open-ended ones. There were a few where this was not possible, such as which of four graphs shows students liked eggs more than bacon . We administered this test to 140 seventh-graders. The reliability, again was much higher: .86
However, did we really solve the problem? After all, these students also were more likely to know (or at least, think they knew, but that’s another blog) the answer. The mean went up from 37% to 46%.
To see whether the change in item type was effective for lower performing students, we selected out a sub-sample of third and fourth-graders from the second wave of testing. With this sample, we were able to see that reliability did improve substantially from .57 to. 71 . However, when we removed four outliers (students who received a score of 0), reliability dropped back down to .47.
What does this tell us? Depressingly, and this is a subject for a whole bunch of posts, that a test at or near their stated ‘grade level’ is going to have a floor effect for the average student in a low-performing school. That is, most of the students are going to score near the bottom.
It also tells us that curriculum needs to start AT LEAST two or three years below the students’ ostensible grade level so that they can be taught the prerequisite math skills they don’t know. This, too, is the subject for a lot of blog posts.
For schools who use our games, we provide automated scoring and data analysis. If you are one of those schools and you’d like a report generated for your school, just let us know. There is no additional charge.