We don’t perfect design in our programming until we’re sure we’re going to keep it.
I wrote a post recently about how social media may be overrated for marketing, but that doesn’t mean, paradoxically, that I think social media is over-rated. I think social media is often under-rated for its value as information.
For example, as part of our Kickstarter campaign, I have received a number of questions and suggestions from people, most of them very reflective and thoughtful. Some made us change our design and some made us decide we need to explain our products, company and project better. Some were just amusing.
Probably. However, our games started on the Spirit Lake Nation and half of the people working on the project are from Spirit Lake or Turtle Mountain. Our original intent was to provide an adventure game to teach kids math that would appeal to children on American Indian reservations. It was only after we had developed a prototype with promising results that we started being contacted from parents and teachers off the reservation to find out how they could get the game, too.
2. Is the game installed on your desktop or played over the web?
Both. The game is installed and played on your desktop, which speeds up the graphics much more than streaming video. However, you log in to our server, where your game state and other data – specifically on the math problems you answered and the lessons you took – are stored.
3. Won’t people hate the fact they are paying for a license rather than getting a permanent license? Isn’t requiring internet access a turn off?
Both of these go back to question 1. The schools WANTED a system set up to be as hands off for them as possible so they had minimum work to do. Having an internet connection allows automatic updates. Because multiple students will play the game in the school computer lab, it’s not like your own personal game console where you start the game you left off. A major factor for schools and teachers is having the automated scoring of math problems and the ability to get an average of their class or school, find out if there are problems their students as a group are having. All of this is easier if all of the data go to a central point. It is also backed up automatically. All of the schools have internet, even in very remote reservation schools. There may be some that don’t but we are not working with any of them.
4. How can you have children’s confidential data on your computer? Don’t you need mad security for that?
We don’t have any confidential information. We have a username, a teacher name, age and gender. We send the school a list of usernames and the teacher assigns them. The same with individuals or social service programs. YOU know that Greybear is Samuel Jackson, Jr. but we don’t.
5. Why do you only take payments through Amazon?
That is all Kickstarter is set up to do. We have had people give checks or cash to a friend to back us on Amazon through their account. Yes that isn’t the most convenient way, but there are a lot of other benefits to Kickstarter. They handle the payments, people come specifically to Kickstarter looking for projects to fund.
6. Why is a signed photo with Ronda one of the prizes/ why don’t you offer signed gloves or other prizes?
Kickstarter rewards have to be related to your product. Ronda was involved in helping us design the game and will be further involved modeling some of the poses for fight scenes (not too much fighting, since the game is aimed at children in grades 4-6 right now, and eventually up to grade 8). Read post 3 on our updates page for more information. Since her fight gloves, getting a seminar from Ronda and other UFC-related or MMA-related rewards have nothing to do with the game, under the Kickstarter terms of service, we can’t offer them.
7. Since you are 52% of your target with 8 days left, do you think you will make it?
Yes. According to Kickstarter, 98% of projects that get to 60% end up fully funded. We are almost to that 60%. We have an ad coming out soon on the American Horror channel on FilmOn host Hart Fisher did just out of the goodness of his heart. I just did two radio shows still to be aired plus several interviews with blogs, newspapers. In addition, according to most resources I have read, most campaigns receive a jump in pledges near the end as people who have been meaning to pledge quit putting it off. If you are one of those people, head on over to Kickstarter and pledge now. Get a license for yourself, give one to a school, get a signed photo, a cool poster or even be a character in the game.
8. Now that your game has received more widespread attention are you going to reduce the focus on Native Americans?
@xek tweeted that “Almost everyone who does twitter stats is a fucking moron”.
I was going to say that no one really knows what the hell is going on when it comes to social media, but I believe his estimate adds a greater level of precision.
For example, the tweet reach tool tells me that I reached 17,999 accounts with my tweets from my annmariastat account in the last five days. In contrast, my judo/ personal account reached 30,336 accounts. Does that mean that 18,000 or 30,000 people READ those tweets? I’m pretty certain not.
Ronda’s “reach” is 137,947 even though her number of followers is around 173,000. So I am interpreting this as the number of people you reached OUTSIDE of your followers.
This article says that there are 465 million twitter accounts and that 30% of twitter users have an income over $100,000. Wait, what? Because I know the average income in the US and it is far from $100,000. Maybe richer people have several twitter accounts. I know I have 2. Maybe Bill Gates has 10,000 – but wouldn’t he still just count as one twitter user?
Readwrite.com did have a little more data I could sink my teeth into. They said that 29% of tweets produce a reaction – a reply or retweet. This really surprised me because I had assumed it was far fewer.
I got interested in this because we have a Kickstarter campaign that just started to support our game to teach math. (Go here to watch the video and pledge. It’s awesome. And I know you have an internet connection because you are reading this.)
Here are how the first 33 pledges broke down, which is the first 10% of our funding – $2,000.
Between my two accounts, I have about 3,400 followers. My daughter, Ronda, has 173,000. We both tweeted about this a couple of times. Ignoring the tweet reach Total = 176,400 Number of pledges from twitter = 10 = 0.0006%
I write two blogs. This one averages about 3,100 visits a day and my judo blog is around 1,000. Total in one day I had the widget link up (look at right)
Number of pledges from blog = 3 = 0.073%
Both of us have our twitter accounts post to Facebook. She has 5,000 friends and I, defining friend loosely as someone I have met at least once, have more like 400 friends. Total = 5,400
Number of pledges from Facebook = 5 = 0.093%
Lastly, I emailed a few dozen or so people in my address book, as did my sister, total = 50.
Number of direct link pledges = 10 = 20%
That’s all for what I did. There were an extra five pledges from the Kickstarter site itself, people who looked for games or just clicked on the new projects link.
What can you conclude from this? Well, there is the obvious fact that the more people actually know you, the more likely they are to pledge. If you’re in my address book, we are probably friends or colleagues. Also, at my age, my friends and colleagues are likely to have a job, their kids are grown and they have extra money to pledge. (More about that in my Logan’s Run post later!)
Does that mean social media is useless? Yes and no. It means that your odds of getting funded just using social media probably aren’t that good unless you have a whole lot of friends and followers. Certainly having someone in my corner with 173,000 followers was helpful – as many of them pledged as my friends did – so twitter followers were far less likely to pledge percentage-wise, but there were far more of them.
I do think that social media effects may be over-estimated. When I said I was doing this Kickstarter campaign some of my younger relatives were sure it would be over in a day, they said,
” Just have Ronda tweet it out once, she has over 170,000 followers. If every one of them gives $1 you’ll have made it several times over in a day.”
Ah, the naivete of youth! Now, our campaign just started two days ago, and I know that not as many people read my blog on the weekends (I do keep stats on that). There is also the fact that people on the east coast are buried in snow and have far more to worry about than a Kickstarter campaign of mine.
On the other hand, Ronda has a title fight in two weeks and a TV show on her aired a couple of times this week, so, as far as random events, there has been plenty of random in both directions.
What to make of this? Well, after seeing the numbers, I ran Faker, an application I found on this interesting article on fake twitter followers of athletes. It between Ronda and I we have 72,000 active accounts following us, not fake and actively tweeting. Still, only 10 of them pledged.
According to Blogher, only about half of the visitors to my blogs come to the site long enough for the ads to load, so some of those “visits” are spambots or just not very interested. That still only brings the percent of pledges to .14% .
On seeing the numbers, I am coming to think this way – you need both. And you need them over a period of time. I never did think one tweet was going to make our Kickstarter campaign. However, a couple of dozen tweets over a couple of dozen days, just might.
Since it’s the international year of statistics, I felt a little guilty that I have not done that much in the way of statistics lately. However, I think that is true whether you are an engineer, statistician, biologist or whatever, within a few years after your Ph.D. (at most!) you find lots of your time not being in the area you expected. Let’s look at this week, starting today
Thursday: Kickstarter campaign kicked off. We have been working for a year developing a computer game to teach kids math. (SERIOUSLY CHECK OUT OUR VIDEO HERE & PLEDGE, IT’S AWESOME!) There are statistics in the fifth game in the series – teaching distribution and variability. There was a lot of statistics involved in analyzing the data to back it up – computing item difficulty statistics, internal consistency reliability for the measures, repeated measures ANOVA to test whether the students who played the game progressed in math faster than those who didn’t. (Short answer: Yes.)
However, the biggest part of my day today was spent on Facebook, twitter and phone calls about the Kickstarter campaign, setting up filming for a commercial to be aired next week that a wonderful sponsor offered to do for free.
Wednesday: We’re submitting a grant proposal to USDA to create a series of video games. It will involve a good bit of text mining and data mining to create propensity scores to identify the best match of instructional videos, web pages or games for students based on their mathematics achievement level and characteristics of the instructional resources. Before we get to the statistics, though, there is a lot of writing – literature review, how it relates to Joe Bob Briggs (2011) study of similar stuff, logic models, etc. etc. Wrote blog post on documentation.
Tuesday: Grant-writing. Wishing I was doing mixed model analysis of some interesting data. Grant-writing. Wishing I was doing PHP program for client. Grant-writing. Wishing I was doing text-mining. Wrote blog post on idiots who annoy me. Internet said that was Wednesday but since I hadn’t gone to bed yet, it was Tuesday to me.
Monday: See Tuesday.
Sunday: 7 hours of video editing using Garageband and iMovie with the assistance from a wonderful video editor the folks from the American Horror channel on FilmOn sent out as part of their service to the community. (Thank you guys!) As much as the help was greatly appreciated I must say that anyone who does film editing for a living must put the hyphen in anal-retentive. By the end, I wanted to shoot myself. Then, I did more grantwriting.
Saturday: Heard darling daughter #3 downstairs talking to The Spoiled One. Came down wearing nothing but The Rocket Scientist’s shirt to be met by a photographer from a national magazine who asked,
“Do you mind if I take your picture?”
“I’m not wearing pants, but, whatever.”
Random fact: Darling daughter #3 has a world title fight in the UFC coming up this month so photographers, film crews and journalists have been all over our house like white on rice.
Second random fact: I also had a book published this week, Winning on the Ground – which is about matwork in judo and martial arts.
So, where the hell is the statistics in here? Well, sadly, not nearly in as many places as I would like. In the grant research design sections, there are lots of statistics mentioned. Come February 28th when the Kickstarter campaign is over and the grant proposals are done, I can get back to doing statistics 50% or more of the time. My points though, are;
1. A good bit of being a statistician at the senior level is writing – writing research papers, writing reports, writing proposals.
2. Another good bit of being a statistician is working collaboratively, whether it is with video editors, like this week, or other researchers writing proposals, like last week.
3. You are NOT too old to do a start-up and anyone who thinks you are washed up by 30 can kiss my ass. If you agree, please go to our Kickstarter site and pledge.
Now, I have to get off the computer and go watch this program on TV on darling daughter #3.
We’re waiting by the phone, um, computer to find that our Kickstarter project is (we hope!) approved. In the meantime, our Chief Marketing Officer asked me why there was a need for documentation, which is one of the expenses that would be funded from the crowd.
It’s like this – we started writing a game and we had certain deadlines to meet. We’d promised to have a playable game in the school by October. And we did. And the kids really liked it. And their math skills improved.
In the process, though, we did everything that had to get done to meet those deadlines and other things had to be put off until later.
Version control – later. User’s manual – later. FAQ – later. Anything that wasn’t promised was put on a backburner while we kept our promises.
We have scripts that validate the answers students give to each question and then route them to the appropriate place – either back to play the game or to a site to study whatever math concept they missed. I know what those scripts are named and what they do – but it’s not written down anywhere. The Rocket Scientist has written a lot of C# code for the 3-D world and I know where the latest version is – but it’s not written down anywhere. I also do NOT know where the previous version is (although I could probably guess, or I could go upstairs and ask him).
My point is that we can’t go past the current level of complexity without documenting everything we have done up to this point or we are going to be royally screwed because:
- The further in time we get from when we wrote the first few game levels, the more likely we are to FORGET where stuff is and have one hell of a time fixing any bugs that come up because we won’t be sure what exactly File X does or where the script that does Y is saved.
- As we write new levels, we probably want to re-use code that does certain things like making the character do a little dance to pow-wow music, but we won’t quite remember why we wrote that bit in the middle, so, best leave it in – this is how one ends up with spaghetti code that is impossible to really maintain as no one knows exactly what it does.
- We are getting to the point of scaling up. This means bringing on another programmer or two. That future person is going to need to know where stuff is and what it does.
The truth is that we need to stop and write stuff down before we go any further. It’s kind of like when you are in college and you forego cleaning your apartment because you have to work and study for finals. Well, after finals are over, there is a point where you need to buckle down and clean your place before the fungus on the dishes in the sink evolves, forms its own government and evicts you.
What will anyone who donates to our eventual (we hope!) Kickstarter project get out of this? In front-end visible terms, they will get an on-line, downloadable user’s manual in pdf and html that explains how to play the game, all the levels, has an index to find answers to questions. We think you could probably figure everything out from playing the game, but some people like to read, and lots of people get stuck some times. We’ll have a Frequently Asked Questions page up on our site, and answers (since just the questions aren’t too helpful), for those people like me who don’t want to read a whole manual but just want an answer.
We’ll have a technical support wiki that you won’t see directly but which will help you indirectly because if you do run into a question not covered on our FAQ or a bug that needs to be fixed, one of us will be able to answer it a whole lot faster and better than, “Your guess is as good as mine” or, the one that always drives me crazy, “Well, it works on my machine.”
The other way documentation will help our supporters is that we really won’t be able to progress much farther nearly as fast without it – again, think about your post-finals trashed apartment. Eventually you realize that you are spending more time looking for your keys and cell phone than it would take to clean the place up.
So…. that is why we need funding for documentation.
It never ceases to amaze me that intelligent people will spend huge amounts of time doing a literature review, designing elaborate theories, generating elegant hypotheses, selecting a three-stage stratified random sample, performing multivariate analyses, and their measures on which this brilliant study rests are some questions they made up with their three best friends over Chardonnay during happy hour one Friday night. This is also known as the “panel of experts” method and it has the added benefit that it allows you to deduct the wine on your taxes. (Not actual tax advice. Consult your accountant. Of course, if you are doing your 1040 based on reading this blog, you are probably beyond help.)
We did not go with this approach. Our original idea was to use released items from the state standards test from North Dakota but, unfortunately, that is one of the states that never releases items. What we did was find standards that were the same, verbatim, as other states and then found items from those states that had been released. For example,
” Compute a given percent of a whole number”
and the problem would be
“What is 40% of 250?”
with the same four multiple choice options that had been used on the state test.
As someone pointed out, even if the same test had not been previously, since we pulled only the items that tested exactly what we included in the game, the individual items had been validated. So, we had content validity.
One bit of evidence for construct validity came from the item difficulty levels. Here is one of several charts. This shows what percentage of the fourth-grade students answered each item correctly. The items are broken down by grade level. It is also important to know that the state tests showed the majority of students at this school to be low-performing in mathematics. What we see is that as students go from second-grade level items, all of which the majority of the students answered correctly, to fifth-grade items, the percentage correct declines. We see that for the fifth-grade items, only one of them did the students exceed the 25% that would be answered correct by random guessing (remember, there were four multiple-choice options).
Since the state’s test have shown these students to be performing poorly, we should see that they generally are not at grade level, that is, they do not answer many of the fourth-grade items correctly at a rate exceeding chance. That, as you can see from the chart, is the exact situation.
Of course, we did more than this, beginning with replicating this identical chart with fifth-graders, who showed pretty much the same pattern but, as would be expected, answered a higher proportion correctly at each grade level than did the fourth-graders.
That’s the sort of thing that too many studies take for granted and never test. This isn’t the exciting part of creating a game, the part where you make an attack scene and the kid gets to shoot flaming arrows. So, what good does this do us? Well, the combination of the different analyses of the measure confirms that the measure we used for students to test whether or not their mathematics achievement increased is, in fact, a valid measure of mathematics achievement.
Also, this method has the advantage of not being required to share any of the wine with our best friend/ expert panel so we get to drink it all ourselves.
On twitter, where I probably spend too much of my time, Dave Winer suggested that start-ups write about themselves, thus cutting out Tech Crunch as the middle man. Since Dave wrote the original blogging software and the first blog on the Internet (really!), his advice is worth considering. So, this week is all about our progress.
The most important thing to know is – we finished our pilot test and it works.
As you can see from the graph, both intervention groups increased substantially more from the pre-test to the post-test than did the control group.
Here is what we did:
The reservation had exactly two schools. One was our control group which simply got tested on specific math concepts and then tested again eight weeks later. The other group played our game over the eight week period. Not continuously because there was a break for Thanksgiving and a mid-way break we had planned to do updates. So, after about six actual weeks of hands-on, the kids math scores went up.
We did a repeated measures Analysis of Variance and the time by group interaction effect was significant – meaning there was a statistically significant improvement for our group that played the game. You might wonder (if you were a statistician) if there was an interaction with grade. There are a couple of possible reasons why the fourth-grade did better. One is that the whole class used the game, in their classroom (we provided laptops). The fifth-graders were pulled out of their classroom and only five from each class participated. Also, the fourth-grade teachers, on their own initiative, got much more involved and used the supplemental materials we provided them, increased their own time spent teaching math.
Looking at teacher effects and the effect of whole-class versus pull-out instruction are two things we will be doing next.
TO LEARN MORE ABOUT THE GAME AND OUR NEW START-UP, PLEASE CHECK OUT www.7generationgames.com
I’m working on the latest two proposals for federal funding for our computer games while sitting by the fire in a ski resort in Vermont, which is funny because I have never skied in my life and don’t intend to start now. I’m meeting with two people who we would like to have begin working with us this year. In fact, I think one will start as our new marketing and social media person next week. THEY like to snowboard so here we are meeting in Vermont.
I have a meeting in Boston on Saturday with another group of people before heading home.
In both the proposals I’m writing and the meetings I’ve been having, I’ve been telling people over and over -
“We’re going to make millions on this.”
Sometimes they look at me a little funny - this isn’t the sort of things people in education normally say, and half the purpose of our games is to be educational. The other half is to be fun games or no one will get to the educational part.
It reminds me a lot of when I was competing in judo and told people I was going to win the world championships. They looked at me funny, then, too, because no American had ever done it.
On the other hand, other people have made millions of dollars writing games. Yes, lots of people have not made much money, but one reason lots of them haven’t is the great majority of apps being cranked out are crap. Someone told me it makes me sound arrogant when I say that and who do I think I am that ours will be better – which makes me think of judo again, because I heard the exact same thing when I said I was going to win a gold medal. It really is true, though, that there is a HUGE amount in the educational game space – apps, online games and desktop games – that is just plain garbage. The world is full of people hoping to get rich quick, who use some drag and drop program or learn a very little bit of coding and then upload some piece of schlock waiting for the money to roll in. I learned from judo that the old “let’s all pretend we are equal and get along” bit didn’t cut it. I don’t want to be equal. I want to be better.
That’s another thing I learned from judo – hard work pays off and harder work pays off more. Between Christmas and January 5th, I’ll have had four meetings in three states with potential customers, employees and collaborators. While I’m working on proposals, the Rocket Scientist is hitting the list of fixes, revisions, and improvements.
One more judo lesson – sometimes you have to put in a lot of work before you see the pay off. I can’t exactly say we are betting the farm on this – come on, I’m blogging from a ski resort – but we will certainly be taking a cut in income as we put almost all of our hours that could be billable into the games we are writing instead.
The main lesson I learned from competing for 14 years is that if you keep focused on your goal and keep hacking* away, you eventually reach it. A lot of people say they want to win a gold medal ,because it sounds good. Not too many people are willing to put in the work for the long haul and I understand that. They have other goals. They want a more balanced life.
For me, I want to make these games. There are a few reasons. Making a lot of money is not the primary one, but having started out with very little, I can tell you money is nothing to sneeze at – it makes life easier, more convenient, more comfortable. Wanting to do it is about half because I am just plain interested for the technical challenge of making something that really teaches math to kids who aren’t the type who usually learn it very well. The other half is that it can make a difference in people’s lives. If you do well enough in school to graduate high school, go on to college, or even just know enough math skills to stay continuously employed – all of that makes a huge difference compared to dropping out of high school, not graduating from college or not being able to get any kind of job.
Why not just do it and give the game away? That’s really not practical. We need artwork, music , editing, testing, documentation, marketing – people aren’t willing to work for no money and we can only do so much ourselves. That is why so much stuff available is garbage – it takes a lot of time and money to make a good game. If you are talking about one to teach math, on top of time, money, talented artists, excellent developers and everything else that goes into a game, you also need people who understand not just math but how to teach it.
I’m not sure if this counts as a New Year’s resolution or more of a prediction, but it’s going to be a great year for math games.
* The Rocket Scientist told me I shouldn’t use the words “hacking” or “programmer” because they are low-level. Consequently, I use them both as much as possible to show that he is not the boss of me and word chooser of the house.
P.S. If you’re going to be at the workshop with JMM next week on Using Games to teach Statistics, say hey.