Years ago, a friend of mine was in college and had an old, beat up car that leaked oil on to the street where it was parked, which, for some reason annoyed her elderly neighbor. When we returned from a trip overseas competing for the U.S., there was a notice on her car – the neighbor had reported the car as abandoned and we got home just in time to stop the city from towing it away. As a joke, the coach got her a bumper sticker that read, “This is not an abandoned vehicle.”
It’s almost two weeks since I last posted. Contrary to appearances, this is not an abandoned blog!
I just this minute – hurray, tap-dancing – submitted a grant I’ve been working on for the past two weeks.
While writing the grant this week, I’ve been in North Dakota, first giving a presentation on Using Native American Culture to Increase Math Performance. You can see a bit of it that was shown on the local TV station here.
After meeting lots of students at Minot State, we headed over to the Minot Job Corps and I met with students and faculty, talking about our games, starting a company and life in general.
On to New Town, on the Fort Berthold Reservation where I met with the staff and students from the Boys and Girls Club, again, gave demonstrations of our games, and threw a judo demonstration in along with it.
Along there somewhere, I finished the final report on our Dakota Math project that once again found significant improvement in performance of students who played our games, hired two more employees, signed another consulting contract, had way too many meetings and squashed a few bugs in the games.
Tomorrow, I head home to Santa Monica, for two weeks, until I head out to Fort Totten, ND. In the meantime, I’m back to blogging. Did you miss me?
The results are in! The chart below gladdens my little heart, somewhat.
One thing to note is the fact that the 95% confidence interval is comfortably above zero. Another point is that it looks like a pretty normal distribution.
What is it? It is the difference between pretest and post-test scores for 71 students at two small, rural schools who played Spirit Lake: The Game.
I selected these schools to analyze first, and held my breath. These were the schools we had worked with the most closely, who had implemented the games as we had recommended (play twice a week for 25-30 minutes). If it didn’t work here, it probably wasn’t going to work.
Two years ago, with a sample of 39 students from 4th and 5th grade from one school, we found a significant difference compared to the control group.
COULD WE DO IT AGAIN?
You probably don’t feel nervous reading that statement because you have not spent the last three years of your life developing games that you hope will improve children’s performance in math.
The answer, at least for the first group of data we have analyzed is – YES!
Scores improved 20% from pre-test to post-test. This was not as impressive as the improvement of 30% we had found in the first year, but this group also began with a substantially higher score. Two years ago, the average student scored 39% on the pre-test. This year, for 71 students with complete data, the average pre-test score was 47.9% , the post-test mean was 57.4%. I started this post saying my little heart was gladdened “somewhat” because I still want to see the students improve more.
There is a lot more analysis to do. For a start, there is analysis of data from schools who were not part of our study but who used the pretest and post-test – with them, we can’t really tell how the game was implemented but at least we can get psychometric data on the tests.
We have data on persistence – which we might be able to correlate with post-test data, but I doubt it, since I suspect students who didn’t finish the game probably didn’t take the post-test.
We have data on Fish Lake, which also looks promising.
Overall, it’s just a great day to be a statistician at 7 Generation Games.
Here is my baby, Spirit Lake. It can be yours for ten bucks. If you are rocking awesome at multiplication and division, including with word problems, but you’d like to help out a kid or a whole classroom, you can donate a copy.
Some problems that seem really complex are quite simple when you look at them in the right way. Take this one, for example:
My hypothesis is that a major problem in math achievement is persistence. Students just give up at the first sign of trouble. I have three different data sets with student data from the Spirit Lake game. Many of the students in the student table are the control group, so they will have no data on game play. There is a table of answers to the math challenges and another table with answers to quizzes which students took only if they missed a math challenge. When students miss a math challenge in the game, depending on which educational resource they choose, they may do one of two or three different quizzes to get back into the game. Also, some of the quiz records were not from quizzes actually in the game but from supplemental activities we provided. So, how do I identify where in the process students drop out and present in a simple graphic to discuss with schools? Just to complicate matters, the username was different lengths in the different datasets and the variable for timestamp also had different names.
It turns out, the problem was not that difficult.
- Merge the student table with the answers (math challenges) and only include those students with at least one answer.
- Merge the student table with the quizzes and only include those students with at least one quiz
- Concatenate the data sets from steps 1 & 2
- Create a new userid variable and set it equal to the username
- Create a new “entered” variable and set it equal to whichever of the datetime fields exists on that record
- Delete the quizzes not included in the game.
- Sort the dataset by userid and the date and time entered.
- Keep the last record for each userid. Now you have their last date of activity.
- If there is a value for the math challenge field then that is the name of the last activity, otherwise the quiz name is the name for the last activity.
- Use a PROC FORMAT to assign each activity a value equal to the step in the game.
- Do a PROC FREQ using that format and the order = FORMATTED option.
Once I had the frequencies, I just put them into a table in a word document and shaded the columns to match the percentage. There may be a way in SAS/Graph or something else to do this automatically, but honestly, the table took me two minutes once I had the data.
I think it illustrates my points pretty clearly, which are:
- A sizable number of students drop out after the second problem.
- 25% of the students drop after the first difficulty they have (missing the second problem)
- Only a minority of students persist all the way to the end, less than 25% of the total sample
This isn’t based on a tiny sample, either. The data above represent a sample of 397 students.
In case you would like to see it, the code for steps 3-11 is below. Particularly useful is the PROC FORMAT. Notice that you can have multiple values have the same format, which was important here because players can take multiple paths that are still the same step in the sequence.
data persist ;
attrib userid length= $49 ;
set mydata2.sl_answers mydata2.sl_quizzes ;
entered = max(date_answered_dt,date_taken_dt) ;
**** DELETES QUIZZES IN EXTRA AND SUMMER SITE, NOT IN MAIN GAME ;
if quiztype in (“problemsolve”,”divide1long”,”multiplyby23″) then delete ;
userid = new_username ;
format entered datetime20. ;
proc sort data=persist ;
by quiztype ;
proc sort data=persist ;
by userid entered ;
data retention ;
set persist ;
by userid ;
if last.userid ;
attrib last_activity length= $14 ;
if inputform ne “” then last_activity = inputform ;
else last_activity = quiztype ;
proc freq data= retention ;
tables last_activity ;
proc format ;
“findcepansi” = “01”
“x2x9” = “02”
“math2x” = “02”
“math2_2” = “02”
“wolves1a” = “02”
“multiplyby5” = “03”
“multiplyby4” = “03”
“multiplyby3” = “04”
“wolves1b” = “05”
…. AND SO ON ….
“horseform2” = “21”
ods rtf file = “C:\Users\Spirit Lake\phaseII\pipeline.rtf” ;
proc freq data= retention order=formatted ;
tables last_activity ;
format last_activity $activity. ;
ods rtf close ;
Many years ago, I was walking through the exhibits at the county fair with my late husband (he was alive then, that’s why he was able to walk with me) and I lamented,
Look at those quilts. My grandmother makes quilts. Look at those crocheted tablecloths. My other grandmother crochets. Look at me – what do I make?
My wonderful husband turned to me and said in his good-old-boy, country accent,
Money. That’s what you make that your grandmothers didn’t make. You make money, darlin’.
Everyone is posting pictures of the cute Halloween costumes their mom made for them or that they made for their children. I never made a Halloween costume in my life, but here is a copy of some code I finished last weekend that makes a graph with different types of pastries. Another function I wrote (not shown here) changes it from Spanish to English. If you get it correct, it takes you to another problem that does bar graphs with actual bars.
I didn’t make a costume but I did make money from working on this project which The Spoiled One can use to buy whatever costume she likes.
Just finished the second week of the Boom Startup Ed Tech accelerator, which is GREAT. I am learning a lot and plan on working insane hours for the next few months to put it all into practice.
We have had to answer a lot of legitimately hard questions about cash flow analysis, burn rate, user base, sales strategy and more. We’re a really small company and creating a product while creating a company is a hard road with just a few people.
After a month away, I sit down at my computer to read more feel good tweets about “teaching girls to code to improve diversity in tech.”
Seriously, fuck all you people.
(If you’re actually in a room teaching girls to code instead of just tweeting about it, that doesn’t apply to you. Anyone who is actually in a classroom teaching instead of pontificating about education has my sincere respect. See Mentoring below.)
Do you know what would increase the number of minorities and women in tech? If they actually saw people succeeding in it and THE REASON THEY AREN’T SUCCEEDING ISN’T LACK OF CODING SKILLS.
The very best sessions (of many great ones) at Boom were by a couple of successful entrepreneurs who talked honestly about their career paths. Repeatedly, they emphasized that there is a knife’s edge of difference between success and failure.
It’s not that only white and Asian males can possibly succeed in technology companies, but the statistics are decidedly skewed.
Let me explain this to you.
No matter how good your product or coding skills, almost everyone faces the same challenges.
- Access to capital. How do you live while building a product? You can do it as a side project after work which then gets you into the Catch-22 of investors want (rightfully so) to see your startup as a full-time job but you can’t afford to quit your job because then you will have no income.
- Access to users. This is another Catch-22 where people don’t want to invest in a product that has no users but with millions of apps available and hundreds of millions of web pages, billions of tweets, it’s very hard to rise above the noise and attract users without capital to advertise, attend conferences, hire sales staff.
- Mentoring. No one knows it all. I have an MBA, a Ph.D. and twenty years of business experience and I am learning A LOT through the accelerator program. There is always going to be someone better than you at some aspects of your business – whether it is sales strategy, financial modeling, or, yes, even coding. Having access to those people is golden.
- Access to capital. Whether it is through sales (not likely) or investor funds you are going to need money to grow. At a minimum you need good credit, but even better investors. Say you do get a million users who download and pay for your game. Now you need enough tech support for a million users, enough administrative staff to respond when people have a problem with payment, updates, incompatible operating systems, etc. etc. If you are dealing with institutional sales, say, to schools, you may need the people right away but the payments come later. How do you hire those people without money?
So, yes, teaching girls to code (or boys, for that matter), is a nice thing. If you are really concerned about having a more diverse workforce, though, maybe you could try supporting the companies run by women.
Really want to support diversity in tech while having fun and getting smarter? Try our adventure games that teach math in a virtual world.
Get games here for $9.99 each. Play it yourself, give it to your kids. You can donate to a child or school, too.
Are you an angel investor? I’d be happy to talk with you. email@example.com
If you are offended by people who say ‘fuck’ a lot, Maria Burns Ortiz, our CMO, would be happy to talk with you.
Previously, I discussed PROC FREQ for checking the validity of your data. Now we are on to data analysis, but, as anyone who does analysis for more than about 23 minutes can tell you, cleaning your data and doing analysis is seldom a two-step process. In fact, it’s more like a loop of two steps, over and over.
First, we have the basic.
PROC FREQ DATA = mydata.quizzes ;
TABLES passed /binomial ;
This will give me not only what percentage passed a quiz that they took,
but also the 95% confidence limits.
I didn’t have any real justification for hypothesizing any other population value. What proportion of kids should be able to pass a quiz that is ostensibly at their grade level? Half of them – as in, the “average” kid? All of them, since it’s their grade level? I’m sure there are lots of numbers one could want to test.
If you do have a specific proportion, say, 75%, you’d code it like this:
PROC FREQ DATA =in.quizzes ;
TABLES passed / BINOMIAL (P=.75);
Note that the P= has to be enclosed in parentheses or you’ll get an error.
So, out of the 770 quizzes that were taken by students, only 30.65% of them passed. However, the quizzes aren’t all of equal difficulty, are they? Probably not.
So, my next PROC FREQ is a cross-tabulation of quiz by passed. I don’t need the column percent or percent of total. I just want to know what percent passed or failed each quiz and how many players took that quiz. The way the game is designed, you only need to study and take a quiz if you failed one of the math challenges, so there will be varying numbers of players for each quiz.
PROC FREQ DATA =in.quizzes ;
TABLES quiz*passed /NOCOL NOPERCENT ;
The first variable will be the row variable and the one after the * will be the column variable. Since I’m only interested in the row percent and N, I included the NOCOL and NOPERCENT options to suppress printing of the column and total percentages.
Before I make anything of these statistics, I want to ask myself, what is going on with quiz22 (which actually comes after quiz2) and quiz4? Why did so many students take these two quizzes? I can tell at a glance that it wasn’t a coding error that made it impossible to pass the quiz (my first thought), since over a quarter of the students passed each one.
This leaves me three possibilities:
- The problem before the quiz was difficult for students, so many of them ended up taking the quiz (another PROC FREQ)
- One of the problems in the quiz was coded incorrectly, so some students failed the quiz when they shouldn’t have,
- There was a problem with the server repeatedly sending the data that was not picked up in the previous analyses (another PROC FREQ).
Remember what I said at the beginning about data analysis being a loop? So, back to the top!
I’m the world’s biggest hypocrite when it comes to documentation. In every staff meeting, I emphasize documenting whatever code you have written that week, but I always put off doing it myself. My excuse is always that it isn’t final and when I get the complete project done I will put it in the wiki.
I’ve come to the conclusion that no complex software is ever done. You just quit working on it and go to something else.
If you have run into the awful problem of having your animation run on some browsers and not others, or even run sometimes and sometimes not in the same browser, you may have a timing problem. In brief, you are trying to draw the image before it is loaded.
Here is what I did today and how I fixed that problem…
In this part of the game (Forgotten Trail) , the player has answered a question that asks for the average number of miles the uncle walked each day when he attempted this journey. If the player gives the correct answer, say, 22 miles, a screen pops up and the mother asks, “Do you really think you can walk 22 miles a day?” If the player says, “No,” then he or she is sent to workout. Each correct answer runs your character 5 miles. After 20 miles, you can go back to the game.
So …. we need animation and sound that occurs when a correct answer is submitted. I had finished the code to randomly generate division problems, so today I was working on the function after the player is correct.
The HTML elements are pretty simple – a div that contains everything, two layers of canvas, a button and two audio elements.
<canvas id="layer1" style="z-index: 1; position:absolute; left:0px; top:5px;" height="400px" width="900"> This text is displayed if your browser does not support HTML5 Canvas. </canvas> <canvas id="layer2" style="z-index: 1; position:absolute;left:0px;top:5px;" height="400px" width="900"> This text is displayed if your browser does not support HTML5 Canvas. </canvas> <button id="workout" name="workout" >CLICK TO WORK OUT</button> </div> <audio autoplay id="audio1"><source type="audio/ogg"></audio> <audio autoplay id="audio2"><source type="audio/mpeg"></audio> </div>
Because the character is only moving horizontally – he is running on a field – there is no dy and no y variable.
I tell clients on our statistical consulting side all of the time that if your conclusion is only valid if you look at this specific subset of your sample, with this particular statistical technique. You need to look for a convergence or results. Does the mean score increase? Does the proportion of people passing a test increase? Do the test scores still increase when you co-vary for the pretest score?
(This is for my friend, Dr. Longie, who tells me I always put too many numbers in things and should get to the point – no matter how we sliced it, the scores of students who played Fish Lake improved over 30% from their pretest. Analysis is continuing on Spirit Lake and other data from Fish Lake. There! Are you happy now?)
We are just at the very beginning stages of analyzing data from the second phase of our research grant funded by the U.S. Department of Agriculture. Coincidentally, we are in Maryland at the National SBIR Conference this week and got the chance to meet in person all of the folks whose email we have been receiving for years.)
When we were in the middle of developing and testing Fish Lake, one of the interns in our office asked me,
“Are you sure this is going to work?”
I told her,
“No, I’m not sure. That’s why they call it research.”
School has now ended at all of our test sites and I have just completed cleaning the data for analysis from the first data set, which is the pre- and post-test data for Fish Lake, our game that teaches fractions as your avatar retraces the Ojibwe migration – canoeing, hunting and fishing your way across the continent.
So … what happened?
The first thing I did was compute the mean and standard deviation for the students who completed the pretest and the posttest. Then, I merged the datasets together and did a paired t-test for the 61 students who took the post-test and pre-test both. I didn’t show you any of those results because I assumed (correctly) that the merge would have to be reviewed because some people would have misspelled their username on the pretest or posttest. Surprisingly, I only found two of those, as well as one record that was just testing the software by one of our interns. The programs that I developed to clean the data (programs presented at a couple of regional SAS software conferences) worked pretty well.
Then, I re-ran the analysis.
Pre-test mean = 22.4%, SD= 16.5% N=260
Post-test mean = 30.8% SD =17.4% N=63
So far, so good. We were not surprised by the low scores on the pretest. We knew that the majority of students in several of our test schools were achieving a year or two below grade level. The improvement from pre-test to post-test of 8.4% represented an improvement in test scores of 37.5%
BUT …. what if the students who did not take the post-test were the lower performing students? Shouldn’t we do a pretest and post-test comparison only including matched pairs?
This brings us to ….
Result 2 – With Matched Pairs
Pre-test mean = 23.6%, SD= 17.4% N=63
Post-test mean = 30.8% SD =17.4% N=63
As hypothesized, the students who completed the post-test scored higher on the pretest than the average, but not dramatically so. The difference was still statistically significant (p < .01)
What about outliers? That standard deviation seems awfully high to me and when I look at the raw data I find five players who have a 0 on the pretest or post-test and one who had one of the highest scores on the pretest whose test is blank after the first few questions.
Now, it is possible that those students just knew none of the questions – but it appears they just entered their username and (almost) nothing else. I deleted those 6 records and got this
Result 3 – Matched Pairs with Outliers Deleted
Pre-test mean =24.4 SD =16.2 N=57
Post-test mean = 32.7 SD = 16.7 N=57
With a difference of 8.3 percentage points, this presents an improvement of 34% (p<.001 )
Conclusion ? Well, we are not even close to a conclusion because we have a LOT of more data still to analyze, but what I can say is that the results are looking promising.
Is that enough acronyms for you? I’ll be speaking at Celebrating Equity: Women in STEM at ELAC.
STEM = Science, Technology, Engineering and Mathematics
ELAC – East Los Angeles College
It’s 12:15 – 1:15 pm and it is free. There are six panelists (including me).
I presented last year and our company also had a booth. We hired two people who I met there.
Often, I hear people say that their company is all white/ Asian /under 40 because all of the developers / animators/ audio engineers that applied just happened to fit that demographic. Here is a thought – perhaps you could go to, say, a college that is predominantly Hispanic and just maybe Hispanic potential employees might meet you there.
Here is another thought – perhaps if you attended events targeted at women in STEM, you might meet some there.
Wonder what a game would look like if it was created by a design and development team that was predominantly women?
Some people may have said that hackathons are a stupid ass idea where a bunch of people who have can’t afford to buy their own pizza spend 48 hours with a bunch of strangers and no showers.
Okay, well, maybe that was me.
I take it all back.
We kicked off our hackathon at noon on Monday and wrapped up at 8 pm on Tuesday. The rules were simple – everyone who was working those days was to wipe their schedule completely for 8 hours each day and do nothing but work on the game. No emails, no blog posts, no meetings except for a kick off meeting each day to assign and review tasks. Jessica, Dennis, Samantha and I worked on the game for (at least) 16 hours. Any emails or interviews got done before the hackathon hours or after they were over. (I did pause for a brief interview with the Bismarck State College paper.)
Maria came in from maternity leave and worked 8 hours on Monday, baby in tow.
Gonzalo and Eric each worked their regular shifts on Monday and Tuesday, respectively, doing nothing but writing code, creating sprites and editing audio. Sam even pitched in a few hours early in the morning from Canada. Our massively talented artist, Justin, completed all of the new artwork before the meeting so we had it in hand to drop into all of the spots where there had been placeholders.
So, in two days a total of 100 hours were devoted just to game development. We made a giant leap forward.
Why did it work so well? For one thing, we were all in the same spot for a long time. Although the original plan was to meet and then people would go there separate ways, on Monday, five of the six people working stayed at my house. Three of us even slept there. That had two positive impacts.
First of all, whenever anyone needed something, whether it was a piece of artwork modified or a question answered on whether we had a sound file of footsteps in the woods or to be shown how to do a voice over in iMovie, there was someone else to provide that assistance right on the spot. Very often, you can spend hours searching for something on Google, watching youtube videos, reading manuals trying to figure out how to do X when someone else can come up and say – Click on Window, pick record voiceover, click on the microphone in the middle of the left side of the window.
There are also those questions that CANNOT be found on Google, like where the hell was the new background image saved and what is it called.
The second positive impact was we got around to tasks that needed doing for a long time. While it may have seemed it kept us from getting real progress done on the game, the fifth time Sourcetree complained about not tracking those damned Dreamweaver .idea files, I HAD it and we removed those from the repository forever. When something bugs you every now and then you may think, “I’ll do it later”, but the fifth time it happens that day …
Anyway, I would share more of the awesomeness of the hackathon experience with you but it is now 9 pm and we are taking the team out for sushi.