Previously, I discussed PROC FREQ for checking the validity of your data. Now we are on to data analysis, but, as anyone who does analysis for more than about 23 minutes can tell you, cleaning your data and doing analysis is seldom a two-step process. In fact, it’s more like a loop of two steps, over and over.
First, we have the basic.
PROC FREQ DATA = mydata.quizzes ;
TABLES passed /binomial ;
This will give me not only what percentage passed a quiz that they took,
but also the 95% confidence limits.
I didn’t have any real justification for hypothesizing any other population value. What proportion of kids should be able to pass a quiz that is ostensibly at their grade level? Half of them – as in, the “average” kid? All of them, since it’s their grade level? I’m sure there are lots of numbers one could want to test.
If you do have a specific proportion, say, 75%, you’d code it like this:
PROC FREQ DATA =in.quizzes ;
TABLES passed / BINOMIAL (P=.75);
Note that the P= has to be enclosed in parentheses or you’ll get an error.
So, out of the 770 quizzes that were taken by students, only 30.65% of them passed. However, the quizzes aren’t all of equal difficulty, are they? Probably not.
So, my next PROC FREQ is a cross-tabulation of quiz by passed. I don’t need the column percent or percent of total. I just want to know what percent passed or failed each quiz and how many players took that quiz. The way the game is designed, you only need to study and take a quiz if you failed one of the math challenges, so there will be varying numbers of players for each quiz.
PROC FREQ DATA =in.quizzes ;
TABLES quiz*passed /NOCOL NOPERCENT ;
The first variable will be the row variable and the one after the * will be the column variable. Since I’m only interested in the row percent and N, I included the NOCOL and NOPERCENT options to suppress printing of the column and total percentages.
Before I make anything of these statistics, I want to ask myself, what is going on with quiz22 (which actually comes after quiz2) and quiz4? Why did so many students take these two quizzes? I can tell at a glance that it wasn’t a coding error that made it impossible to pass the quiz (my first thought), since over a quarter of the students passed each one.
This leaves me three possibilities:
- The problem before the quiz was difficult for students, so many of them ended up taking the quiz (another PROC FREQ)
- One of the problems in the quiz was coded incorrectly, so some students failed the quiz when they shouldn’t have,
- There was a problem with the server repeatedly sending the data that was not picked up in the previous analyses (another PROC FREQ).
Remember what I said at the beginning about data analysis being a loop? So, back to the top!
I’m the world’s biggest hypocrite when it comes to documentation. In every staff meeting, I emphasize documenting whatever code you have written that week, but I always put off doing it myself. My excuse is always that it isn’t final and when I get the complete project done I will put it in the wiki.
I’ve come to the conclusion that no complex software is ever done. You just quit working on it and go to something else.
If you have run into the awful problem of having your animation run on some browsers and not others, or even run sometimes and sometimes not in the same browser, you may have a timing problem. In brief, you are trying to draw the image before it is loaded.
Here is what I did today and how I fixed that problem…
In this part of the game (Forgotten Trail) , the player has answered a question that asks for the average number of miles the uncle walked each day when he attempted this journey. If the player gives the correct answer, say, 22 miles, a screen pops up and the mother asks, “Do you really think you can walk 22 miles a day?” If the player says, “No,” then he or she is sent to workout. Each correct answer runs your character 5 miles. After 20 miles, you can go back to the game.
So …. we need animation and sound that occurs when a correct answer is submitted. I had finished the code to randomly generate division problems, so today I was working on the function after the player is correct.
The HTML elements are pretty simple – a div that contains everything, two layers of canvas, a button and two audio elements.
<canvas id="layer1" style="z-index: 1; position:absolute; left:0px; top:5px;" height="400px" width="900"> This text is displayed if your browser does not support HTML5 Canvas. </canvas> <canvas id="layer2" style="z-index: 1; position:absolute;left:0px;top:5px;" height="400px" width="900"> This text is displayed if your browser does not support HTML5 Canvas. </canvas> <button id="workout" name="workout" >CLICK TO WORK OUT</button> </div> <audio autoplay id="audio1"><source type="audio/ogg"></audio> <audio autoplay id="audio2"><source type="audio/mpeg"></audio> </div>
Because the character is only moving horizontally – he is running on a field – there is no dy and no y variable.
I tell clients on our statistical consulting side all of the time that if your conclusion is only valid if you look at this specific subset of your sample, with this particular statistical technique. You need to look for a convergence or results. Does the mean score increase? Does the proportion of people passing a test increase? Do the test scores still increase when you co-vary for the pretest score?
(This is for my friend, Dr. Longie, who tells me I always put too many numbers in things and should get to the point – no matter how we sliced it, the scores of students who played Fish Lake improved over 30% from their pretest. Analysis is continuing on Spirit Lake and other data from Fish Lake. There! Are you happy now?)
We are just at the very beginning stages of analyzing data from the second phase of our research grant funded by the U.S. Department of Agriculture. Coincidentally, we are in Maryland at the National SBIR Conference this week and got the chance to meet in person all of the folks whose email we have been receiving for years.)
When we were in the middle of developing and testing Fish Lake, one of the interns in our office asked me,
“Are you sure this is going to work?”
I told her,
“No, I’m not sure. That’s why they call it research.”
School has now ended at all of our test sites and I have just completed cleaning the data for analysis from the first data set, which is the pre- and post-test data for Fish Lake, our game that teaches fractions as your avatar retraces the Ojibwe migration – canoeing, hunting and fishing your way across the continent.
So … what happened?
The first thing I did was compute the mean and standard deviation for the students who completed the pretest and the posttest. Then, I merged the datasets together and did a paired t-test for the 61 students who took the post-test and pre-test both. I didn’t show you any of those results because I assumed (correctly) that the merge would have to be reviewed because some people would have misspelled their username on the pretest or posttest. Surprisingly, I only found two of those, as well as one record that was just testing the software by one of our interns. The programs that I developed to clean the data (programs presented at a couple of regional SAS software conferences) worked pretty well.
Then, I re-ran the analysis.
Pre-test mean = 22.4%, SD= 16.5% N=260
Post-test mean = 30.8% SD =17.4% N=63
So far, so good. We were not surprised by the low scores on the pretest. We knew that the majority of students in several of our test schools were achieving a year or two below grade level. The improvement from pre-test to post-test of 8.4% represented an improvement in test scores of 37.5%
BUT …. what if the students who did not take the post-test were the lower performing students? Shouldn’t we do a pretest and post-test comparison only including matched pairs?
This brings us to ….
Result 2 – With Matched Pairs
Pre-test mean = 23.6%, SD= 17.4% N=63
Post-test mean = 30.8% SD =17.4% N=63
As hypothesized, the students who completed the post-test scored higher on the pretest than the average, but not dramatically so. The difference was still statistically significant (p < .01)
What about outliers? That standard deviation seems awfully high to me and when I look at the raw data I find five players who have a 0 on the pretest or post-test and one who had one of the highest scores on the pretest whose test is blank after the first few questions.
Now, it is possible that those students just knew none of the questions – but it appears they just entered their username and (almost) nothing else. I deleted those 6 records and got this
Result 3 – Matched Pairs with Outliers Deleted
Pre-test mean =24.4 SD =16.2 N=57
Post-test mean = 32.7 SD = 16.7 N=57
With a difference of 8.3 percentage points, this presents an improvement of 34% (p<.001 )
Conclusion ? Well, we are not even close to a conclusion because we have a LOT of more data still to analyze, but what I can say is that the results are looking promising.
Is that enough acronyms for you? I’ll be speaking at Celebrating Equity: Women in STEM at ELAC.
STEM = Science, Technology, Engineering and Mathematics
ELAC – East Los Angeles College
It’s 12:15 – 1:15 pm and it is free. There are six panelists (including me).
I presented last year and our company also had a booth. We hired two people who I met there.
Often, I hear people say that their company is all white/ Asian /under 40 because all of the developers / animators/ audio engineers that applied just happened to fit that demographic. Here is a thought – perhaps you could go to, say, a college that is predominantly Hispanic and just maybe Hispanic potential employees might meet you there.
Here is another thought – perhaps if you attended events targeted at women in STEM, you might meet some there.
Wonder what a game would look like if it was created by a design and development team that was predominantly women?
Some people may have said that hackathons are a stupid ass idea where a bunch of people who have can’t afford to buy their own pizza spend 48 hours with a bunch of strangers and no showers.
Okay, well, maybe that was me.
I take it all back.
We kicked off our hackathon at noon on Monday and wrapped up at 8 pm on Tuesday. The rules were simple – everyone who was working those days was to wipe their schedule completely for 8 hours each day and do nothing but work on the game. No emails, no blog posts, no meetings except for a kick off meeting each day to assign and review tasks. Jessica, Dennis, Samantha and I worked on the game for (at least) 16 hours. Any emails or interviews got done before the hackathon hours or after they were over. (I did pause for a brief interview with the Bismarck State College paper.)
Maria came in from maternity leave and worked 8 hours on Monday, baby in tow.
Gonzalo and Eric each worked their regular shifts on Monday and Tuesday, respectively, doing nothing but writing code, creating sprites and editing audio. Sam even pitched in a few hours early in the morning from Canada. Our massively talented artist, Justin, completed all of the new artwork before the meeting so we had it in hand to drop into all of the spots where there had been placeholders.
So, in two days a total of 100 hours were devoted just to game development. We made a giant leap forward.
Why did it work so well? For one thing, we were all in the same spot for a long time. Although the original plan was to meet and then people would go there separate ways, on Monday, five of the six people working stayed at my house. Three of us even slept there. That had two positive impacts.
First of all, whenever anyone needed something, whether it was a piece of artwork modified or a question answered on whether we had a sound file of footsteps in the woods or to be shown how to do a voice over in iMovie, there was someone else to provide that assistance right on the spot. Very often, you can spend hours searching for something on Google, watching youtube videos, reading manuals trying to figure out how to do X when someone else can come up and say – Click on Window, pick record voiceover, click on the microphone in the middle of the left side of the window.
There are also those questions that CANNOT be found on Google, like where the hell was the new background image saved and what is it called.
The second positive impact was we got around to tasks that needed doing for a long time. While it may have seemed it kept us from getting real progress done on the game, the fifth time Sourcetree complained about not tracking those damned Dreamweaver .idea files, I HAD it and we removed those from the repository forever. When something bugs you every now and then you may think, “I’ll do it later”, but the fifth time it happens that day …
Anyway, I would share more of the awesomeness of the hackathon experience with you but it is now 9 pm and we are taking the team out for sushi.
Our Project Manager, Jessica, made the very insightful comment at lunch the other day,
No one cares how hard it was for you to make. When people are looking to buy your product, all they want to know is what it will do for them.
That young woman has a bright future in marketing. Unfortunately for those who read this blog, I do not, so I am going to tell you how hard it is to make that last push to the finish line.
I quit counting the number of hours I worked this week when I got to 80. I’m sure The Invisible Developer had put in even more, because many nights (mornings?) I have gone to bed at 2 a.m. and when I wake up and check the latest build in the morning I find it was put up at 5 or 6 that morning. There hasn’t been much blogging going on lately and I only have a bit of a minute now because I’m waiting to get the latest latest latest build so that I can make the Windows installer.
I’ve blogged before on the great value I place on “details” people and this week is a prime example of the importance of details.
You’d think that down to and past the wire – the last build of the game was supposed to be today and we have negative 68 minutes left in today – that we would be moving forward pretty quickly. Um, not so much.
At the beginning of development, you can easily find the problems – the question is what fraction of the fish are over one foot long when you caught 125 fish last summer and 25 were over a foot long. The correct answer is 1/5. However, 25/125 is also a correct answer, as is 5/25 . Finding those problems is easy. You can check the answer while you are creating the pages, have it write to the console the correct answer, step through the logic. No problem.
Same thing with playing the 3-D part of the game. If you are at the part where you are supposed to be spearing the fish and there is no spear, then it is an easy enough fix.
HOWEVER, now we are supposedly at the end. So…
- We make a version of the build for Mac and another for Windows.
- We zip the Windows file because many systems block .exe files downloaded from the Internet to prevent malware installation.
- We upload the zipped file to our server.
- We download it.
- We play the game from beginning to end on Mac.
- We play the game from beginning to end on Windows.
That is, we go through every step that a user would — and somewhere along the way we find an error that we somehow missed in all of our earlier testing. Maybe something we fixed in a later stage of the game was a script that was used in an earlier level and now that doesn’t work.
So … we go through all of the steps all over again. Yes, we do have debugging capabilities where we can skip to level 6 and test that, for example, but at the very end, you NEED to go through all of the steps your users will. Trust me. You can put in every unit test you want but it will not let you know that Microsoft or Chrome or some other organization put on this earth to try my patience now has a security feature that blocks the game from installing. You won’t see that three problems and all of the accompanying instructional material were left out.If you start at level 6 you will miss the fact that there is a problem in the transition from level 5 to level 6. And so on ad inifinitum until you go to speaking in Latin and wanting to tear out your eyeballs.
We go through all of the details so that when you play it all you see is a game that works.
My high school English teacher told me,
If something is easy to read, you can damn sure believe that it was hard to write.
I think this is also true,
Any kind of software that is easy to use, you can damn sure believe it was hard to make.
This is the most depressing chart I have seen in a long time. Below are the results of our pretest on knowledge of fraction operations of 322 students in grades 3 through 7, attending schools on and adjacent to American Indian reservations.
These are questions like,
“Drag 6/1 to the correct spot on the number line.”
Which was one of only two questions that at least 50% of the children answered correctly.
Identify the letter that marks 7/8 on a number line
14% of the children answered that right.
Then there are the word problems,
“Bob and Ted painted a wall. Bob painted 1/5 of the wall and Ted painted 2/5 of the wall. How much of the wall is left to paint?”
38% of the children answered that correctly.
Looks like they did better on item 7, which asks which of these statements is true
5/ < 3/4
2/8 < 1/4
3/6 = 6/
2/ = 4/5
26% of them got that correct. Guess what? That was one of the few multiple choice items on the test, so random guessing would have gotten it correct 25% of the time.
This is a test of what is ostensibly third- through fifth-grade math. Two-thirds of the test is at the fourth-grade level or below. As our results indicate, the majority of the students who took the test would not understand what that statement means.
For the 163 fifth-graders who took our pretest, the mean score was 28%.
For the 114 fourth-graders, the mean was a dismal 14.7%.
It wasn’t that the students didn’t try. I looked and there were very few places they left the items blank. They simply did not know.
These students came from several different schools, and while there may be differences between schools, there is nothing to suggest one school with abysmal results pulled down all of the others.
I called our lead cultural consultant, Dr. Erich Longie, out at Spirit Lake, and told him that I was concerned about presenting these results to the schools that they might want to shoot the messenger. After all, it is important to us that these schools continue to provide us their input and guidance. He told me not to worry about it too much.
“They know,” he told me, “As someone who has been a teacher and administrator in schools on the reservations, I’m not surprised by the results and I can’t imagine these schools will be, either. What we all ought to be worried about is making sure that the post-test scores don’t look like this.”
So … students will start playing Fish Lake in the schools next month. No pressure here.
Excuse me while I get back to work.
I was reading a book on PHP just to get some ideas for a re-design I’m doing for a client, when I thought of this.
Although I think of PHP as something you use to put stuff into a database and take it out – data entry of client records, reports of total sales – it is possible to use without any SQL intervention.
You can enter data in a form, call another file and use the data entered to determine what you show in that file. The basic examples used to teach are trivial – a page asks what’s your name and the next page that pops up says, “Howdy” + your name .
Generally, when a player answers a question, the answer is written to our database and the next step depends on whether the answer was correct. Correct, go on with the game. Incorrect, pick one of these choices to study the concept you missed. Because schools use our games, they want this database setup so they can get reports by student, classroom, grade or school.
I could also use PHP.
In the case where we drop out the database interface altogether, is there a benefit to keeping PHP? I couldn’t think of one.
Still thinking about this question.
I’m pretty certain that I’m a woman in technology.
Last night, I was using SAS on a virtual machine through a remote desktop connection to prepare data from the National Hospital Discharge Survey for use in examples of MANOVA and multinomial logistic regression.
Next week, I will start on a contract to completely re-do the PHP/ MySQL database for a client to bring it to something more secure and up to date.
Oh, and I also was reviewing my notes for the graduate courses in biostatistics and advanced multivariate statistics that I’m teaching this fall.
Pretty certain that by any standard – writing code, founding companies, graduate degrees, university appointment, successful Kickstarter – I am definitely a woman in tech/ STEM whatever the day’s buzzword.
I read SO many articles, blog posts, tweets about the need for women in tech, women-led start-ups, women entrepreneurs.
If you ask me, the U.S. Department of Agriculture is the greatest proponent of women in tech that there is, because they have actually put up money and funded us to do a prototype of an adventure game that teaches math.
When results from that were positive, they funded us again with a Phase II Small Business Innovation Research award to develop the games for commercialization.
I have written here before about the troubling nature of the Black Girls Code, Latina Girls Code emphasis that seems to completely overlook the grown women who are here now. I am NOT saying those aren’t good programs. I assume they are but I have no personal experience. What I am saying is pretty much what I said in January.
It seems to me that when people are looking at minorities or women to develop in their fields, they are much more interested in the hypothetical idea of that cute 11-year-old girl being a computer scientist some day than of that thirty-something competing with them for market share or jobs. If there are venture capitalists or conference organizers or others out there that are sincerely trying to promote WOMEN who code, not girls, I’ve never met any.
(Since then, I have met a couple of conference organizers.)
I suppose Ada Lovelace was cool – my two-year-old granddaughter has a shirt with her picture on it. Still, I don’t think a trending hashtag of #fuckyeahadalovelace did anything for me as a woman in tech.
You know what helped me as a woman in tech? Seed money from the USDA. You can see what we did with it here at our 7 Generation Games site.
One thing Sheryl Sandberg got right in her book, Lean In, was that women tend to be judged on their accomplishments where men are judged on their potential. Of course, you also don’t want to be “too old” to be an innovator so by the time women have those accomplishments, they are past their prime as entrepreneurs according to those VCs who believe that people over 30 are too old to do a start-up.
It’s hard for me to complain about my life when my morning starts out with reading technical books with lines like, “Figure 1 shows the sprite with the red and green blood particles for player and zombie”.
My point is that our company is in the situation we are in not because of any “help minorities code” program but because USDA and our backers on Kickstarter gave us cold, hard cash to develop our products.
Want to help women in tech? Back them on Kickstarter. Buy their products. Tweet about their products and companies to help their marketing. Invest in their companies.
USDA got it right.
I’m working on a section of a game that teaches fractions. If a player misses the question about where to meet up with the returning hunter, he or she gets sent to study. There is a movie that plays before this about needing to get back to the camp before dark.
Here is the question,
“The sisters begin to worry their brothers won’t make it back by dark. They start down the trail to meet them. They decide to stop and wait at the spot where their brothers will be 3/4 of the way back to camp. How far FROM the camp will the girls be?”
I used this question because I want students to think about a few ideas:
- Distance between two points can be thought of as a whole.
- If you are a/b distance FROM point X, the remaining distance TO point X is 1 – a/b . Of course, I don’t expect them to state it like that.
- 1/4= 2/8
- Number lines can be numbered in either direction. You can have 0 on the left or 0 on the right. The distance will be the same. The size of each interval will be the same.
These are kind of important ideas in math – equivalence, the arbitrary nature of labeling points on a line.
Students can click on GIVE ME A HINT, and a hints page pops up that explains, among other things, why you were wrong if you answered that the sisters would be 3/4 of the distance to the hunting grounds FROM the camp. If, even after reading the hints, (or if they skip the hints and just guess, we’re talking kids, after all) they get the problem wrong, the player is sent to watch a video clip explaining the problem, and then has to take a quiz to get back to the game.
SO … I had the thought instead of writing the quiz questions out of thin air, I might read what some more experienced teachers were giving to students in this grade as math problems. After all, I haven’t taught middle school math since the 1980s. I went to several sites, I even purchased some things like “One year of fifth-grade homework problems” etc.
When I looked at page after page of what students are being given as homework assignments, the only thing I could think was “Are you fucking kidding me? No wonder kids hate math.”
All of the homework was like this:
1/4 + 1/3 = ?
For FIFTY problems. That’s it! Then, the next day, it would be another fifty problems like this:
5/6 – 1/4 = ?
Okay, you need to learn to add and subtract fractions, but is that ALL you need to learn? Obviously not. How boring must it be to sit and just calculate answers to the same type of problem over and over? This stuff made me start to hate math and I LOVE math.
How can you possibly think that is teaching kids math? That’s like making them copy down all of the words in the dictionary and pretending you taught them literature.
Don’t even get me started on teaching statistics – wait, too late. I’m started. That is my rant for tomorrow.