It ought to be easier than this and perhaps I could have found an easier way if I had more patience than the average ant or very young infant. However, I don’t.
Here was the problem. I wanted control charts for two different variables, satisfaction with care, surveyed at discharge, and satisfaction with care 3 months after discharge.
The data was given in the form of the number of patients out of a sample of 500 who reported being unsatisfied. PROC SHEWHART does not have a WEIGHT statement. You could try using the WEIGHT statement in PROC MEANS but that won’t work. It will give you the correct means if you have the number unsatisfied (undisc = 1)  and the number satisfied (undisc =0) out of 500, but the incorrect standard deviation because the N will be 2, according to SAS.
So, here is what I did and it was not elegant but it did work.
  1. I created two data sets, named q4disc and q4disc3, keeping the month of discharge and the number dissatisfied at discharge and dissatisfied 3 months later, respectively.
  2. I read in the 3 values I was given, month of sample, number unsatisfied at discharge and number unsatisfied 3 months later.
  3. Now, I am going to create a data set of raw data based on the numbers I have. First, in a do loop, for as many as people said they were unsatisfied, I set the value of undisc (unsatisfied at discharge) to to 1 and output a record to the q4disc dataset.
  4. Next, in a do loop for 500- the number dissatisfied, I set undisc = 0 and output a record to the same dataset.
  5. Now, repeat steps 3 & 4 to create a data set of the values of people unhappy 3 months after discharge.
  6. Following the programming statements are the original data.

So, now, I have created two data sets of 6,000 records each with three variables. Doesn’t seem that efficient of a way to do it but now I have the data I need and it didn’t take long and doesn’t take up much space.

data q4disc (keep = undisc month) q4disc3 (keep = undisc3 month) ;
input month $ discunwt disc3unwt ;
Do I = 1 to discunwt ;
    undisc = 1 ;
    output q4disc ;
end ;
Do J = 1 to (500-discunwt) ;
   undisc = 0 ;
   output q4disc;
end ;
Do k = 1 to disc3unwt ;
   undisc3 = 0 ;
   output q4disc3 ;
End ;
Do x = 1 to (500 -disc3unwt) ;
  undisc3 = 1 ;
   output q4disc3;
end;
datalines ;
JAN 24 17
FEB 44 24
MAR 36 15
APR 18 8
MAY 16 11
JUN 19 7
JUL 17 11
AUG 18 9
SEP 27 10
OCT 26 15
NOV 29 12
DEC 26 11
;
RUN ;
proc shewhart data=WORK.Q4disc;
xschart undisc * month /;
run;
According to SAS

“The XSCHART statement creates and charts for subgroup means and standard deviations, which are used to analyze the central tendency and variability of a process.”

For the three months after discharge variable, just do another PROC SHEWHART with q4disc3 as the dataset and undisc3 as the measurement variable.

OR , once you have the dataset created, you can get the chart using SAS Studio by selecting the CONTROL CHARTS task

Control charts window with month as subgroup and undisc as measure

Either way will give you this result:

Control chart

Let me begin by acknowledging that anyone who can afford to fly to Trinidad and Tobago to learn more about the culture and ends up pitching a reality show is in the extreme of privileged and fortunate people on this planet.

I also realize that anyone who travels as much as I do is bound to have some bad experiences as the law of averages catches up with her.

This is not one of those rants about how Airline X sucks and I will never fly them again. First of all, I fly American a lot and they are usually quite good. Secondly, you are usually lying when you say that because in many cases, you have no choice. If I’m flying into Devils Lake, North Dakota, there is one airline and that’s it.

HOWEVER, since we are negotiating for said reality show to be filmed in Tobago, it’s likely that my family, my staff and will be flying into Port of Spain on American Airlines a lot in the near future, I want to offer them some specific advice for how not to suck.

1. Don’t lie. I don’t know when it became your corporate policy to lie, but I highly recommend you stop it. I even suspect lying is making you some money in the short-term. Stop it anyway. Two different American Airlines employees told people they would have to sleep on cots set up in an auditorium because there were no hotel rooms. They gave excuses like there were a lot of festivities in the city. They said they had been trying all day and there were just no hotel rooms. Several people in front of me went off to the auditorium unhappily. I said, “I don’t believe it. There is not one hotel room in the city of Miami? There is no fucking way I am sleeping on a cot. I don’t believe that it is legal for an airline to have your flight delayed by hours and then tell you to sleep on the floor with no compensation.” After I pushed the issue for quite a while, someone finally admitted that, yes, I could get a hotel, pay for it myself and get reimbursed through customer service. After I questioned repeatedly the truth of every hotel room in Miami being booked, they finally admitted that no, it was only the hotels with which American had contracts that were booked. It took me 30 seconds to find a nice hotel 4 miles away using the Travelocity app on my phone. Immediately, 2 other people in line did the same thing. Now, I’m sure that it saved American a lot of money that people slept on cots instead of nice hotel rooms that were not deeply discounted to American. Still, don’t lie to people. That’s bad.
2. Don’t waste people’s time. I don’t know at what point American ran out of those discounted hotel rooms but there were a lot of people in front of me and behind me getting the same story. We waited in line for over an hour. If they write slow, it would have taken 145 seconds for an American Airlines employee with a marker and a piece of cardboard to put up a sign saying: Out of rooms at contract hotels. Your choices are a) Wait in line for food vouchers and sleep on a cot in the auditorium or b) Get your own room and mail in receipts for reimbursement. Hell, they could have made an announcement. Airlines announce something every 3.3 seconds anyway. All the people who wanted to get their own room could have left at that point and not wasted their time. The people who did want to go for the cot could have saved waiting behind all those other people.
3. Take responsibility. It was 100% the airlines fault that we missed our flights. The flight from Miami to Port of Spain was hours late, so the flight back was hours late. Why did it become MY responsibility to find a hotel room, pay for the hotel room and write American Airlines to get paid back? Why did I have to wait in line for 2 hours in Trinidad to get my ticket re-booked and another hour in Miami to be told to sleep on a cot? Call in extra employees to work. YOU fucked up. Why should your customers who pay you have to wait in line for a total of 3-4 hours? That’s not right. Call in more staff to handle customers. Figure out how to pay for people’s hotels. Give out visa gift cards. You can buy them at fucking Wal-Mart, for God’s sake. As the gentleman in front of me said about himself, “I have money and I’m going to go stay at an expensive hotel, no thanks to you people. What about all of these other people who can barely afford to travel – you have older people, families with small children – they shouldn’t be sleeping in a gym. That’s not right.”
4. Don’t get self-righteous when your passengers are angry because you have lied to them, wasted their time and failed to take responsibility. When the American employee told me we would have to sleep on a cot in an auditorium, I told him, “There is no fucking way I’m sleeping in an auditorium. Are you fucking kidding me? I KNOW that the airline can’t have passengers delayed overnight and just say that’s too bad and I fucking GUARANTEE you that there are hotel rooms in Miami.” He told me not to swear and he threatened call security on the gentleman in front of me. If your job is to deny responsibility, waste people’s time and lie to them, don’t be surprised when they get mad at you.

On the other hand Travelocity and your app, you rock. Hotel Colonnade in Coral Gables, we will be back for more margaritas and we loved the family loft.

And American, you can do better. I’m counting on you. We have a lot of travel coming up for that reality show and me launching myself at the next lying bastard would make good reality TV but probably not look to good on my permanent record.

Oh, and by the way, I was supposed to get in last night and I’m still sitting on the plane waiting for a gate at LAX.

Perhaps you have watched the Socrata videos on how to do data visualization with government data sets and it is still not working for you. Here is a step by step example of answering a simple question.

 

Is the prevalence of alcohol use among youth higher in rural states than urban ones?

You can click on a link below to go directly to that step.

First, I went to this Chronic Disease and Health Promotion Data & Indicators site.

Second, I selected Chronic Disease Indicators for my health area of interest.

Third, I selected ALCOHOL – which brought me to the screen showing all the columns of data and a bunch of choices.

Fourth, I clicked  on FILTER on the right of the screen and then select a column to filter by.

select a filter
Chrome did not give me a scroll bar so the furthest option I could get was Topic. I switched over to Firefox and was able to get this menu where I selected Question and Alcohol use among youth. You have to type in the value that you want. Make sure it is spelled exactly the same as in the data set.

Fifth, since I wanted to compare urban and rural states, I clicked Add a New Filter Condition and then selected California, New York, North Dakota and Wyoming with LocationDesc as the filter condition. Make sure the box next to each condition you want is clicked on.

2secondfilterSixth, I looked at my data, saw there was no data for California and I was sad. Not every state participates in every data set.

Data withiout California7. So, I decided to compare urban, eastern states wth rural midwest/west and I selected New York, New Jersey, Massachusetts, Wyoming, North Dakota and Montana  All had data so I was good to go.

In case you were wondering,  I based my choice on the listing of states by urbanization , New Jersey is #2, MA 5 and NY 13
On the other extreme, Wyoming is 39, North Dakota is 42,  and Montana 47 so I thought this was a pretty good split.

8. I clicked on visualize on the right, selected Column as the type of chart, Location Desc as the label data, DataValueAlt as the data value, and there was my chart

Note: I could not select DataValue. My guess is that was a string variable. I had to select DataValuealt, which was the exact same value

4graph

9.Just to make it more obvious, I went in and sorted on data value, which caused the chart to be recreated automatically.

Sorting the table

You can see below the chart it created. It’s pretty clear that in these data there is no relationship between urbanization and alcohol use among youth.

graph

New York and New Jersey where the lowest and highest prevalence, respectively. I was hoping to see a pattern with more rural states higher, but it seemed to be pretty unrelated.

HOW TO DOWNLOAD THE DATA SETS FOR  ANALYSIS

Perhaps you would prefer to download the data set for import into some other tool, say, Excel or SAS. The first three steps are the same, into you find the data set you want.

This next step is not required , but the data sets can be pretty big, so I’d suggest filtering on at least one major variable first. For example, you can click the three rows next to any column, say, Question, and then select the question that interests you, say Binge Drinking.

selecting a filter

Next, click the EXPORT button at the top right of the screen. Select the format in which you want your file to be downloaded. That’s it!

download menu

 

Have kids? Know anyone who has kids? Like kids? Own a computer? Fish Lake will teach fractions and Native American history, with no whining and all for under ten bucks.

Buy our games

 

I wouldn’t normally consider Excel for analysis, but there are four reasons I’ll be using it sometimes for the next class I’m teaching. First of all, we start out with some pretty basic statistics, I’m not even sure I’d call them statistics, and Excel is good for that kind of stuff. Second, Excel now has data analysis tools available for the Mac – years ago, that was not the case. Since my students may have Mac or Windows, I need something that works on both.  Third, many of the assignments in the course I will be teaching use small data sets – and this is real life. If you are at a clinic, you don’t have 300,000,000 records.Four, the number of functions and ease of use of functions in Excel has increased over the years.

For example,

TRANSPOSE AN ARRAY IN EXCEL

Select all of the data you want and select COPY

Click on the cell where you want the data copied and select PASTE SPECIAL from the edit menu. Click the bottom right button next to TRANSPOSE and click OK. Voila. Data transposed.

PERFORMING A REGRESSION ANALYSIS

Once you have your data in columns (and if it isn’t, see TRANSPOSE above), you just need to

Excel add-ons window

  1. Add the Analysis Pack. You only need to do this once and it should be available with Excel forever more.  To do that, go to TOOLS and select EXCEL ADD-INS. Then click the box next to Analysis ToolPak and click OK.
  2.  Now, go to TOOLS, select DATA ANALYSIS and then pick REGRESSION ANALYSIS

Regression analysis menuYou just need to select the range for the Y variables, probably one column, select the range for the X variables, probably a column adjacent to it, and click OK. You may also select confidence limits, fit plots, residuals and more.

So, yeah, for simple analyses, Excel can be super-simple.

Believe it or not, this is what I do for fun. In my day job, I make video games that teach math and social studies.

You can check out the games we make here.

Wigwam

Working on some fun things  using  SAS Studio, so, expect a number of short posts over the next few days. Last time, I talked about the utilities and how easy it is to import an Excel file. Now let’s say maybe you are not aUnix person  and you have no idea how exactly to code a LIBNAME statement  that is not on Windows.  Never fear, it’s super easy.

Right click on the folder where you want to save your data set. From  the  menu that appears, select the last choice which is ‘properties’.

A window will come up that shows the name of your folder and its location, it’s easy to spot because it’s right next to the word Location. It will look something like this:

/home/your_name/data_analysis_examples

to save your data you have uploaded an Excel file and imported into  SAS, remember that the files were saved in the work directory and named import, import 1 etc.If I wanted to  sort those data sets and then merge them together into a permanent data set, I’d do it in the exact same way as if I was using Windows. The only thing different is the LIBNAME statement, as you can see below.

LIBNAME in “/home/your_name/data_analysis_examples”;

Proc sort data = work.import;
By username;

Proc sort data = work.import1;
By username;
data in.crossroads ;
merge work.import work.import1;
By username;
run;

If, later on, I want to use that data set in a program, again I would do it exactly the same as in Windows and the only thing different would be my LIBNAME.

 

LIBNAME in “/home/your_name/data_analysis_examples”;

 

Proc means data = in.Crossroads;

Run;

Completely random fact, unrelated to SAS studio, or maybe it is related,  I hurt my arm again, so I have been writing my SAS programs using Dragon voice recognition software.  If you are going to use SAS studio on a Mac, you should be aware that Dragon does not work on Firefox on the Mac so open up Chrome if you want to use voice recognition software, or at least the software from Dragon. This has nothing to do with SAS specifically.

 Support My Day Job!  Buy Games That Teach Social Studies And Math And Have Fun! (You can even get Making Camp for free! 

Buy our games

 

It’s been about a year since I last looked at SAS Studio much  –

OKAY, LISTEN UP PEOPLE

In my previous life, I taught for years at a small liberal arts college, with under 2,000 students. I also taught at a tribal community college with less than 500 students. In neither of those situations did we have the funding to pay for expensive software. SAS Studio is FREE. I could have really used this when I was teaching at those small schools. Check it out.

students

So, it’s free, but I don’t teach that often because I have a day job as president of The Julia Group where clients want me to do some much stuff we quit taking new clients years ago and also president of 7 Generation Games where they want me to do more stuff.

The last class I taught, we used SAS on a remote desktop – which I liked a lot. So, yes, no SAS Studio for me for a while.

In case, like me, you are more a programming type and haven’t been too pointy-clicky, perhaps you missed the TASKS AND UTILITIES. Well, don’t.

Let’s say you want to import a file from Excel into SAS. First, upload it by clicking on the folder where you want it stored and then clicking the upload button at the top left of your screen.

Look to the bottom left of your screen and you will see this. Well, you’ll see the Tasks and Utilities anyway, the stuff above it is files for class examples.

Tasks and utilities menu

Click on the arrow next to Tasks and Utilities and you’ll find all kinds of cool stuff.  Click the arrow next to utilities and pick IMPORT DATA

upload window

Drag the file you uploaded into the window on the right and, voila!

There you go, your Excel file is imported into SAS. You can see the code in the CODE window. DON’T FORGET TO CLICK THE LITTLE RUNNING GUY AT THE TOP OF YOUR SCREEN TO RUN THIS.

Note that the file is named WORK.IMPORT because you’ll need that name for the next task, but that’s next time because I have to go back to work.

/* Generated Code (IMPORT) */
/* Source File: testit.xlsx */
/* Source Path: /home/annmaria.demars/homework */
/* Code generated on: 2/6/17, 11:27 PM */

%web_drop_table(WORK.IMPORT);

FILENAME REFFILE ‘/home/annmaria.demars/homework/testit.xlsx’;

PROC IMPORT DATAFILE=REFFILE
DBMS=XLSX
OUT=WORK.IMPORT;
GETNAMES=YES;
RUN;

PROC CONTENTS DATA=WORK.IMPORT; RUN;

%web_open_table(WORK.IMPORT);

 

Screen Shot 2017-02-06 at 11.36.53 PM

SAS nicely runs the PROC CONTENTS, too, so you end up with a table telling you the contents of your new data set.

Once you have your data imported, you can use the TASKS menu to complete (what else) statistical tasks. I wrote about those in some other posts below:

My point is, there is a lot of stuff under that little tab and you should check it out. Also, if you are a small school, SAS Studio is an awesome resource you can get for free and I bet you could use it.


Support my day job! Learn about Ojibwe history and culture. Practice multiplication and division

FREE GAME FOR iPad or Android tablets

wig wam in snow

Click over here to find links to Making Camp in the App Store or on Google Play. Yes, it’s free.

I have several ideas for blog posts I haven’t written. One is for this blog on simple statistics that answer useful questions. Three others are for my 7 Generation Games blog, on various features and functions in Javascript.

I haven’t written any of them.

Instead, I wrote an annual report that was due for one project, worked on a grant proposal due soon and a final report for another project (all of these things bring me money) , attended a few meetings and worked A LOT on development of a new mobile game to teach decision-making and a major update of Spirit Lake, which teaches multiplication and division.

While I was doing this, I did not do nearly enough on promoting our games like Fish Lake

Fishes

(awesome and available everywhere – Steam, Amazon and  our website)

or Making Camp (free for iPad and Android).

Wigwam

You know, it doesn’t really matter.  The fact is that every day you will end up with some things you didn’t get around to doing. There is never going to be a day when I walk into my office and say,

“That’s it. I have completed all the programming. Yes, done. There is no more code to be written. Marketing, finished. Everyone has been completely managed. All budgets are completed forever. Now,  I must ride off into the sunset on my unicorn.”

unicorn

It wasn’t that I wasn’t interested in writing those blog posts – I was  – but at the moment, it seemed more pressing, potentially profitable and/or , to be honest, fun, to do those other things.

The older I get, the more of a long view I have and I can see that those grants get funded, contracts get done, articles get published and if you don’t get it done today, at the end of the day, there’s another day, because that’s how time works.

As long as you are moving forward, you are making progress.

I can hear you saying, though, all the way here in Santa Monica,

Sure, that’s easy for you to say, but what about those things you definitely do not want to do, like filling out your tax returns? If you just do your happy-happy programming and don’t send in your 1040 then you get massive fines, go to jail and have other bad unicorn-less things happening. What about THAT?

I have two answers to that.

First of all, it is possible that you can make enough money from your other endeavors that you can pay someone to do that stuff. I am not entirely sure if 1040 is the form or the thing you put in your oil – I think that’s WD40. Anyway, 15 years or so ago, I hired an accountant whose job it is to keep me out of white collar prison. She is batting 1,000 so far.

Secondly, for those things that you do have to do yourself, like renew your drivers license or attend some boring-ass required training on sexual harassment or email that person you really meant to answer or read that article or write that blog, really, the world will not implode if you didn’t get it done today.

Too often, we make ourselves crazy acting as if whatever it was we didn’t get done today was THE crucial element that would determine our success. Trust me, it’s not.

As for me, since it is actually tomorrow, since it is almost 1 am, I’m going to have a glass of Chardonnay, read something with zero redeeming social or educational value and not worry about what I didn’t get done today at all.

Bottle of wine

Well, it’s been a minute. In fact, it’s been over two weeks. I started this blog NINE YEARS AGO. That is pretty amazing. According to some guy named Patrick, who cited ‘research’ the average blog lasts only 100 days.

Actually, when I backtracked this statistic, I came to an Atlantic article that said the average WEB PAGE sticks around for 100 days, which seems awfully short to me.

I recall years ago reading that the average blogger persisted for about 31 days. That statistic only stuck with me because of the comment that the average blog has the lifespan of a fruit fly. I probably read that 9 years ago. Now, when I searched to find that statistic, all I came up with was blogs on the life expectancy of a fruit fly.

I did search in the university library database for a bit to find the average blogger persistence but all I found was some blogs on persistence. Humph.

When I started this blog, I wrote a lot about SAS, data analysis and statistics. I also wrote a bit about math and educational games. I am still really interested in all of those things and have far more idea for blog topics than time to write them. One topic students always struggle with, for example, is finding the area under a normal curve between two z-scores.

However, there is not enough time in the day to do everything I want to do.

If you wonder what I have been up to, lately I’ve been writing a series of posts on making JavaScript games using impact.js

You can read the first six posts starting here, and tomorrow, time permitting, I’m writing the seventh one on making a character.

I didn’t get to it today because I just got back to work and had to debug some PHP scripts and start wading through > 2,600 emails. Almost down to 1,000.

Back to work I go.

Still, I’m pretty happy with everything I am doing and the new year is starting out well. How about you? How is your 2017 shaping up?

You can check out the games we make here.

Wigwam

I’ve hardly blogged, answered emails or talked to anyone these past few months. We just received funding for two games and I have been working night and day, crisscrossing the country recording voices, demonstrating prototypes, contracting for art and animation, reviewing designs and fixing bugs. 

termite

Ah, the bugs. We have arrived at the level of complexity in our games where improving one part almost inevitably breaks some other part. After months of revision, Fish Lake will be released on Steam TODAY (not a Steam member ? You can get it on our website, runs on Mac or Windows).

Is it finished?

Fish lake splash screen

Well, as a wise man once told me, games are never finished, they’re just abandoned. There are a thousand enhancements we’re still planning, but we have fixed all of the bugs, tested it a hundred times, passed review, for sale on Amazon .

Now what?

Last night, I read Michael Raethel’s book “It only hurts when I hit enter”. It’s a humor book about life as a programmer. I can guarantee that if you have spent years as a programmer you will find it funny – and familiar. I had to read aloud to my husband the prank on the guy who always claimed credit for solutions from the consultants. Maybe you wouldn’t find it funny if you hadn’t been there, but we had both been there. Everyone has worked with a Henry.

Reading the book made me realize how LONG I have been at this whole program endeavor. When he mentioned COBOL, I smiled and said to myself, “I remember that”.

The description of the green and white lined computer paper stacked up everywhere was another memory that took me way, way back.

Then, there’s yesterday when I was writing JavaScript code to have a math problem replaced with an animated rabbit when you get the right answer.

Emilia and CalIt’s been a good run. I’m inside, dry and warm in my nice place by the beach while it pours down rain. If you have always had a house with heat and a roof that doesn’t leak  then you probably don’t appreciate that in the same way. We have plenty to eat and more toys than we need.

Still, there are times when it feels as if I have done all of this before. When I look in a filing cabinet or on my computer for a contract for consulting services, I’ll find a dozen more contracts, going back a dozen years. Looking for a review of a grant, I’ll come across four others, three of them funded, one of them not.

Don’t get me wrong, I like programming and I like research. It’s pretty amazing that I can get paid to sit and type numbers into a computer and even more amazing that those numbers can turn into a game that kids play and it raises their math scores.

And yet … In our nice building by the beach, there is a white-haired man with an office on the 12th floor, overlooking the ocean, filled with very nice furniture. He’s a very successful attorney, I hear, and he is in the office every day unless he’s in court. If he isn’t 90 years old, he’s damn close. 

Now, some people say,

“That’s just what I want when I’m 90 years old – to be in full possession of my faculties, still be in demand, productive.”

That is NOT what I want. Well, I want the possession of my faculties part, I mean, I don’t want to be drooling on myself. However, I don’t want to drop dead at my desk after having killed one last bug. 

I’m not sure what I do want but I’m sure I DON’T want to keep running like a hamster on a wheel. 

An angel investor once asked me what I planned to do once I got 7 Generation Games where I wanted it to be, played by millions of people. I said I might just sell the company and go on to whatever the next chapter in my life might be. He didn’t like that answer. He said they were looking for people who were going to go off and start another company after that. He said they were looking for “serial entrepreneurs”. That’s a whole ‘nother post , like, if you made a billion dollars the first time, you wouldn’t need to be a serial entrepreneur.

Frankly, I thought mine was a perfectly fine answer. The thing about a series is that at some point it ends. Yes, Dennis, even The Simpsons will some day end.

 

Where we left off, the reliability was unacceptably low for our measure to assess students knowledge of sad iconmultiplication, division and other third and fourth grade math standards. We were sad.

One person, whose picture I have replaced with the mother from our game, Spirit Lake, so she can remain anonymous, said to me:

But there is nothing we can do about it, right?I mean, how can you stop kids from guessing?

mother from game

This was the wrong question. What we know about the measure could be summarized as this:

  1. Students in many low-performing schools were even further below grade level than we or the staff in their districts had anticipated. This is known as new and useful knowledge, because it helps to develop appropriate educational technology for these students. (Thanks to USDA Small Business Innovation Research funds for enabling this research.) 
  2. Because students did not know many of the answers, they often guessed at the correct answer.
  3. Because the questions were multiple choice, usually A-D, the students had a 25% probability of getting the correct answer just by chance, interjecting a significant amount of error when nearly all of the students were just guessing on the more difficult items.
  4. Three-fourths of the test items were below the fifth-grade level. In other words, if you had only gotten correct the answers three years below your grade level, the average seventh-grader should have scored 75% – generally, a C.

There are actually two ways to address this and we did both of them. The first is to give the test to students who are more likely to know the answers so less guessing occurs. We did this, administering the test to an additional 376 students in low-performing schools in grades four through eight. While the test scores were significantly higher (Mean of 53% as opposed to mean of 37% for the younger students) they were still low. The larger sample had a much higher reliability of 87. Hopefully, you remember from your basic statistics that restriction of range attenuates  the correlation. By increasing the range of scores, we increased our reliability.

The second thing we did was remove the probability of guessing correctly by changing almost all of the multiple choice questions into open-ended ones. There were a few where this was not possible, such as which of four graphs shows students liked eggs more than bacon . We administered this test to 140 seventh-graders. The reliability, again was much higher: .86

However, did we really solve the problem? After all, these students also were more likely to know (or at least, think they knew, but that’s another blog) the answer. The mean went up from 37% to 46%. 

To see whether the change in item type was effective for lower performing students, we selected out a sub-sample of third and fourth-graders from the second wave of testing. With this sample, we were able to see that reliability did improve substantially from .57 to. 71 . However, when we removed four outliers (students who received a score of 0), reliability dropped back down to .47.

What does this tell us? Depressingly, and this is a subject for a whole bunch of posts, that a test at or near their stated ‘grade level’ is going to have a floor effect for the average student in a low-performing school. That is, most of the students are going to score near the bottom.

It also tells us that curriculum needs to start AT LEAST two or three years below the students’ ostensible grade level so that they can be taught the prerequisite math skills they don’t know. This, too, is the subject for a lot of blog posts. 

If you’re a teacher (or parent) and you’d like students to take the test for practice, you can see it here

—-

For schools who use our games, we provide automated scoring and data analysis. If you are one of those schools and you’d like a report generated for your school, just let us know. There is no additional charge.

Next Page →