Before we went to Arlington, VA to get our hands on the National Indian Education Study , my colleague, Dr. Erich Longie, hypothesized that schools that had more cultural activities would have lower academic achievement. In addition to being an old friend, Dr. Longie is president of Spirit Lake Consulting, Inc. , a published author of research on American Indian education and did a dissertation on the impact of absenteeism on academic achievement in tribal schools. Oh and he was the first member of the Spirit Lake Dakota Nation to earn a doctorate. Suffice it to say, he’s got a truckload of personal, professional and academic credentials that give his opinions on the subject above-average weight.

Spirit Lake Consulting, Inc. LogoI asked Erich why he thought this was the case, since I would expect students who had schools that were more consistent with their culture would have higher achievement. Yes, I read Urie Bronfenbrenner’s Ecological Psychology. In short, he thought that there were micro-systems, like families, meso-systems, like schools and exo-systems, like countries, and the more consistency among those different systems that a person operated in, the better the outcome. So, if a child had the same language, values and expectations at home,  school and the larger society, he or she should do well. This seems to be the opposite of what Erich was saying.

My first thought was that schools that had classes in Native American languages, field trips to Native American historical sites, classes studying a specific tribe’s culture and similar activities were far more likely to be on reservations and in towns adjacent to reservations with low income, high minority enrollment and negative school factors, such as high teacher turnover. I thought that any negative relationship with cultural variables would be due to those factors, kind of like babies who get early intervention under one year of age tend to do worse on developmental measures. It’s not that early intervention hurts kids, it’s that if we can tell you need help when you are three months old, you probably have some severe needs. It’s a selection bias.

Erich nodded and said that he agreed that was some of it but he thought the cultural activities themselves were a negative factor. He explained that most cultural programs he’d seen were not included as part of an academic curriculum. The teachers weren’t held to the same standards. They weren’t expected to have lesson plans or academic objectives. Their programs were just offered to pacify complaints by parents or tribal board members. There are a finite number of hours in a school year. If kids, and teachers, took hours out to watch movies about treaty rights, learn words in their native language or visit the site of a former Indian boarding school, those are hours not devoted to studying mathematics and reading. Important point here – Erich wasn’t arguing that those activities COULD NOT be integrated into the academic curriculum, but rather that they ARE NOT.

We drew the analogy to an early literacy program I evaluated years ago. The program did a lot of nice things, like had Avon parties for the young mothers, took them and their children to an amusement park. When I asked about some of these activities, the director told me that these mothers had very low self-esteem and activities like a party where they were all made up to look good was positive for their self-esteem. That may or may not be true, but what was true is that the literacy measures for children and parents did not go up because there wasn’t much in the way of literacy activities going on. Not surprisingly (to me, anyway),  when the project got a new director who changed the activities to emphasize reading – an excellent tutor for adult literacy, parent-child reading activities, readings by local authors of children’s books – literacy scores started to go up.

So, enough of speculating, we dived into the data. To do the analyses, I used the AM statistical software , a free package that I blogged about earlier this week.

CULTURE SCALE We used the 2007 NIES data and a cultural scale that consisted of 13 items such as how often the child’s native language was used, how often they attended ceremonies, how often they used media in school that portrayed Native Americans,  how often they went on field trips to Native American sites. Cronbach’s alpha internal consistency reliability coefficient was .83. The variables were scored 1= Yes, 2 = No, so a HIGHER score means students have FEWER cultural activities.

SCHOOL ENVIRONMENT SCALE We used several items that we thought measured school quality, including the percentage of teachers absent on an average day, the percentage of teachers absent on an average day, full-time teachers who left before the end of the school year. The reliability of this scale was .69

We tried a lot of items we THOUGHT should combine to form a home environment scale, like having a computer at home, subscribing to a magazine, mother’s education, owning an encyclopedia, how much time the student spent on homework. The best alpha we could do was .58 and when we tried to replicate that with the 2009 data it was even worse, less than .50. So, instead of a home environment scale, we used mother’s education. The final variable we used was the number of days the student was absent in a month.

The dependent variable was the five plausible values for the overall mathematics score at eighth grade. Here is what we found:

…………………      z-score                 p

CULTURE            8.215             .001

SCHOOL             -4.766            .001

DAYS ABSENT   – 4.755           .001

MOM’S EDUC         5.217         .001

Okay, so everything was highly significant. With culture, the more students said, NO, they did not use their tribal language, attend ceremonies, have school field trips to cultural sites, etc. the HIGHER their mathematics achievement.  As we would expect, the worse the school environment – the more students were absent on a given day, the more teachers were absent, the higher turnover of teachers – the LOWER their mathematics achievement. The more days a student was absent, the LOWER achievement and the higher mother’s education, the HIGHER achievement.

We re-did these analyses with the 2009 data, although it was not an exact replication, because I added  a few more cultural and school items, we again found that the more cultural experiences, the LOWER students’ mathematics achievement. We still couldn’t get a reliable home environment measures, but remember we only had less than two days to work with the data and it was restricted use so they even made us turn the DVD in to the monitors when we went to the bathroom. If I had time, I would work on getting more reliable measures of home and school environment and controlling for the variance due to these before entering cultural factors. I’d also look to see if there were schools that had a high cultural component and high achievement and compare those to schools with low achievement.

BeadworkWith the limited time we had with the data, I would say we did find support for the hypothesis that cultural activities are not being included in schools in a way that supports the academic goals. That isn’t to say that we CAN’T have a cultural program that supports the academic curriculum, but it does tend to support Dr. Longie’s contention that we DON’T.

 

Any time you add another layer of complexity you better have a damn good reason.

I’m often skeptical of proponents of both Item Response Theory and multiple imputation procedures, not because either IRT or MI is a bad thing in itself but because its inclusion makes data analysis and reporting more complicated. At the National Center for Education Statistics (NCES) seminar this week, Dr. Emmanuel Sikali gave some damn good reasons why the National Assessment of Educational Progress (NAEP) makes use of both IRT and multiple imputation.

In very brief, he noted that it is only fair when conducting an assessment to measure what is taught in the curriculum. Your good old basic content validity. However, to do that for an area like mathematics or reading would take a very large number of questions. We not only want students to answer the test questions but we also want to get information like if they have a computer at home, how many minutes a night they spend on math homework and a few dozen similar questions. We can’t really expect schools to volunteer to have students take a four-hour long, 250 item test for each subject. We can’t expect students to be willing to spend that much time on testing without getting tired or sick of it. (The tests are done in grade 4 and grade 8. I can already hear the world’s most spoiled 13-year-old saying, “This stupid test sucks. I’m not doing it.”)

Unhappy camperSO … how do you fairly assess the curriculum without giving students a zillion item test? You create say, four different tests and give each student 1/4 of a zillion items. Then, based on the answers students gave for the questions they did receive, you estimate the scores they would have gotten on the other items. Because students randomly get one version of a test, data really ARE missing at random. Of course, you don’t want to treat this data you imputed the same as the items the students actually answered because there is some uncertainty as to whether your estimate is correct. So, you do this imputation multiple times. Hence the name. You can read a pretty nice introduction to multiple imputation here by Joseph Schafer at the Penn State University Methodology Center. They provide a ton of useful information on their site. Between them being funded by DHHS and the Dept of Ed funding NCES, I have decided that I will not become a Republican this week because I have proof that the government DOES do some things right.

Okay, so we are convinced of the greater goodness of multiple imputation. Now what do we do with those plausible values? Also, I should throw in that students being sampled within schools, you need to account for the cluster in sampling. Oh, and it is not a simple random sample, you need to include student weights. You could use SAS. If you pay for the complex samples module, you can use SPSS.

The Department of Education funded development of AM Statistical Software (no, it was not named after me). You can download it for free and it is unbelievably simple to use. It is all pointing and clicking. As far as I know, it only runs on Windows. I used an earlier version that only imported SPSS datasets. The AM website says they now import SAS datasets also. It’s no problem if you have the older version. I just did the creation of factors, recoding, etc. that I wanted in SAS, then exported it as an SPSS file.

Importing your data is simple – FILE > Import Data > SPSS type then pick the file.
Analysis is also super simple. More on that tomorrow, though.

I’ve been gone for a week and the cat litter, guinea pig cage, frog tank , carpet and my clothes all need to be cleaned.

← Previous Page