baby mashing cake

Kappa is a useful measure of agreement between two raters. Say you have two radiologists looking at X-rays, rating them as normal or abnormal and you want to get a quantitative measure of how well they agree. Kappa is your go-to coefficient.

How do you compute it? Well, personally, I use SAS because this is the year 2015 and we have computers.

Let’s take this table, where 100 X rays were rated by two different raters as an example:

Rating by   Physician 1

————-Abnormal   |  Normal

Physician 2
————————————–

Abnormal        40             20

Normal             10            30

 

So ….. the first physician rated 60 X-rays as Abnormal. Of those 60, the second physician rated 40 abnormal and 20 normal, and so on.
If you received the data as a SAS data set like this, with an abnormal rating = 1 and normal = 0, then life is easy and you can just do the PROC FREQ.

 

Rater1    Rater2

1                1
1                 1

and so for 50 lines.

 

However, I very often get not an actual data set but a table like the one above. In this case, it is  still relatively simple to code

DATA compk ;

INPUT rater1 rater2 nums ;

DATALINES ;

1 1 40
1 0 20
0 1 10
0 0 30
;

 

So, there were 40 x-rays coded as abnormal by both rater1 and rater2.  When rater1 = 1 (abnormal) and rater2 = 0 (normal), there were 20,  and so on.

The next part is easy

PROC FREQ DATA = compk ;

TABLES rater1*rater2/ AGREE ;

WEIGHT nums ;

 

That’s it.  The WEIGHT statement is necessary in this case because I did not have 100 individual records, I just had a table, so the WEIGHT variable gives the number in each category.

This will work fine for a 2 x 2 table. If you have a table that is more than 2 x 2, at the end, you can add the statement

TEST WTKAP ;

This will give you the weighted Kappa coefficient. If you include this with a 2 x2 table nothing happens because the weighted kappa coefficient and the simple Kappa coefficient are the same in this case.

See, I told you it was simple.

 

As you have no doubt cleverly deduced from the title, this is the second installment in my series, “Mama AnnMaria’s Advice on How Not to Get Your Ass Fired.”

In Part 1, I explained why employees who (in their view) never make mistakes end up getting their asses fired.

Today’s lesson is:

Understand that your boss doesn’t have to explain everything to you.

Let’s say that you believe that you are doing stellar work. Let’s further assume that the work you are doing is both something objectively difficult –  3-D modeling, software development, teaching probability theory – and that you believe to be important.

You are doing a great job and are highly skilled but they are not paying you the salary you believe you deserve

As I see it, there are four possibilities.

  1. The organization cannot afford to pay you more than you are getting.
  2. Your boss is a moron.
  3. They are hoping you will quit.
  4. The work you are doing is not of the quality or importance that you believe it is.

Let’s consider possibility number one. If that is the case and everything you assume about your performance is true, then the organization should be trying to make it up to you in other ways. I once worked for a non-profit where my boss couldn’t offer me a raise but cut my hours 25% and gave me the same pay. They might give you flexible hours, a nice office, a travel budget, an impressive-sounding title. None of those things are the same as cash but they at least show the organization is making an effort.

Let’s assume that they do none of that and you are treated like a peon. They even make you wear that Hotdog on a Stick uniform even though you don’t work at Hotdog on a Stick. There remain the last three possibilities.

Why would your boss want you to quit? In my experience, the two most common reasons are:

  • You are a pain in the ass
  • The work you are doing is not of the quality or importance that you believe it is.

There are things your boss may not tell you for very good reasons, say,  non-disclosure agreements.

It is possible that your work is not really all that important.

Yes, you have done a fabulous job creating the company website. However, the company is in merger negotiations after which the website will be taken down and all of the products listed under Acme Giant Corporation, Inc. You may be fabulous at teaching probability theory but the university is discontinuing its Statistics major due to lack of students enrolling in Northern Alaska On a Mountaintop University.

They also may not be telling you that they are hoping you will quit to avoid the unpleasantness of having to fire you.

Why on earth would anyone ever want to fire you, other than that your work isn’t as important as you think?

 

Another possibility, as much as this may cause you pain, is that your work isn’t as good as you think it is

fish

Perhaps you suffer from the Big Fish in a Small Pond Syndrome. Maybe you know more about Windows 8 than any of your friends. Let us at least entertain the notion that you may have particularly stupid friends. Yes, you were the IT expert at your old job but there were only six people in that company and they were in the janitorial business. Now you work IT at Microsoft and they are not impressed that you know what a printer driver is.

You should be aware of the Performance: Pain in the Ass Theorem

To whit, there is no performance so great that it cannot be exceeded by the pain in the ass of putting up with you. (Note that even Steve Jobs got fired from Apple).

Since it is past 2 am in North Dakota, from which I returned at midnight on Friday, and I have not one, but two meetings in the morning, further explanation of this highly important theorem in the field of Not-Getting-Your-Ass-Fired will have to wait until another post.

 

Listen my dears and you will learn the difference between incidence and prevalence and why effective treatment for a disease may mean that more people have it.

Prevalence is the number of people in the population who have a disease. It is computed as

Number of people with disease   X 1,000
Number in population

As I mentioned before, there is nothing magical about 1,000 and you could compute prevalence per 100, per 100,000 or per million.

The important point is that prevalence is the number who HAVE the disease.

Below, you see a graph of prevalence and incidence of HIV in the U.S. from 1997- 2006.

 

Graph of HIV prevalence and incidence

 

The green line is prevalence, the number who have the disease. Clearly, prevalence went up after 1995.

So, did the AIDS epidemic get worse? No, actually, it stayed the same in one sense and got better in another.

The red dotted line is incidence. If you define epidemic, as occurrence of a disease in excess of normal frequency, then there has been no change and no evidence of a worsening epidemic, even though more people HAVE the disease. How can that be?

INCIDENCE is the number of new cases occurring, you compute it like this

Number of new cases during a period of time      x 1,000
Number in population at risk during period

Prevalence is affected by both incidence and duration. If a lot of people get the disease but the duration is short the prevalence won’t be as high as it would be if the duration was very long. That’s one reason you’ll find the prevalence of diabetes is higher than, say, chicken pox. Even though a lot of people get chicken pox, it doesn’t last very long. In contrast, once you have diabetes, you have it for years.

Another reason duration could be low is that mortality is high. If people die of a disease at a high rate, within a short period of time, prevalence would be low, but this would generally not be considered a good thing. Once effective treatment is found and people live longer, the prevalence would go UP, which is exactly what happened with HIV. In 1997, there was a 40% decline in AIDS deaths as a result of new, effective, anti-retroviral drugs. That lower death rate has been maintained so year after year the number of people who have HIV has increased even though the incidence has been pretty stable.

Diabetes is another example of a disease that has been affected by effective treatment. A friend of mine told me this story:

An elderly patient was against taking the medication he had been prescribed for his diabetes and high blood pressure. He demanded of his physician.

They didn’t have all of this medication back in the old days. What did they do then, huh?

To which Jake politely replied,

Well, sir, they died.

However, effective treatment is only one reason that diabetes prevalence is rising in modern times. Remember prevalence is affected both by incidence and duration. Well, the incidence of diabetes has been increasing dramatically.

Since it is nearly 3 am in North Dakota though, and I am flying home today, that will have to be a post for another day.

 

You’d think the ultimate example of simplicity in measurement would be mortality rate.

dead guy

Dead guy

Count up the dead people  – they aren’t hard to catch. Divide by the total number of people you had when you started.

Done.

It’s not super complicated but it is slightly more complicated than that.

First of all, how do you figure the number of people in the population? The population changes all of the time. People are born, people die. So to compute annual mortality rate you use this formula

Total number of deaths from all causes in 1 year   X   1,000
Number of persons in the population at midyear

(You can also use per 10,000,  100,000  or million. There is nothing magic about the number 1,000).

You also may want to compute mortality for certain groups, for example, by age. You would then calculate the age-specific mortality rate  For example, if we want to know the mortality rate of children aged 15 and under it would be

Number of deaths from all causes in 1 year of children under 16  x  1,000
Number of children under 16 in the population at midyear

There is also cause-specific mortality rate

Number of deaths from specific disease in one year    x 1,000
Number of persons in the population at midyear

CASE-FATALITY is a very different thing than mortality rate. Case-fatality is

Number of individuals dying during a period after disease onset X 100
Number of individuals with the disease

For example, the annual mortality rate from measles in 1971-1975 was .17 per million

During that same period, the case fatality rate was about .1% or  .98 per 1,000.

Think about that. One person out of 1,000 who got measles died.

After a measles epidemic in 1989-91, increased vaccination rates resulted in a dramatic drop in measles mortality rates.

Unfortunately, measles are making a comeback in the United States due to stupid bastards who don’t vaccinate their children. Whether this will translate to an increase in mortality rate remains to be seen. With only 644 cases in 2014, there hasn’t been a reported death – yet.

What’s it like starting a company with your spouse?

I get this question fairly often.

Several times now, I’ve heard investors say they don’t like to invest in couples, while in the same breath saying that they like to invest in companies with co-founders who have known each other a long time. So … starting a company with your high school best friend, good. Starting a business with your spouse, bad.

Me and The Invisible Developer

This makes me very curious about the types of marital and family relationships these investors have. I’m well aware that many marriages end in divorce and possibly many families that stay together are dysfunctional. However, it’s also true that family businesses are a staple of the American economy.

I can think of reasons why working with your family can be a disadvantage in growing a business – the primary ones being nepotism – promoting relatives over more qualified applicants, and difficulty separating work and family relationships.
However, I think there can be significant advantages, especially, as in our case, when we have been together a long time.

  • We know each other’s strengths and weaknesses. The Invisible Developer is great with software but he hates meetings. He is never going to pitch – ever. On the other hand, if a new language, Swaboomi, is invented next week and it would be a significant benefit to our business, he will be writing excellent Swaboomi code before anyone else you know.
  • We have learned to work together as a team. My husband does the lion’s share of his debugging in the Unity editor on a top of the line Mac. Last week, I spent 50 hours going through our latest game on Windows, on a cheap computer with crummy wireless, playing the game over from beginning to end. Bugs that might have slipped through the cracks, I picked up and fixed.
  • We can communicate with each other without getting emotional because we are on the same team. Not all couples can do this but you have to understand that I am married to the calmest person on earth. So, I can say, “Look, it’s 4 am and we have spent the last year working on this game. I know we want it to be perfect but unless you find a bug that causes the computer to turn into a dinosaur and eat the player, we are releasing this public beta tomorrow.”
  • We have commitment. When starting a company, losing a key team member is the kiss of death. The sole founder of another startup in our building (who doesn’t know we are married) tells me every time he sees me that he is  going to lure The Invisible Developer away as a co-founder. Since his fringe benefits at 7 Generation Games include having sex with him and raising his children, I’m not too worried.
  • We’re used to pulling our weight. I hate audio editing, but I do it. Dennis doesn’t like design meetings, but he does it. It’s like in a good marriage. When something needs to be done, there’s no saying, “That’s not my job.”
  • We don’t have problems at home with a spouse because of all of the hours put in on the job. In fact, it is appreciated and understood because we are both pursuing the same goal.
  • Time. Every investor would like to think the team is thinking about their company 24-7. When we think of a new game engine, JavaScript library or  idea for a function that could really expedite development, whether it is during dinner, as we’re having our first cup of coffee or at 10 pm, there is someone else right there eager to hash it out.

Of course some couples have dysfunctional relationships and that can be a disadvantage. It seems to me that a couple who would have enough shared interests to start a business together are more likely to be the type who have commitment and good communication. Why would investors assume that all couples starting a business are the other type? Do they all have dysfunctional families themselves?

Our Project Manager, Jessica,  made the very insightful comment at lunch the other day,

No one cares how hard it was for you to make. When people are looking to buy your product, all they want to know is what it will do for them.

That young woman has a bright future in marketing. Unfortunately for those who read this blog, I do not, so I am going to tell you how hard it is to make that last push to the finish line.

tired eyes

I quit counting the number of hours I worked this week when I got to 80. I’m sure The Invisible Developer had put in even more, because many nights (mornings?) I have gone to bed at 2 a.m. and when I wake up and check the latest build in the morning I find it was put up at 5 or 6 that morning. There hasn’t been much blogging going on lately and I only have a bit of a minute now because I’m waiting to get the latest latest latest build so that I can make the Windows installer.

I’ve blogged before on the great value I place on “details” people and this week is a prime example of the importance of details.

You’d think that down to and past the wire – the last build of the game was supposed to be today and we have negative 68 minutes left in today – that we would be moving forward pretty quickly. Um, not so much.

At the beginning of development, you can easily find the problems – the question is what fraction of the fish are over one foot long when you caught 125 fish last summer and 25 were over a foot long. The correct answer is 1/5.  However, 25/125 is also a correct answer, as is 5/25 . Finding those problems is easy. You can check the answer while you are creating the pages, have it write to the console the correct answer, step through the logic. No problem.

Same thing with playing the 3-D part of the game. If you are at the part where you are supposed to be spearing the fish and there is no spear, then it is an easy enough fix.

HOWEVER, now we are supposedly at the end. So…

  1. We make a version of the build for Mac and another for Windows.
  2. We zip the Windows file because many systems block .exe files downloaded from the Internet to prevent malware installation.
  3. We upload the zipped file to our server.
  4. We download it.
  5. We play the game from beginning to end on Mac.
  6. We play the game from beginning to end on Windows.

That is, we go through every step that a user would — and somewhere along the way we find an error that we somehow missed in all of our earlier testing. Maybe something we fixed in a later stage of the game was a script that was used in an earlier level and now that doesn’t work.

So … we go through all of the steps all over again. Yes, we do have debugging capabilities where we can skip to level 6 and test that, for example, but at the very end, you NEED to go through all of the steps your users will. Trust me. You can put in every unit test you want but it will not let you know that Microsoft or Chrome or some other organization put on this earth to try my patience now has a security feature that blocks the game from installing. You won’t see that three problems and all of the accompanying instructional material were left out.If you start at level 6 you will miss the fact that there is a problem in the transition from level 5 to level 6. And so on ad inifinitum until you go to speaking in Latin and wanting to tear out your eyeballs.

We go through all of the details so that when you play it all you see is a game that works.

My high school English teacher told me,

If something is easy to read, you can damn sure believe that it was hard to write.

I think this is also true,

Any kind of software that is easy to use, you can damn sure believe it was hard to make.

 

I think descriptive statistics are under-rated. One reason I like Leon Gordis’ Epidemiology book is that he agrees with me. He says that sometimes statistics pass the “inter-ocular test”. That is, they hit you right between the eyes.

moving eyeballs

I’m a big fan of eye-balling statistics and SAS/GRAPH is good for that. Let’s take this example. It is fairly well established that women have a longer life span than men in the United States. In other words, men die at a younger age. Is that true of all causes?

To answer that question, I used a subset of the Framingham Heart Study and looked at two major causes of death, coronary heart disease and cancer. The first thing I did was round the age at death into five year intervals to smooth out some of the fluctuations from year to year.

data test2 ;
set sashelp.heart ;
ageatdeath5 = round(ageatdeath,5) ;
proc freq data=test2 noprint;
tables sex*ageatdeath5*deathcause / missing out= test3 ;
/* NOTE THAT THE MISSING OPTION IS IMPORTANT */

THE DEVIL IS IN THE DETAILS

Then I did a frequency distribution by sex, age at death and cause of death. Notice that I used the missing option. That is super-important. Without it, instead of getting what percentage of the entire population died of a specific cause at a certain age,  I would get a percentage of those who died. However, as with many studies of survival, life expectancy, etc. a substantial proportion of the people were still alive at the time data were being collected. So, percentage of the population, and percentage of people who died were quite different numbers. I used the NOPRINT option on the PROC FREQ statement simply because I had no need to print out a long, convoluted frequency table I wasn’t going to use.

I used the OUT = option to output the frequency distribution to a dataset that I could use for graphing.

More details: The symbol statements just make the graphs easier to read by putting an asterisk at each data point and by joining the points together. I have very bad eyesight so anything I can do to make graphics more readable, I try to do.
symbol1 value = star ;
symbol1 interpol = join ;

Here I am just sorting the data set by cause of death and only keeping those with Cancer or Coronary Heart Disease.
proc sort data=test3;
by deathcause ;
where deathcause in (“Cancer”,”Coronary Heart Disease”);

 

Even more details.  You always want to have the axes the same on your charts or you can’t really compare them. That is what the UNIFORM option in the PROC GPLOT statement does. The PLOT statement requests a plot of percent who died at each age by sex. The LABEL statement just gives reasonable labels to my variables.

proc gplot data = test3 uniform;
plot percent*ageatdeath5 = sex ;
by deathcause ;
Label percent = “%”
ageatdeath5 = “Age at Death” ;

cause of death by age by gender

When you look at these graphs, even if your eyes are as bad as mine you can see a few things. The top chart is of cancer and you can conclude a couple of  things right away.

  1. There is not nearly the discrepancy in the death rates of men and women for cancer as there is for heart disease.
  2. Men are much more likely to die of heart disease than women at every age up until 80 years old. After that, I suspect that the percentage of men dying off has declined relative to women because a very large proportion of the men are already dead.

So, the answer to my question is “No.”

After I recommended a business incubator to an acquaintance, one of my co-founders protested,

“But I thought you said incubators were useless.”

I told her no, I didn’t say incubators were useless.  I said they were useless for US at this stage in our company. There are times when the right incubator can be extremely helpful. The wrong incubator is useless always.

I remember walking into one of the latter type years ago in a small town in North Dakota. The “business incubator” consisted of one bored staff member with no education or experience in starting or running a business. There were a few very old computers she pointed me to  that she said I could use to search the internet. When I asked her if she suggested any particular websites she just shrugged and said,

I don’t know. I’m just here to keep people from stealing the computers.

Other than the computers, the office had a few copies of a brochure on starting a business and that was it.

Useless.

When Erich, April and I started Spirit Lake Consulting, Inc., we used the services of a business incubator at the University of North Dakota. Erich was in the doctoral program there at the time and someone had mentioned it to him. They were extremely helpful. At this point, we all had some business experience – April was the controller for the tribe, Erich and I had both done consulting and Erich had been tribal college president. I had an MBA. Still, there was a lot we did not know about building and running a company. The UND incubator helped us in many ways.

First, they told us about a lot of resources of which we were unaware. The Small Business Innovation Research awards were one huge benefit to us that we learned about from them. They directed us to the Small Business Administration website that has lots of free resources for developing business plans.

Second, they provided us financial support. They gave us a $1,500 Phase 0 award that bought some of my time so that I could afford to take time away from clients and write our first SBIR proposal, which got funded. The set us up with two business consultants, paid for by the incubator. One was a graduate student, well meaning but pretty useless, who did something like a market analysis. The second was a very experienced and I am sure very well paid executive who gave us a lot of sound advice on business planning, writing SBIR proposals and marketing.

Third, they provided us contacts. When April was offered a job as General Manager of the casino, she was required to sell all of her interest in any other entity, including our company. We needed an accountant, since she had handled the payroll, taxes, and generally keeping the IRS from throwing us in jail. The incubator recommended our accountant, Donna Remer, who we still work with to this day on The Julia Group and 7 Generation Games.

baby

An incubator is just like it sounds – for little baby companies just getting started. We’re not that any more. We have an accountant, a business plan, an SBIR award. We know enough about the GSA schedule to know it’s not for us. We have an office, computers, printer, server.

At this point, an incubator would not be very useful for us. At one point, though, it was extremely helpful, and it may be that its extremely helpful for you.

Frequently, I hear adults who should know better argue against learning something, whether it is algebra, analysis of variance or learning a programming language. They say,

I’m 47 years old and I’ve never used (insert thing I didn’t learn here).

Yes, that is true. However, if you had learned it, there is a good chance that you would have used it .  For those of you protesting, “Hey, I learned algebra!” , maybe you did and maybe you didn’t.  Read my post on number sense.)

Let’s take this morning as an example. In keeping with my New Year’s Literary resolution, I started out the day reading the jQuery cookbook. There were two things I learned that I expect to use this year. One of them was very simple, but I didn’t know it.

var t1 = +new Date ;

This returns the current date in milliseconds converted to a number. Yes, you could use the Javascript Number() function but this saves you a step.

Now you can use this useful bit of code which I am planning on applying next week when I get done with what I’m working on now. I can use it to see how long a student worked on each individual problem and how long he or she took for the whole test.

(function() {
var log = [], first, last ;
time = function(message, since) {
var now= +new Date ;
var seconds = (now - (since || last)) /1000 ;
log.push(seconds.toFixed(3) + ':' + message + 'br/>') ;
return last = +new Date ;
};
time.done = function(selector) {
time('total', first) ;
$(selector).html(log.join('')) ;
};
first = last = +new Date ;
})() ;

Now, the author’s interest was in seeing how long each bit of code took to run. However, I can see how this could be really useful in the pretest and posttests we use for our games to see how long the student spent on each problem. We could call this function each time the student clicks on the next arrow to go to the next problem.

One of the Common Core Standards for mathematical practice is “Make sense of problems and persevere in solving them.”

How do you know if a student is “persevering”? One way would be to measure how long he or she spent on a particular problem before going on to the next. We cannot know for a fact that the student spent time thinking about it rather than staring off into space, but we can at least set a maximum amount of time the student spent thinking about it before going on to the next thing.

This takes me to the point Morton Jervens was making about not everything that counts can be counted and data does not always equal statistics.

While there is truth in that, I would say that much more that counts can be counted and much more of data can be turned into statistics if you know how to do it.

Learn how.

According to the New York Times, 45% of people make New Year’s resolutions and although 54% of them fail, the percentage who succeed (46% , proof that I can subtract and understand the concept of percentages) is more than ten times higher than for those people who say they would like to change but don’t make specific, stated resolutions.

confetti

I am one of those people who always makes New Year’s resolutions, and I usually keep them, putting me in the top 46% (woo-hoo!)  I believe there is value to setting public goals. Here are my resolutions for this year:

1. Read one technical book a month. I already have a list for the first seven months.

2. Get accepted in and complete an accelerator program.

3. Get three more games on the market. That won’t be as difficult as it sounds because our second game is just going through the last finishing touches and will be in at least six schools in January.

4. Work FULL-TIME on 7 Generation Games!  I am pretty excited about this one. I have two classes I have agreed to teach in February and May, and one consulting client with whom I have a long-term contract, but other than that, it is a year of game development and building a company. I am so excited.

5. Analyze all of the data we collect on our games’ effectiveness. Again, this is an easy one because it is a requirement of our grant funding. We already have pretest data on over 300 users from age 9-13, and I expect to collect data on 200-300 more as well as post-test data within the next few months.

6. Get moderately proficient with photoshop. I’m not an artist and I mostly use graphic converter, but since I have photoshop as part of the Creative Suite license, I figured I may as well learn a few things.

7. Learn more about the Mac OS. That might sound funny since I have had a Mac ever since I got one in 1984 instead of a big diamond engagement ring. (I told my to-be husband that I really wanted one of those new Macintosh computers more than a fancy diamond.) However, there are no doubt details that I don’t know and have never taken the time to learn.

8. Finish the pilot project of the data mining study I have been working on.

That is enough for the year, I think.

My sister, my mom and The Perfect Jennifer  always tells me that I need to have interests outside of work. I am not wholly convinced of that, since I am pretty happy the way I am. I don’t have any resolutions about visiting exotic places because I already have a lot on my plate for 2015. I keep saying I want to go back to Costa Rica and take some immersion program to improve my Spanish, but that will have to wait until 2016.  So, here is my non-work resolution:

9. To go visit my family at least twice this year. I’m already planning to go hiking in some mountains with my sister for a day or two this summer, and I want to take The Spoiled One to visit my mom in Florida and check out colleges in the south.

So …. those are my New Year’s resolutions and I even threw in a non-work related one. (Joy, I hope that makes you happy, or joyful or something.) I looked through my blogs for the last few years and i think  I have done okay on following through on resolutions and non-resolutions.

If you don’t have a blog but are still convinced of the whole efficacy of public accountability thing, please feel free to post your resolutions below. Or just do it because I’m nosy and wondering what other people’s resolutions are.

 

 

 

← Previous PageNext Page →