A few months ago, in between everything else in my life, I had been working on applications for accelerators and an executive summary for investors.
While money is always a good thing, I was surprised by the unexpected benefits. The questions asked by investors are the same ones you should be asking yourself.
- If you had additional funds, how would you spend it? How much would go to marketing, to software development, to legal, accounting and operating expenses? Asking this question made us look at how we re spending the money we do have, and re-allocate it in some areas.
- What is your cost of customer acquisition? What marketing channels have paid off for you and which have not been worth it? Asking these questions led us to limit our efforts in some areas and expand others. Ads on Google and Facebook did not pay off for us . Social media – blogs and twitter – have been effective. ( Subliminal advertising: Check out our Fish Lake demo here!)
- What are your biggest personnel needs? Marketing? Software Development? Management? Have you identified people that you want to hire? If not, how do you intend to find those people?
- What are the strengths of your founders? Your staff? Are any of those people not pulling their weight? This is a question people are reluctant to ask and equally reluctant to quantify. Asking that question of ourselves has resulted in everyone taking inventory of their base camp projects and making items specific with fixed deadlines.
Now, it’s our second week as part of the Boom Startup Ed Tech accelerator and I have to say it has been an exhausting and productive time.
Getting people to invest in your company is much more than refining your pitch (although we have done that as well). I used to mock those requirements that you describe the opportunity, your solution and your key team members in two minutes or less. No more. You know why? Because it shows that you know what is truly important to your customers and what is unique about your product and company.
As I’ve said over and over the past two weeks, almost everyone here knows they should have done things like identify market segments, track results of social media efforts, call potential customers and a hundred of other things. The fact is, most founders are so busy focused on building a product that they don’t spend near enough time building a business.
I read this in a review of a study on teacher expectancy effects but it could really apply to so many other studies.
If these results bear any relationship at all to reality, it is indeed a fortunate coincidence.
Those of us who choose careers in research like to believe that it is all like everyone learns in their textbooks: hypothesis, data collection, analysis, results, conclusion and *PRESTO* knowledge.
In a few weeks, I will be in San Diego at the Western Users of SAS Software conference presenting results of the past year of testing with Fish Lake and Spirit Lake: The Game.
Occasionally, colleagues will ask me about my interest in the nuts and bolts of data analysis and why I ‘bother’ presenting at SAS conferences instead of ‘the real thing’, like the American Educational Research Association or the National Council on Family Relations. One of the main reasons is that I like to be very transparent about how my data were collected, scored and analyzed. I find it odd that these “details” are given short shrift in publications when, in fact, all of the conclusions ever published rely on the assumption that these “details” were done correctly.
Presenting the nuts and bolts of the data cleaning, coding and analysis assures any funding agency or consumer of the research that it was done correctly. Or, if anyone wants to dispute the way I’ve done the analyses, at least it is crystal clear how exactly the data were processed. In most cases, the reader has no idea and is just taking it on faith that the researcher did everything correctly – which given some of the bozos I know is pretty shaky ground.
Once I have confidence that the data sets are in good shape, have corrected any data entry problems, deleted outliers, accurately scored measures and identified any statistical assumptions that need to be met, then I’m ready to proceed to the analyses with confidence.
Think about that next time someone with a turned up nose says,
“I don’t go to that type of conference.”
Yeah? Well, I do.
In assessing whether our Fish Lake game really works to teach fractions, we collect a lot of data, including a pretest and a post-test. We also use a lot of types of items, including a couple of essay questions. Being reasonable people, we are interested in the extent to which the ratings on these items agree.
To measure agreement between two raters, we use Kappa’s coefficient. PROC FREQ produces two types of Kappa coefficients. The Kappa coefficient ranges from -1 to 1, with 1 indicating perfect agreement, 1 indicating exactly the agreement that would be expected by chance and negative numbers indicating less agreement than would be expected by chance . When there are only two categories, PROC FREQ produces only the Kappa coefficient. When more than two categories are rated, a weighted Kappa is also produced which credits categories closer together as partial agreement and categories at the extreme ends as no agreement.
The code is really simple:
ODS GRAPHICS ON;
PROC FREQ DATA =datasetname ;
TABLES variable1*variable2 / PLOTS = KAPPAPLOT;
TEST AGREE ;
Including the ODS GRAPHICS ON statement and the PLOTS = KAPPAPLOT option in your TABLES statement will give you a plot of both the agreement and distribution of ratings. Personally, I find the kappa plots, like the example below, to be pretty helpful.
This visual representation of the agreement shows that there was a large amount of exact agreement (dark blue shading) for incorrect answers, scored 0, with a small percentage partial agreement and very few with no agreement. With 3 categories, only exact agreement or partial agreement is possible for the middle category. Two other take-away points from this plot are that agreement is lower for correct and partially correct answers than incorrect ones and that the distribution is skewed, with a large proportion of answers scored incorrect. Because it is adjusted for chance agreement, Kappa is affected by the distribution among categories . If each rater scores 90% of the answers correct, there should be 81% agreement by chance, thus requiring an extremely high level of agreement to be significantly different from chance. The Kappa plot shows agreement and distribution simultaneously, which is why I like it.
Want to play the game ? You can download it here, as well as our game for younger players, Spirit Lake.
Having been in Rio the past two weeks where Internet access occurred in five-minute increments roughly 11 hours apart, I have a few blog posts saved up I was unable to upload. Expect a few more over the next few days.
I was in our office at the Water Garden exactly 1 day before heading out of town again. I’ll get back in early August, be home for less than 24 hours before leaving again.
The receptionist at our office (are they called receptionists still, or do they have a new title at the same pay, like ‘telecommunications operative’ ?) said to me:
Wow, you have a really exciting life!
It was funny because I had just been thinking how this was getting a little old. Most of the time, whatever airport I’m in, my laptop connects automatically to the wi-fi because I’ve been there before.
Whether it is Minneapolis, Washington National or Tokyo, all airports are pretty much the same and I know the drill – change planes, eat mediocre airport food, which is marginally better than nasty airplane food, pick up luggage, get car, get to hotel. It doesn’t matter if I’m in a hotel in Dallas or DC or Devils Lake, my routine is pretty much the same – get some exercise, get on the Internet to get some work done and get to sleep.
I work pretty much every day, so I have trouble keeping track of what day it is or what time zone I’m in. My iPad says it’s 9:08 because that’s the time in Texas, which was the last place I had an internet connection. It’s 7:08 at home in Santa Monica and 11:08 in Rio.
My point is that you do anything often enough it becomes routine. I’m guessing that the person who does the fireworks show at Disneyland every night comes to work and says,
Ho, hum. Another day of explosions, another dollar.
This isn’t to say that my life isn’t really fun and awesome at times. It is. The fun times aren’t so much sleeping on a 10-hour flight to Rio, though.
The most fun I have had lately has been with people.
At Spirit Lake, I went to dinner with the teachers and site coordinator who were involved in 7 Generation Games from the very roughest alpha version. Dr. Erich Longie, whom I am proud to call a friend, had to leave early for an event the next day honoring the president of UND who had overseen the retiring of the Fighting Sioux nickname.
The teachers and I spent the evening lakeside sitting around a fire talking about our children, love lives (or lack thereof) and future plans. As a female tech entrepreneur, one doesn’t get a lot of chances for “girl talk”, so it was precious time just hanging out.
It’s the closest I am ever going to get to camping since The Invisible Developer believes that sleeping outside on the ground is ingratitude to our ancestors who evolved to the point where they could build houses.
I’m not so jaded that I haven’t taken the time to take in the sights in Rio. Everything from whatever this is (papaya?) and espresso for breakfast to the flavelas. In fact, I’m actually taking more time to “stop and smell the roses”, or, as in the North Dakota State Fair that I stopped in on last week, the sheep manure, than I ever have in the past.
In part, that is because as I get older, I am beginning to appreciate the opportunities I have had to see everything from Athens, Tunisia, Beijing, the Grand Canyon, New York City, to the Peace Garden, the Minnesota State Fair and Disneyland. Some of the time I have taken full advantage of the opportunity and others not so much.
More and more, I’m having the sense to realize that those opportunities will not be infinite and not to let them pass by.
There is also the fact that I’ve spent so much of my time in various ‘exciting’ locales working and I’ve come to accept there will never come a day when I say,
“That’s it. I have done all of the work there is to do.”
So, I may as well take a break now and then and take in the scenery. It’s also taken me about 30 years in business to quit worrying that I will run out of work if I don’t take every single opportunity that comes down the line.
The other thing I have learned about my exciting life is that just about every place I land has something worth seeing. Sully’s Hill Game Preserve on the Spirit Lake Nation is definitely worth spending a day or two exploring. So is the Prairie Village in Rugby, North Dakota. Maybe it’s routine to you but to me it’s exciting.
Which is why, when I finally get some time at home late next month, I’m going to take an afternoon off and spend it lying on Santa Monica Beach.
——- My Day Job
Stuck in an airport and bored? You can play Fish Lake and get smarter.
(You don’t have to be stuck in an airport. You can play it if you are bored at home, too.)
Sometimes, you can just eyeball it.
Really, if something truly is an outlier, you ought to be able to spot it. Take this plot, for example.
It should be pretty obvious that the vast majority of our sample for the Fish Lake game were students in grades, 4, 5 and 6. Those in the lower grades are clearly exceptions. I don’t know who put 0 as their grade, because I doubt any of our users had no education.
I use these plots especially if I’m explaining why I think certain records should be deleted from a sample. For many people, it seems as if the visual representation makes it clearer that “some of these things don’t belong here.”
Did you know that you can get a plot from PROC FREQ just by adding an option, like so:
PROC FREQ DATA= datasetname ;
TABLES variable / PLOTS=FREQPLOT ;
This will produce the frequency plot seen above, as well as a table for your frequency distribution.
Well, if you didn’t know, now you know.
I’m really busy here in Brazil. Honest. You think I am sitting here like this:
But really, it’s like this:
Still, I have taken time away to explain to you how not to get your sorry ass fired, so listen up niños . I had the benefit of starting writing software at a large organization, General Dynamics. Having the experience from the beginning being part of a time prepared me in a way I seldom see from people who have been working solo their whole careers.
It’s easy to be the smartest person in the room if you’re the ONLY person in the room. If you’re used to being THE computer whiz and suddenly find yourself part of a development team, let me give you a few pieces of advice:
2. Don’t assume everyone is stupider than you. The CSS expert removed those classes because clearly everyone else had been too stupid to do it. When you come into a new organization, make the reasonable assumption that other people have jobs there because they are not incompetent morons. (This is not always the case, but I believe in giving everyone the benefit of the doubt.) ASK! The people who treat you like you are stupid because you don’t know the answer to a question are insufferable pompous asses and you don’t care about their opinions. Normal people realize that you are new and you don’t know everything. They are happy to help as long as you don’t overdo it. Even on those rare occasions that I have gotten a question from a new person that made me say to myself, “Are you fucking kidding me?” I tried to give the benefit of the doubt that maybe the person was confused or having a bad day. After all, I’ve made stupid mistakes a time or two myself, and I’m pretty sure I’m not a moron.
3. Document! I admit to being a hypocrite because as soon as I have finished something I’m SO tempted to go on to the next analysis/ part of the game, etc. As our wonderful 7 Generation Games CMO Maria says, documentation is one of those things no one ever wants to do but everyone wishes was done. Maybe you know exactly what you were thinking when you wrote that code, but I am pretty certain that your colleagues were not hired for their mind-reading skills.
If you’re reading this, you’ve probably mastered multiplication, division and fractions, but you may be fond of a small human who has not. Check out 7 Generation Games. Play adventure games in a virtual world and learn stuff.
Games that make you smarter ! Buy them for your children, sponsor a school or play yourself (we won’t judge!)
Well, I don’t know about you, but I am anyway, and I’m guessing if you are reading this that you probably are, too.
If you read the bottom half of the internet (a.k.a. the comments section), you’ll find that any mention of privilege sets some people off …
I’m not privileged! My parents didn’t have any money when I was growing up! We bought our clothes at K-Mart when we were lucky and the thrift store when we weren’t. When I was in college, I worked for $2.12 an hour ! I work 60 hours a week! I raised 3 children while going to school and holding down a full-time job.
Yeah, me, too.
I’m still privileged.
Being without privilege when you were young and being privileged as an adult are not mutually exclusive.
I’m typing this on an iPad with a Logitech keyboard, on the 12th? 15th? flight this year and it’s only July.
Before I got on this flight, I was in the United lounge, drinking free Chardonnay and working on my laptop. A lot of other people were working there as well. It is a lot easier to work in the lounge because it’s quieter and there’s lots of outlets and plenty of comfortable chairs.
While Darling Daughter Number 3 arranged for the lounge and this flight to Rio, I paid for all of the other flights this year, the iPad, keyboard, laptop and all of the software on it, with the money I earned from working really hard. I also bought an iPhone and paid for a service with a personal hotspot.
My point is that since I walk around with thousands of dollars in technology in my briefcase (and many of you do, too), it is a lot easier for me to spend six hours en route writing a conference paper, analyzing data for a research study, editing graphics or video or programming the next level of a game. It’s much easier for me to get 4 hours work done in whatever hotel I find myself in the evening.
If I think it would benefit me to take a few days off and attend a conference on Unity 3D, SAS software, small business innovation research or serious games, I do it. I don’t have to go through three layers of management to get permission.
Being privileged and working hard are not mutually exclusive.
It’s like Dave Winer said – Money sweats.
He was talking about interest, that once you make money you get paid money just for having it. It’s also true in technology careers – once you have made some money, you can use that money to buy you advantages that keep you ahead of the competition, like practice with the latest version of whatever software you use, or 20 more productive hours each month, as you sit in airports.
It’s true in athletics as well. The better you get, the better coaches, nutritionists, strength trainers you can afford. You don’t have to take public transportation or work a full-time job at Starbuck’s, so you have more training and better training than the competition, even if they are working just as hard.
In America, a great many of us are privileged. If more of us recognized it, maybe as a society we’d be a little more humble, a little more grateful and a little more generous.
It was recently noted on our company twitter account that I’m out of town so often that I’ve come to resemble the Travelocity gnome in more than stature.
I’ve been traveling on business for nearly my entire career. Despite the proliferation of Google hangouts, Skype, Webex, Go-to-Meeting, FaceTime and God knows how many other technological innovations, there are still a lot of situations that require me to head out of LAX to points north, south, east and west.
This lifestyle has definitely shaped who I am.
I’m always surprised at people who travel frequently and come home to a seven-bedroom house.
The reality shows on the “tiny house” movement fascinate me, in part because of the creative uses they make of space in their designs, but also because I can see myself living in a very small space. After all, much of my time is spent in one or two rooms in a hotel.
For me, living out of a suitcase for weeks out of every month has meant that I have pared down greatly the amount of clothes, books and other personal possessions I “need”. The Invisible Developer points out that we could not live in a tiny house because his stuff alone would fill it up. That’s what happens when you stay put – you accumulate things.
It was a big advantage for me growing up in a house with a lot of siblings and not a lot of space. That may not sound like an advantage, but the result was that I had to be able to study wherever I found a spot that was unoccupied – in the room I shared with my sister and younger brother, in the attic, on the back porch, a corner of the living room table.
To this day, I can work anywhere. I’m typing this on an ipad as the plane is landing. I’ll sit in the airport and review a data analysis for a client that I’ll download on to my laptop using the personal hotspot from my iphone. When I get to the hotel, HOPEFULLY there will be cell phone and Internet access so that I can finish the online course I’m taking on a new game development library. (Thanks, lynda.com !)
I really do travel far and wide, which means there are few things I can depend on having – no, not even internet or cell phone reception. This seems self-evident to some people but inconceivable to those inside the Silicon Beach/ Silicon Valley bubble.
Before I met The Invisible Developer, the longest I had ever lived in one house was 4 years. I’d lived in Japan, Canada, Pakistan, California, Minnesota, Colorado, Illinois, Missouri, Mississippi and no doubt other places I’ve forgotten by now. Since I had moved so often, traveling to a new place wasn’t all that different from being in the relatively new place where I was living.
After 18 years in Santa Monica, I’ve gotten used to a location, and for the first time I find traveling a trifle unsettling because I have actually gotten settled somewhere. That’s been reinforced by the fact that I work out of my home office a lot. Working at home is convenient because it is where all of my stuff is and it’s full of people who know me,
Maybe that explains why I’ve started to give some thought to traveling less. I don’t think that will actually happen, for a while, though. If I were to just stay home and write code, I could make a fairly good living, but then someone else would be flying hither and yon to meet potential partners, customers and investors and the final decisions would rest with that person. With responsibility comes a certain level of discomfort, regardless of what you told yourself it was going to be like “when I’m running things”.
I’m not ready to turn over the reins just yet – which is why I’m finishing this from a plane to Minneapolis where I was re-routed after my flight from Denver to Minot was cancelled. So, now, I’ll finally get a Minneapolis to Minot flight that lands around midnight and then drive 2 hours to Spirit Lake.
Wake me up when the glamorous part of travel starts.
Previously, I discussed PROC FREQ for checking the validity of your data. Now we are on to data analysis, but, as anyone who does analysis for more than about 23 minutes can tell you, cleaning your data and doing analysis is seldom a two-step process. In fact, it’s more like a loop of two steps, over and over.
First, we have the basic.
PROC FREQ DATA = mydata.quizzes ;
TABLES passed /binomial ;
This will give me not only what percentage passed a quiz that they took,
but also the 95% confidence limits.
I didn’t have any real justification for hypothesizing any other population value. What proportion of kids should be able to pass a quiz that is ostensibly at their grade level? Half of them – as in, the “average” kid? All of them, since it’s their grade level? I’m sure there are lots of numbers one could want to test.
If you do have a specific proportion, say, 75%, you’d code it like this:
PROC FREQ DATA =in.quizzes ;
TABLES passed / BINOMIAL (P=.75);
Note that the P= has to be enclosed in parentheses or you’ll get an error.
So, out of the 770 quizzes that were taken by students, only 30.65% of them passed. However, the quizzes aren’t all of equal difficulty, are they? Probably not.
So, my next PROC FREQ is a cross-tabulation of quiz by passed. I don’t need the column percent or percent of total. I just want to know what percent passed or failed each quiz and how many players took that quiz. The way the game is designed, you only need to study and take a quiz if you failed one of the math challenges, so there will be varying numbers of players for each quiz.
PROC FREQ DATA =in.quizzes ;
TABLES quiz*passed /NOCOL NOPERCENT ;
The first variable will be the row variable and the one after the * will be the column variable. Since I’m only interested in the row percent and N, I included the NOCOL and NOPERCENT options to suppress printing of the column and total percentages.
Before I make anything of these statistics, I want to ask myself, what is going on with quiz22 (which actually comes after quiz2) and quiz4? Why did so many students take these two quizzes? I can tell at a glance that it wasn’t a coding error that made it impossible to pass the quiz (my first thought), since over a quarter of the students passed each one.
This leaves me three possibilities:
- The problem before the quiz was difficult for students, so many of them ended up taking the quiz (another PROC FREQ)
- One of the problems in the quiz was coded incorrectly, so some students failed the quiz when they shouldn’t have,
- There was a problem with the server repeatedly sending the data that was not picked up in the previous analyses (another PROC FREQ).
Remember what I said at the beginning about data analysis being a loop? So, back to the top!
People often ask me how I get so much done. Over and over, I have found one of the simplest ways to increase productivity is by not reading my email in the morning. Some days, I don’t get around to reading it at all.
This evening, when I finally opened my email, I had over 1,100 messages. In less than an hour, I have winnowed it down to 480.
As you can imagine, the majority is spam – offers to optimize my site for search, improve my sex life and sell me dishes. For some reason, Pottery Barn, Williams-Sonoma and the like are convinced I’m a good market for housewares. Bizarre, because the only thing domestic about me is that I live in a house and the only thing I make for dinner is reservations.
The spam takes a few minutes to sift through.
Then there is the large category of email that is unnecessary. If you are guilty of any of these, do the community a favor and reform.
- Notifications I don’t need. It’s nice that someone wants to thank me for speaking at an event, that students from Billy Bob Elementary appreciated the donated site license. I appreciate that you thought of me, I do. I don’t, however, appreciate it so much that I’m going to put off starting work for an hour to read all of this. Sorry, not sorry. I also don’t need to be informed that I have changed my password (I know, because I did it) or that The Spoiled One has a game at 3 pm in San Francisco ( do they think I didn’t notice she was gone?)
- Copies of email that I don’t need to be copied on. If someone else will be attending an event on behalf of our company, held a meeting or has been assigned an action item, nice. I don’t need to be informed and I don’t need a copy of the agenda of meetings I won’t attend unless there is an item along the lines of, “In an attempt to curry favor with the venture capital gods, we will be making a human sacrifice of the CEO in the lobby at noon on Wednesday.” – in which case I might want to avoid the office mid-week.
The biggest reason for not reading my email, though, is that I already have an idea of what my priorities are for the day and I start on the highest priority first. There has yet to be a day when I looked at my to-do list and it read:
Priority #1 : Read email.
The real time suck in my email is the emails from people who want me to do things – complete this form, write this letter, review this contract, give me your opinion on this, let me know when you can schedule this meeting. The key point here is that all of these involve someone else’s priorities.
There is such a temptation to take 5 or 10 minutes to respond to each of these requests, or to at least consider it, then decide it is not a priority and I’ll do it later.
Occasionally, I do miss something that I needed to know. However, that inconvenience is minor compared to not starting off my day with an hour or two of reacting to what other people request, rather than acting on priorities for my own company.