Aug
31
Yes, Virginia, There IS Discrimination against Women in Technology
Filed Under Dr. De Mars General Life Ramblings, Technology | 2 Comments
My work day started with a call on research design and ended ten hours later after I fixed a program that wasn’t working. I just resigned from my position as senior statistical consultant at a major research university so that I could concentrate on research. I’m on the technical staff on several projects, have a Ph.D., a record of scientific publication, am frequently an invited speaker on assessment, methodology and SAS programming. So, what am I whining about?
Who the fuck are you to say that I am whining?
That, my dear, is probably one of the reasons that I have been successful in this field, and one of the problems women in technology face.
I’m the size of the average twelve-year-old, female , Hispanic and over fifty to boot. Despite all of these disadvantages, I am doing well in this field, thank you because I have a few …
Qualities that I don’t think should be necessary for women in technical fields, but they are….
1. I can be a straight-A dyed in the wool bitch when the situation warrants.
One day, I was sitting in a faculty meeting with the suspicion that women in our department were not being taken seriously. As a statistician, I decided to collect a little data. I drew a cross-tabulation. The rows were gender of the speaker and the column was whether the next speaker responded – questioned, followed up, elaborated – or ignored the comment as if the speaker hadn’t even said anything. Of the speakers, 80% were male (the department was about 50% female) and of those 20%, most of the comments were made by me. Near the end of the meeting, I made a comment and again, a male member of the faculty made a remark that was if I hadn’t spoken. I pounded on the table and said,
“I said something and God damn it, you are all going to listen to me!”
Then, I mentioned the data I had been collecting during the meeting (believe me, the chi-square was highly significant). The two department chairs present were somewhat embarrassed but no one argued with my data. We discussed whatever the topic was – I think it was reducing our mathematics requirement for general education.
Personally, I don’t have a problem pounding on the table and swearing if that’s what it takes. Three points:
- The men in the room didn’t need to be that way.
- Not all women are like me.
- Not all women should HAVE to be like me. I have a pretty high self-esteem but not so high that I think everyone must be like me because I am so perfect.
Women who support Arrington’s view on Tech Crunch that it isn’t men’s fault that there aren’t more women in tech because “After all, look at me, I’m not complaining and I’m doing great” are perhaps missing the point that they are doing well because they have certain characteristics that men don’t need to have.
In my copious spare time, of which I have none, I teach judo. In 1984, I was the first American to win the world judo championships. One very important lesson it took me a while to learn as a coach is that not every athlete is me. Not every world class athlete is me. I would have been a better coach if I had learned that lesson earlier
2. I aggressively seek out mentors and figure everything out all by myself if I can’t find them.
I was discussing this with a young woman today. She’s probably my daughter’s age. We were working on a program and I commented to her that I had noticed she did not get the off-handed kind of help that the male staff members got. The men tell one another excitedly about new apps, new functions, bug fixes and other interesting and useful information they come across. She said,
“Now, I don’t understand this program at all, but you are explaining it to me where the guys would just be like – here, you’re not interested in this, let me do it. Or, you don’t know how to do this, so just go away.”
I know she is telling the truth because I have seen just exactly that happen to her many times. Maybe I should be more of a mentor. I feel a little bad about that, but she doesn’t work for me, and hey, I am busy. I told her,
“Well, of course you don’t understand it! No one comes out of the womb knowing this shit. But you’re smart, you’ll get it. Just keep plugging away. If you have any questions, ask!”
The program she wrote in the end was very good. Women, much more than men, in my experience, need to be immune to subtle and not so subtle discouragement, to disrespect. While Arrington says that Tech Crunch goes out of their way to invite women, these are the women who have already made it. Where men generally don’t go out of their way, and in fact, don’t even think about it, is in the unexamined assumptions and treatment of women. Most of the men this woman works with are very nice people who like women in general and are married to one or would very much like to be some day. They don’t treat me like this because …
3. I got all the credentials
I have a Ph.D., two masters degrees, 28 years programming experience, articles published in academic journals and so on. There is an enormous body of literature on social psychology on bias. In brief, the same study done over and over runs like this.
The identical resumes are sent out. Half of them have experience but no degree. Half have a degree but no experience. Of the entire sample of resumes, half of them have a name (or picture) that shows them to be female (or black). The other have are white (or male). The two resumes with experience/degree , white/black or male/ female are then sent out randomly to a group of college students/ personnel managers/ or whatever group.
The results are always the same. Overwhelmingly, the male (or white) candidate is selected. Those who choose the male candidate swear it had nothing to do with gender, he had experience. However, the managers/students/whatever who had the reversed resumes swear it had nothing to do with gender, he had a degree.
This is why all those people who loudly proclaim “I’m not a racist” or “I’m not sexist” have me wanting to slap them.
I have experience and I have the degrees. I work with men who don’t have nearly the educational qualifications I have. These guys are SMART and they’re fun and I like working with them. I truly don’t believe any woman would get the jobs they have without a graduate degree – and guess what, there are very , very few.
The one you hear next, of course is, “You just wouldn’t fit in with our team.”
Despite the impression I might give, I actually believe that most people are genuinely good at heart and well-meaning, that the false assumptions and subtle discrimination is not intentional and they really would try to change most of them, if it was pointed out. Some people are just jerks, though. There are people who would never want me working for them because I “am not a team player”.
Let me give you an example of a person who said he would never have me work for him..
I had written a program that was, if I do say so myself, a pretty kick-ass awesome piece of work. As most things that are that awesome, there were other people who helped, who came up with design suggestions, reviewed the results and made recommendations for improvement. All of the coding was done by me. I don’t get the chance to just write code that much and I was justly proud of this product.
4. Have the luck to have awesome bosses and mentors
We had a matrix management model at the time and the project manager, who was not my boss, came to me and wanted to have UMF review all of my work and “check that it is correct”. Now UMF is male and fits the stereotype of what a programmer should look like, which, I could gauge from this is not a Latina grandmother. UMF also is complete waste of oxygen as a programmer. Think the absolutely stupidest code you have ever seen written and that is UMF. I did not make up the acronym UMF. This is how he is referred to by the other programmers. The U stands for useless.
I said,
“No.”
Short version of long story, the project manager weenie went to my boss and told him that,
“AnnMaria says she’s not going to do this.”
to which my boss responded,
“Well, I guess that means she’s not going to do it.”
Dr. Richard Eyman was my doctoral advisor. He spent endless hours teaching me statistics. When everyone but me dropped one of the upper level doctoral courses, he taught it to me as an independent study. He introduced me to his friends who were profoundly competent in psychometrics, people like Jane Mercer. He made sure I took courses from people like Keith Widaman and Lew Petronovich.
It was just luck. I attended UC Riverside because I was pregnant with my second child, my husband had just taken a job at Rohr Aircraft in Riverside and I didn’t want a blank spot on my resume while I was out of the job market having a baby (which turned out to be two babies in thirteen months).
5. I’m not bothered that no one in the room looks like me.
Being in judo probably helped my career. I’m startled by the number of female judo competitors I meet who are in the tech field. It’s kind of ironic that a non-male, non-Japanese American would be the first from the U.S. to win the world championships because that is certainly not the demographic of U.S. judo competitors. I’ve spent so much of my life being the only woman in the room that I am used to it. It’s actually gotten better. I remember 28 years ago when I was pregnant (hence needing a bathroom every 30 minutes) at a meeting in an aerospace plant where NO ONE knew where there was a women’s restroom because all of the people I was meeting with were male engineers. I finally spotted another woman, grabbed her and said I KNOW you know! Turns out she was just visiting, and, in fact, did not!
Whether it should bother you or not that no one is like you (that you “just don’t fit in”) is a separate issue.
My point is that there are a number of characteristics that women must have that men don’t need to be successful in technology.
These are but PART of the reasons I see that there are few women in technical fields. And why, exactly, is pointing this out called whining?
Aug
29
A friend of mine mentioned that a woman had invited him to her apartment. Let me just say that my friend does not exactly rival Mother Teresa for celibacy. Astounded, I asked him why he hadn’t taken her up on the offer. He answered that his son had died recently and he didn’t want to spend the rest of his life having one-night stands. My friend is Dakota Sioux and they believe that what one does in the year after the death of a significant person – a spouse, a child – is how you will be the rest of your life. 
I think the Sioux have a great deal of traditional wisdom when it comes to dealing with death. Maybe there is something to this part of it. I know after my husband died I worked from when I got up until I fell asleep from exhaustion. I taught psychometrics and developmental psychology at one college, drove 80 miles and taught evening courses in market research (I have an MBA along with an MA & Ph.D.). Then, late into the night, I did analyses of data for evaluations, research projects, needs assessments for grant proposals. I paid off the medical bills, funeral bills, put two children through college, watched a third compete in two Olympics, and kept working 70-80 hours a week.
Finally, this year, my daughter’s coach asked me out of curiosity,
“Do you really need the money? Because you have a little girl at home and you are teaching until past her bedtime and then doing whatever you do at the university on the computer all day and flying around the country doing that other thing you do.”
And I realized, no, I didn’t really need the money and hadn’t for quite some time. Three of my daughters are living on their own, my (not-so-new any more) husband has a job and if I quit working completely the main difference in our lives might be that there would be less useless crap that I have to box up and give to Goodwill every month. If the economy really is stalling, it’s not our fault. In China, there’s probably an entire Factory of Useless Plastic and Electronic Crap named after my family.
So, I politely declined the opportunity to teach at two graduate schools this year. Then, I was on Twitter one day and Eric Greenspan, who I would not know if he came to my house and fixed my plumbing, asked the question, to the world in general,
“If I could do what I really wanted ___________ “
I thought about it for a minute and I almost wrote back, if I could do what I really wanted to do, I would just work on research projects that interest me. I would write papers on aspects of programming that I happened to be interested in that day. I’d learn as much as I could about everything I could in statistics and satisfy my curiosity about some things. For example, I really am skeptical that generalized models or mixed models are that superior to a plain old GLM if you have a million data points or if there are very large differences among your groups. (Okay, well maybe you wonder about the meaning of life, but as my daughters always tell me, I’m not THAT interesting.) I’d only work on projects that I thought had the potential to make a difference – or that I felt like doing just for the hell of it. And I would telecommute because what’s the point of living by the beach if you are away at work all the time. Besides, my house is where all my stuff is.
I didn’t write him back, first of all, because he wouldn’t know me if I spilled my Starbucks coffee on him, secondly because that is way the hell over 140 characters and thirdly, as I thought about this, my jaw dropped open because I realized there was not a single reason in the world that I couldn’t do exactly that this moment.
Within the next 48 hours, I was asked to work on not one but two separate projects that I thought would be awesome (oh, I’m charging them money. I’m not stupid). I turned in my letter of resignation. Wednesday, I leave from my last day at the university to Boston, where I have a meeting on research design that I think will be totally awesome. (If you like that sort of thing, which I really, really do.) I’ll see my daughter for her birthday the next day and take my granddaughter to the aquarium and then fly home.
Don’t take everything you read on twitter too seriously though. According to @vwadhwa companies prefer to hire younger engineers and tech people. This may be true in general, but my two weeks of notice isn’t even up yet and I am already booked for the next six months.
Vwadhwa seems to keep working, too, and from his teeny little picture on twitter he looks to be slightly older than Justin Bieber, if you ask me, which, you shouldn’t because I haven’t the faintest idea what Justin Bieber looks like, but I’m going with what it would be like if Abercrombie & Fitch sold people.
Aug
20
Does moving from being a novice to a not-so-novice programmer mean knowing everything there is to know about PROC TABULATE? Well, yes and no.
It would be hard to call someone who knew everything there was to know about SAS ODS or every possible regression procedure in Stata from regress to nlogit a newbie. However, how useful is that person when you need to perform calculations, format your data differently, merge files or write more efficient, readable code? In the long-term, whether going deep or going wide is better is a debatable topic. For a beginning programmer, though, my recommendation is to spread out and increase your knowledge across a range of topics from programming to presentation to new procedures.
So, you are just learning a programming language and are like all of the eager, young people I have worked with over the years, you are excited to branch out. Yay, you!
There are a lot of possible next steps. One is to start taking advantage of all of the resources for learning. With SAS you want to check out the mailing list SAS-L, sasCommunity.org, SAS publications, of course, and some of the many blogs by both SAS Institute employees and SAS users. For Linux, I like the Ubuntu forums. The SPSS India website has a lot of free tutorials on statistics (of course with SPSS) and it really surprised me how good it was because I hadn’t thought of SPSS as offering a lot of resources. It may be the effect of the IBM purchase, or maybe it was always that way. Raynald Levesque’s site has cool macros, scripts and syntax for SPSS. Yes, I realize it’s not a blazing insight to say that you can find some good stuff on the Internet, but I mentioned those because I think all of them are very friendly for the programmer who is just beginning and wants to move ahead (and I mean that in the knowledge sense not in the stepping over the bloodied bodies of your co-workers to get to the top sense).
A second direction is to become familiar with more than just your current limited area of expertise. If you are a SAS programmer, you could learn more SAS products, such as SAS Enterprise Guide, Enterprise Miner, and smaller features like the Power and Sample Size application. Or, maybe you want to learn to use Stata on Linux and start learning about the Linux operating system. SAS and SPSS both have interfaces with R. SPSS was pushing Python procedures at one time. I’m not sure if they’re still heading in that direction. It doesn’t matter whether they are or not, really, if it is something that interests YOU.
Whichever direction you take shows that you have become interested enough to search for new knowledge and that is always a great sign. I was asked today who I would recommend for a new position that just opened up and why. I said,
“There are several people I can think of who could learn what you need. What I would look for is someone who is genuinely, sincerely interested, and not just in a $10 an hour raise, but in statistics, because that person will learn on their own, grow and develop into the kind of person you want.”
I was at the Predictive Perspectives seminar today, met someone who was excited to be there, not because she was looking for a job but because she wanted to learn. I gave her my card and asked her to contact me. It may be a cliche but it’s still true. You can’t buy passion (well, maybe in Las Vegas, but it’s legal there).
The biggest step forward, though, I think is – PLAY! As you learn more about programming languages, statements, functions, procedures and products you’ll invariably like some more than others. For example, my brain tries to crawl out of my skull to escape being melted by boredom whenever I have to look at SQL code. You may love SQL (unlikely as that may be). The more time you spend working on different projects, the more you’ll learn and the more you’ll discover what really interests you. I don’t have a profound thought to help you decide which direction to go except this. Remember the movie, Pleasantville, where his mother said
“…I had the right house. I had the right car. I had the right life.”
And her son answered,
“There is no right house. There is no right car.”
Learning SAS (or any programming language) is very much like that. There is no right choice. There are a lot of choices. The more you learn, the greater the number of choices you get to make.
How cool is that?
Aug
11
Read a great line in Seth Godin’s book, Linchpin,
“It’s not an effort contest, it’s an art contest.”
The point being that no one cares how hard you worked, they care how great your product is. Of course, great products tend to result from hard work along the “necessary-but-not-sufficient-condition” lines, but that’s a whole different topic.
This fit with what I have been thinking about statistics a lot lately,
“It’s not an IQ contest, it’s a knowledge contest.”
I read a lot of statistics articles, attend a lot of presentations on statistics – and I see a huge disconnect between these really brilliant people and those who are making decisions on policy, funding and management. This is our fault. Well, maybe not yours and mine personally, but the fault of the discipline as a whole. We spend much of our time researching new statistics, writing papers on the latest developments. Some of these decrease the standard error, or at least give us a more accurate estimate. That’s all good, right?
We spend FAR less time on making sure our results are interpretable to people other than ourselves. In my previous life, I led faculty development workshops on teaching mathematics and science. Many of my colleagues were dismissive of the idea that we ought to work to make our ideas discernible to the average person. One of them actually sniffed dismissively (I didn’t know people did that outside of Victorian novels). He told me that he had gone to the effort to get a Ph.D. and these students, if they didn’t have the intelligence and/or weren’t willing to work hard enough to learn from him, then they all deserved to fail and that people like me were ruining academia.
SIGH. It gave me empathy with a school psychologist I knew who once wrote as a child’s diagnosis “Dyspedagogia”. (Latin for “bad teaching”.) After he was caught out, he refused to change it, too, saying he stood by his professional opinion.
Anyway…. I don’t know if that professor was as brilliant as he thought – he did obtain a doctorate in a scientific field from an Ivy League school – but I really don’t care. It’s not an IQ contest. As for everyone else who is not a statistician, as Sheila Tobias said in her wonderful book by that title, “They’re not dumb, they’re different”.
As a statistician, I KNOW that I see things differently and my job is to explain what I see, and not necessarily in the way it makes sense to me but to make sense to other people.
For example, when I look at this:
When I see the table, the formula below automatically pops into my head.
Then, I think, that there should be about the same number of males and females saying “yes”.
The total sample is (roughly) evenly divided by gender and about 300 people said they planned to join the military after high school, so the expected value is 150 women.
Subtracting 72 from the 150 one would expect gives a value of about 80, which squared is 6,400.
It is already obvious this is significant.
Really, I don’t get nearly that explicit. What I’m more likely to think is,
“150 – 72 squared is a lot. That’s significant.”
Then I run it in JMP or SAS or SPSS and see that the chi-square is 110, p < .001 and I am right.
Occasionally someone shows me a table like this in their dissertation and the chi-square value is 1.06 and it is non-significant. They tell me this is what the computer gave them. I look at it and tell them.
“Well, then, the computer is wrong.”
The uncomprehending shock on someone’s face when I say this always strikes me as a bit odd, as is their complete amazement when I turn out to be right.
Of course, that just means that somewhere along the line they incorrectly programmed the computer. It happens to everybody.
Yet, people go away shaking their heads in disbelief, and I have a reputation as “that scary smart person in the corner”. Truly, here was the great dramatic insight that popped into my head that lead to the conclusion that their chi-square value was incorrectly calculated. I looked at the table for two seconds and this was my exact thought.
“6,400 is a big number.”
Not very profound, is it? You might be thinking my point here is that I should work back to the steps that led me to that insight. If you are thinking that, you weren’t paying attention to the “different” part. In fact, this is what I did;
“Take a look at this chart. You can see that about three times as many men plan to enter the military as women. That might not be significant if you had five or ten people, but you can see (on the Y axis), you’ve got over 1,000 people in each group.
When you get a fairly large number of people and the difference between the two groups is a factor of three to one then you can be pretty certain there is a relationship between being in that group and whatever the other variable is, in this case the decision to join the military.
Now, this fits with what you already were pretty sure of in this case, which is that men are significantly more likely to enter military service than women.
However, that’s not necessarily the case that the results will always bear out your pre-conceived notions. For example, the next set of results, on race and ethnicity, did not fit at all what we had expected….”
While this might not make me sound as intelligent as going through the chi-square formula (which is really a pretty simple formula, after all), it does accomplish three things:
- I am pretty certain that everyone in the room when I explained this understood exactly what I was saying and could follow my logic.
- Because they had the confidence that they understood exactly what I was saying and how it fit with what they already knew, they agreed with me, they didn’t just go along with me. This is crucially important because if later on I want them to support some initiative based on this information they’re a hell of a lot more likely to do it than if they’re not really sure what I was talking about.
- Having gained some confidence that they can understand what is going on here and that it is not completely random, when we come to the next set of results, which just happens to be contrary to what was expected, I have their attention and I am half-way to having their agreement and support. Come on now, admit it, aren’t you just a little curious to find out what we learned about the relationship to military service to race and ethnicity?
I don’t know if any of this shows how intelligent I am. If anything, I think I might have more claim to intelligence based on the fact that I figured out how to do the chart in JMP only a few minutes after starting to use it. I wanted ”male/ female” and “Yes/No” instead of ones and zeros and reasoned that since SPSS has value labels and SAS has proc format, there must be some way to do it. (There is, right click on the column you want. Click on COLUMN INFO then select COLUMN PROPERTIES and then select VALUE LABELS”.
I remember a quote on the definition of intelligence that,
“The essential difference between a genius and a moron is the ability to generalize.”
So, is any of this proof that I’m a genius? I don’t know. I do know that nobody cares.
Aug
3
An apology to Victoria Brookhart & why new Ph.D’s think they are smarter than God
Filed Under Dr. De Mars General Life Ramblings | 1 Comment
I owe Victoria Brookhart an apology. One day, we were in the graduate student lounge discussing research methods and she burst out,
“Isn’t this great? Here we are warming ourselves at the fires of knowledge. These are the times we’ll remember our whole lives as the good old days.”
The rest of us threw spitballs at her, I think. We were too cool and cynical to think like that, or, if we did, to say it out loud. On top of which, Vicky was one of the very few in the group to pursue qualitative research, or as, us quantitative types referred to it, creative writing.
It might have been me that wrote on the white board outside her office,
“Victoria’s sex life is participant observation.”
Actually, I think it was Dio, but it might have been me. Of course, a few years later, when I was working in North Dakota and doing research on Native Americans with developmental disabilities, I had to call up and grovel and ask for her advice on how to get started in an area where there was NO published literature. That’s not why I owe her an apology, though,
Vick was right. If you are fortunate like we were, and like so few people are any more, that you can devote full time to learning, that is an incredible blessing. Yes, we felt more broke than blessed at the time, plus there is that whole thing about being a professor’s slave, or at least indentured servant. And yet ….
We had the opportunity to delve into subjects that interested us, to read books about them, collect data, analyze data, formulate theories, test them, swear, talk things out with people who had whopping years more experience than us who could steer us in a better direction, be mentored, read the latest articles, hang out at the computing center until 10 p.m. when maybe we could talk somebody into loading OUR tapes on the tape drives since no one else would be there until morning and running OUR data right now so we could get the results and …
I don’t know if I was smarter back then but I sure THOUGHT I was smarter. It made me smile several years later when the department chair showed me his evaluation of a new member of our department which began,
“Like most new Ph.D.’s, X thinks he is smarter than God, but we expect he’ll get over it …”
I have heard a lot of comments like that over the years. I remember when the dean of a state college came to campus recruiting. Someone (it could have been me) asked him how many articles he had published in the 20 years he had been there. He said he hadn’t published any but he had received over $30 million in grants and he was proud of that. After he left, my friends and I discussed with a superior attitude how WE were not impressed because WE already each had a couple of articles in press. WE obviously were so much smarter than him.
Fast-forward twenty-five years and Victoria was right. One of the reasons that dean didn’t publish is it takes an enormous amount of work to write a grant and then to administer it. I’m sure that grant money funded a lot of graduate students over the years and a lot of demonstration and service programs. He was right to be proud of it.
I think of that gentleman from time to time when I make a mistake and ask myself how I could have been so stupid or careless. Well, really, I’m not particularly stupid or careless but what I am is pulled in many directions. Some things, whether it is serving on departmental search committees or putting together a justification for a major software purchase or analyzing data on the number of students applying so we can project the need for courses – they just have to be done.
Several years ago, when I was doing a lot more grants management, I know some of my students secretly classified me as one of those old professors who understood statistics but had to have other people write programs for them.
I’m very happy that I get a chance to do more programming these days. Now that three of my children are grown, I can afford to choose to do more of the work I want to do, a luxury I haven’t had as much since graduate school.
I listen to doctoral students talk about how behind the times and overrated their professors are, with that attitude of superiority I remember we all had. Now I laugh at it realizing that part of the reason that our professors weren’t up to our standards of research productivity and error-free equations and code is that they had other shit to do. Sad, but true.
Every decade or so, I step back and work in an academic setting, and it is always amazing to me the new developments, new statistics, new software, that I haven’t had time to learn, and it takes me a year or two to catch up.
Coming from a business into academia, I’m not nearly as convinced of the greater practical benefit of some of these. We often give ourselves far too much credit in the universities. I hear my colleagues say,
“This article is in an academic journal but eventually it gets used by the people in the field.”
And I think to myself,
“No, usually, it doesn’t.”
It never occurred to me at the time, but now I wonder what that dean thought of us. I am guessing he was partly wistful and partly amused. Probably like most people in the field, I remember the first computer program I ever wrote, back in 1975, in BASIC. It was a program to create poems, as a basis it started with one by Ogden Nash,
“Behold the hippopotamus
We laugh at how he looks to us
And yet in moments dark and grim
I wonder how we look to him
Peace, peace, oh hippopotamus
We really look all right to us
As you no doubt delight the eye
Of other hippopotami”
Here’s to you, Victoria. You were right all along.
Jul
31
Meta-blogging, social media & naked mole rats
Filed Under Dr. De Mars General Life Ramblings | 1 Comment
This is a blog about blogging. On the fun scale, meta-blogging probably falls midway between metadata and meta-analysis but I am going to do it anyway. In stages, because, just like if it has bullet points it must be serious business, if it has stages, numbers and units you’ve never heard of, like petapixels, it must be serious research:
Stage 1: Claim Guru Status
I am a social media expert because I have three blogs and accounts on Facebook, LinkedIn, twitter and Bebo. I’m also a member of sasCommunity.org and a bunch of things like Second Life I randomly signed up for and then forgot about.
Stage 2: Scoff at other self-styled gurus
Seriously, what is it with these people who claim to be experts on social media, anyway? As someone said, I must listen to them because they have days of experience. How long does something have to exist before you can claim to be an expert on it? What makes you an expert, anyway?
I have been told over and over by self-proclaimed media gurus that everyone must have an account on Facebook or they’re missing out because half a billion, 700 million, 5 teraflops or a petabyte of shakes of a lamb’s tail of people are on Facebook.
We make money at The Julia Group by doing survey design, evaluation research, quantitative and qualitative analysis. We have some consultants who are terrific statistical programmers and others who are experts in qualitative research. Now, twice in the past twenty years, I have gotten substantial contracts because I was walking out a door as someone I knew was walking in and she (both times it was a she) said,
“I can’t believe I ran into you! I really need a statistician. Are you available?”
So, given that…. it is theoretically possible that someone will see my Facebook page (well, actually, they won’t because it’s private) and say,
“Hey, you rock at Mafia Wars! How about doing $150,000 worth of data analysis for us over the next five years to evaluate our $3 million project?”
I think it more likely, though, that we will continue to get business by submitting grant proposals, bidding on contracts and having people I have worked with in the past call and say,
“We need someone with your expertise on this project. Can you work with us?”
Before the economy went south it seemed to me like the only thing on LinkedIn was headhunters and job seekers. Since I was neither, I never had any use for it. Lately, I’ve given it a second look and it seems mildly interesting. I can’t say I’ve seen any huge benefits but I haven’t been using it much either, so I’d say the jury is still out.
What about Second Life, the place a few years ago that any serious company would be and if you weren’t there you were a dinosaur missing out on the “new Internet”? This article that conveniently was written this week, entitled “How to save Second Life”, well I guess the title gives a clue how that’s going.
Stage 3: Pontificate
What does work? I have run into interesting people all over the U.S. and in a few other countries who say they read my blog, which has gotten me a few free beers. I have occasionally gotten work or hired people I met at conferences, so, it is possible that some of these people who read my blog may eventually become clients or employees. If not, hey, I got free beer, so hurray blog!
Someone said that Facebook is the people you went to school with while Twitter is the people you wish you went to school with. For me, personally, twitter has been the most useful social media in that it has given me a lot of references to useful websites. To paraphrase Douglas Adams, the Internet is big, really big. Having people with interests like mine who pass along sites of interest to them has yielded a number of very useful leads on development in fields I’m interested in, latest research results or just new ideas.
“The most exciting phrase to hear in science, the one that heralds new discoveries, is not Eureka! (I found it!) but rather, ‘hmm… that’s funny…’”
- Isaac Asimov
Twitter has delivered more than its fair share of those “that’s funny” moments.
Stage 4 (optional): Say something remotely sensible. If that fails you, just rant
This is the third blog I started. My earlier two blogs were focused on judo (I was the world judo champion 26 years ago and still coach) and for Spirit Lake Consulting, which was our parent company before The Julia Group was spun off as a separate company.
I write this blog because I feel like it. The main benefit for me, in addition to the free beer, is that I have forgotten more about statistics and statistical software than most people know. (Part of this is simply a function of being older than most people.) The problem is, that I have forgotten it. So, I post it here. Suppose I vaguely remember that I did something with AGGREGATE in SPSS that added the number of times a person appeared in the database to each case, or that I did install SAS Enterprise Miner for Desktop and here I am a year later wondering how the hell was it that I did that. I can go search my blog and it pops up.
Since I write it for me, I am often surprised that other people read it, and I am really surprised when I Google a phrase looking for information that my blog comes up on the first page. Sometimes it reminds me of a book I liked when I was about ten years old, called The Pushcart War. At one point, there is a public TV show on the topic of traffic problems with various experts and a movie star. During the show, the movie star states that the problem with traffic is “The trucks are too big and there are too many of them.” And, the author says, since no one understood what the experts said and they all understood what the movie star said, that was the part they paid attention to.
I think it is because I am writing for me that much of what I say is very accessible. I wonder how many statisticians when they are writing notes for themselves do like me and explain
AUTOREGRESSIVE – autoregressive means something is regressed on itself
As this very cool page from the National Institute on Standards and Technology says
An autoregressive model is simply a linear regression of the current value of the series against one or more prior values of the series.
… and how many of them write as if they are trying to fit a mold.
Of course, now that I find more people are actually reading my blog I have given some thought to being (slightly) more serious and not say things like PROC MI is used to make up data, the MI standing for “my imagination”. I haven’t actually done anything. I’ve just thought about it. Sort of like the world’s most spoiled twelve-year-old’s view on cleaning her room.
[If you have a mad desire for a serious blog on multiple imputation, I did write one. Your life is now complete. You're welcome.]
The rant part
If we were a bigger company, and if I wasn’t the majority owner, this blog would be very different. There would be no profanity. There would be no mention of other company’s products unless they paid us. There would definitely be no mention of any company’s products having all the attractiveness of a sexual experience with a naked mole rat for fear they might sue us. There would be a legal review, an editorial review, a graphic designer to tell me that my blog too often features pictures of naked mole rats.
Someone would tell me that I cannot end sentences with prepositions, that “cannot” is actually one word, that the word count for the average post exceeds what research says people will read and that I don’t write in the tense recommended by the Microsoft Style Guide. All of these people would be paid. My blog would cost many, many times what the current cost to produce (which currently is the value of my time between midnight and 1 a.m. and a glass of Chardonnay) and take at least 60 times longer from beginning to actual publication (which is currently from the time I get an idea to the end of the hour it takes me to slowly sip a glass of white wine, ramble and locate pictures of naked mole rats).
I don’t know how blogging went from being an individual’s musings on life to the property of corporate weenies. This isn’t to say there aren’t some corporate blogs that don’t suck. Jon Peck did a good one for SPSS that I hope will eventually be reinvigorated once the whole IBM takeover gets sorted out. Chris Hemedinger does a cool one on SAS Enterprise Guide.
Most corporate blogs, though, fit the cartoon by Hugh Macleod in his awesome book, Ignore Everybody,
“Welcome to Nobody Cares. population 6 billion.”
I started this blog when a web site editor for an organization I was working for told me that my web pages on statistics could not be approved until they were rewritten because,
“Our entire website has to have one voice and that is the CEO. You cannot use examples that might offend anyone. You must speak in a formal tone. Your pictures are too big and often inappropriate. Your headers are not the right size. You need to conform with the Microsoft Style Guide (attached). I am sure you put a lot of work into this and some people (sniff) seem to think you are very bright but I am sorry we cannot post anything that might cause anyone to call and complain. I suggest that if you want to have your own voice and express your own opinions, you write a blog.”
So, I did. And the Microsoft Style Guide (whatever that is) can bite me.
In case you were wondering, no naked mole rats were molested in the writing of this post.
Jul
16
There are two kinds of people in organizations; those who can count and those who claim to have “people skills”.
When David Wechsler created the most commonly used intelligence test in America, the results gave two IQ scores, Verbal and Performance Intelligence. Dr. Wechsler said he had noticed that there are some people who were good at using words and some people were good at solving problems with things, and that those were both types of intelligence. Steven Baker, wrote an interesting book about one subset of those people, those who control and analyze data. He called them the numerati. I used that term here to describe everyone high on Wechsler’s second intelligence score, because it was simpler than saying “technologists, mathematicians, statisticians, engineers, scientists and people like them”. Besides, I liked the term and it’s my blog.
In a great many areas, from the BP oil spill, to global warming, to curing diseases like AIDS or cancer, to genetic engineering to technology start-ups, many people in American society can be heard to say, “Scientists can certainly find a solution for this”, sometimes prefaced by “If we could send a man to the moon… ”
Listening to the news, my husband, an actual rocket scientist type, has responded sardonically more than once to these comments,
“Well, your faith is touching but… ”
Yes, it would seem we LOVE scientists, engineers, mathematicians, statisticians, computer programmers, all of those people who are going to figure this stuff out, right? I consider myself to be very fortunate to often be right in the thick of machinery that powers science. I get to help people create propensity scores to quantify mortality risk, write macros to create simulated data for parallel analyses, modify programs so they run on a supercomputer and a lot more fun stuff. Some nights I leave the building hours later than I had planned or go home and work into the morning because I am chasing a problem and lose track of time. What’s not to love about that?
And yet, when I look at who is supervising our technical staff, the engineers, physicians, and scientists, it is often a different story. You would THINK, that the bright young people coming up would be the ones you want to encourage. And yet ….
TRUE STORY #1:
There once was a technical support center with some very savvy technical staff. The kind of people who took computers apart just to see if they could put them back together again or who would run thirteen virtual machines at a time just to see what would happen. Their department supervisor was pretty decent with Unix and even better hacking into the Windows operating system. When he left, some of the staff applied for his position as well as some very good technical people from outside. The new supervisor had no technical expertise but “people skills”. The training to teach the staff more about Unix, more about systems administration was limited to guest lecturers. Recently, I was copied on an email to the staff regarding the proper “phraseology” for answering the phone and telling people how happy you are they called.
This troubled me enough that I mentioned to an executive for that organization how misguided I thought it was. I pointed out that when people call technical support they want their technical question answered. Further, since this is an entry point for many people, it was a great opportunity to train and develop people who already have some skills and talent to be successful. I was told that while, yes, for people like ME, this was true but these technical people did not have MY abilities (whatever those might be) and thus what they really needed was not an explanation of the difference between a 32-bit and 64-bit operating system or parallel versus serial processing. What they really needed was signs saying, “Smile when you answer the phone.”
For a while, when people from tech support would call me, I would answer the phone with,
“Hi, this is AnnMaria, I’m very fucking happy-ology you called.”
Then I would answer their questions. They seemed to be very fucking happy-ology about it, too.
Except for one middle manager type who overheard me and told me I was wildly inappropriate and asked me what if it had been the president calling me. I pointed out that I have caller ID and unless Barack Obama happened to be visiting technical support, borrowed someone’s phone and called me just to ask a question about logistic regression, it wasn’t very likely to be an issue.
So, we’re not mentoring those with potential to be up-and-comers. What about the existing “numerati”?
At the university level – sadly, for the last thirty years, the number of tenured professors in all fields has been dropping dramatically . The proportion of classes taught by full-time professors has been dropping. There is a rising new group called “clinical professors” who are paid only to teach and don’t do any research at all. Then, there are the for-profit universities, a rapidly rising group that takes up almost a quarter of all federal student aid. They don’t support any research at all.
This article from the Chronicle of Higher Ed discusses both the fact that the tenure track isn’t all it’s cracked up to be “you can’t speak your mind for seven years” and the number of positions is declining anyway.
From what I have seen, in technology companies SOME SUBSET of the numerati are well-treated. A software company may esteem its programmers but disregard the market research staff that can hold some whiz-bang statisticians. A pharmaceutical company may treat very well the clinical researchers but completely ignore the programmers who run their accounting and inventory systems.
Doesn’t this make sense? Isn’t it the old cliche about staff versus line positions we learned about in business school? Maybe, maybe not but certainly it is stupid. Those professors, I would think, would be “line jobs”. As for the accounting, market research and inventory folks, if you let them apply some of those equations they might make or save you millions of dollars. Why do we generally think that science and technology are the answers to all of our national ills but overlook those skills in specific situations?
TRUE STORY #2
An organization planned to expand the software licensed. A new purchase, available to all researchers, for a very modest fee, would have given them the capability to easily do decision trees, neural networks, survival analysis and more. The purchase was stopped because the vendor’s attorney and the client’s attorney could not agree on a phrase in the contract. This was reviewed by two managers and two attorneys, none of whom actually knew what the software could do for the organization.
As I hear these stories, and many, many more like them, I wonder what exact “people skills” these middle managers bring to organizations. If the skill is to develop people, you’d think they would bring in people to train them. Maybe they would look at data that showed the greatest areas of need. If it was to support existing researchers, you’d think they’d ask them what it is they need and try to ACTIVELY promote new technologies rather than “Say no and see if anyone screams”.
In looking at some of the behavior (think the phraseology example and the fact that this individual was hired) it shows an active distrust, disrespect and dislike for the technical staff.
I cannot state for sure why this happens in some organizations (certainly not all), but this distinction between “people skills” and “research skills” got me thinking of the difference in security.
What are technical skills? The ability to conduct an experiment, diagnose a patient, write a program. Generally, these are very portable. As a consultant, when I leave one client and go to the next >95% of what makes me valuable goes with me. Yes, the next client may have some specific system I need to learn, but the definition of a training dataset, how to select a stratified random sample and all the programming languages I know go with me. The same is true of anyone in a technical or scientific field. The more you apply your skills, the more value you have and you take that value with you wherever you go.
I’m a bit confused by the “people skills” that some middle managers supposedly have. As a friend of mine commented about the manager for his department,
“They say he was hired for his ‘people skills’ and not his expertise. Well, we’re all people in this department and we all think he’s a dork. “
People skills include the ability to motivate and communicate. Those are a lot harder skills to document. How do you know your staff didn’t succeed despite you? For middle managers, a good deal of success seems to depend on connections. It’s not what you know, it’s who you know. I say that not in a perjorative sort of way but because I have noticed that many middle managers LOVE meetings. The point is, as I have been told many times,
“So we can all get to know each other.”
and I wonder,
“Why?”
One reason middle managers and the numerati don’t get along is they seem to think differently. Take meetings. My view on most meetings with a middle manager with a Gantt chart is
“Why are you here?”
I’m not talking about the person from the department we are supposed to serve who can tell me about how the data are stored, what questions they hope the data can help them answer and problems with data quality. I totally get why she is there.
I also understand people at a higher level of management who have an enormous project and need to parcel out parts of it to different teams, who need to set priorities for resources. I understand what they are doing and why we need them.
What I DON’T get is the guy in the middle who organizes meetings, requires agenda and minutes so they can be forwarded to “upper upper management”.
Here is what I am thinking:
“My team and I are going to do the absolute best we can. Tell us how much money is available and when you need it done. Then, go away.”
I really, really don’t know what the middle managers are thinking. What I deeply suspect, though, is that when I read about people in the New York Times who used to have a job that paid $60,000 or $80,000 or $100,000 a year and now they have been unemployed for two years, that I am reading about THEM.
Jul
11
Nine out of ten businesses owned by trapeze artists are not in this survey
Filed Under Dr. De Mars General Life Ramblings | 2 Comments
Back when I was in college, there was a group advocating burning rock albums. A major investigative journalist wrote a story on their motivation (I think he either wrote for Rolling Stone or Playboy, the latter of which, yes, I really did read for the articles. Despite having competed on my college track team and the U.S. judo team, worked as a programmer and played rugby, I am actually not a lesbian, a fact which frequently surprises people. But I digress. Even more than usual.)
One question that I remember was how the group came about their figure of 80% of out-of-wedlock babies were conceived by listening to rock music. The founder said they had heard this figure cited by an evangelist during a revival in their town. The reporter followed up with the question,
“So, do you have any data to support your album burning other than the traveling evangelist poll?”
There were many things wrong with this study, the first of which being, I suspect, that it didn’t exist. Beyond that, there is the sampling issue. Is 80% a high number? Perhaps it is the music listened to by women of child-bearing age, the Big Band, Lawrence Welk fans being primarily post-menopausal and thus not at-risk of pregnancy on either side of wedlock.
A causal relationship is at least implied, otherwise what was the whole burning point? To test this hypothesis, I turned on Blinded by the Light by Bruce Springsteen at full volume. Two unmarried daughters of child-bearing age were in the house, as was my husband.
No pregnancies ensued. Said husband remained downstairs building a robot with the world’s most spoiled twelve-year-old, although he did come up momentarily to ask if I minded if he turned down the music.
One daughter announced she was going to the apartment of the other daughter because, and I quote,
“No offense, but you people are boring.”
Which brings me to my tangentially related point… Lately I have been trying to come to the source of the frequently stated “facts” that
A. Small businesses produce the jobs that lead the economy out of a recession
B. Most jobs are created by start-ups
C. What small businesses really need are credit and counseling. Business plans always feature in there big.
I have no idea whether A and B are true or not. I rather suspect A is in part because there are way more small businesses than any other type. It goes back to the Traveling Evangelist Poll (whether it existed or not). If there are way more people working in small businesses then a 10% increase in them is going to be more than a 10% increase in the fewer number of people who work for large business.
As far as C, I am a bit confused. Vivek Wadhwa, who is a pretty interesting writer on this topic, had this article on Tech Crunch on July 10, with which I agree completely. The title is “You’re no Steve Jobs” and his main point is that the problem with many start-ups is that no one wants to buy their crap. He said it way more nicely than that, though.
Years ago, I used to spend some time on a forum for small businesses. One of the reasons I quit was because in the start-up section, no one ever said,
“What? Are you crazy?”
Instead, there were always supportive comments like,
“Live your dream, baby! I’m sure your business making hand-knit sweaters for turtles will make millions by next summer.”
These people are always saying that if they only had the money, they could have this amazingly amazing life but that the big bad banks would just not lend them the money to go out and buy a building to turn into a turtle sweater making factory.
The very odd thing, odder even than the turtle sweaters, is that the same week I read an article with which I mostly disagreed by the same Vivek Wadhwa ! (No, I am not stalking him, it was coincidence. I swear.)
Well, those aren’t 100% contradictory … they don’t start because of lack of knowledge and financing but they DO fail after they start because no one buys their products. (Maybe some of that fear of failure is realism.)
Having been in business 25 years and never once been part of a survey (hey, I’m WORKING here!) I was curious as to the source of these figures.
Being a good academic, Wadhwa did provide references, and the first was to a study of 549 entrepeneurs in high-growth industries.
I don’t doubt that one might find for this specific group that access to capital is a big barrier. To a small company becoming a big company in a short period of time, the capital to buy a building, working capital to meet a growing payroll, all are important.
What percentage of jobs are those, though? I don’t know but I don’t think it is a lot. Google and Yahoo both have offices in Santa Monica. Geocities had its headquarters a few blocks from where I’m sitting. Even in our relatively tech sector of the world, the number of “high-growth” employees are dwarfed by those working at the restaurants, hotels, liquor stores, car dealers, movie and TV industry.
In other areas where I work frequently, like North Dakota, and Washington, D.C., the proportion of “high growth” industry personnel is even smaller.
What about the “more jobs are created by start-ups”? I looked into that, too. There was a really interesting study by the Kauffman Foundation that pointed out that start-ups can ONLY create jobs. Their definition of a start-up is a business that started this year.
Jobs created = Jobs This Year – Jobs Last Year
Since the second part for a start-up is zero, it can ONLY add to the number of jobs.
Existing companies may hire ten people (cool for the ten people hired) but have 15 who retired, were laid off, fired or had a heart attack due to having sex while listening to rock music. Even though ten people were hired, the company has a net loss of five jobs.
I am not convinced, though, that the answer to economic malaise is to have a massive number of start-ups as many of them (like turtle-sweater lady) may be negative on the job-creation number by the next year.
Where do they get this idea that what small businesses really need is credit, so the government should give the banks more money to lend?
I went to Google, the source of all knowledge, and typed in “Small Business Survey.” The first several that came up were places like the North Texas Small Business Development Center, Citibank and the Huffington Post Survey on the Credit Crunch.
The latter asks :
“Small business owners: have you applied for business credit? Was it approved, or turned down? Have you not applied because you didn’t think you’d have a chance?”
I’m just sayin’ that perhaps organizations whose main function is to give credit, help you obtain credit and polls asking you if you have applied for credit might be a bit biased in the proportion of those reporting credit is an issue compared to say, the general population of small businesses.
On Monday, the world’s most spoiled 12-year-old is starting trapeze school.
Supporting the Economy
I don’t think what the Trapeze School (which is a small business, not vulnerable to out-sourcing) needed was a line of credit or a business plan. From their perspective, what they needed was to swipe my credit card.
I have some more thoughts on representativeness (or lack thereof) in surveys but the world’s most spoiled twelve-year-old is asking to be tucked into bed and my husband is suggesting that perhaps The Rolling Stones would be better than Bruce Springsteen.
I doubt it. We already have a fifteen-year gap between the oldest child and the youngest. A few years ago, I thought I might be pregnant again (we DID go to a Rolling Stones concert around that time). He was very cool about it until we got the results that said I wasn’t pregnant and then he exclaimed,
“THANK GOD!”
So… I punched him.
Jul
7
What Small Businesses Need to Create Jobs
Filed Under Dr. De Mars General Life Ramblings, The Julia Group | Leave a Comment
I’ve been in business for over twenty years. All of that time, I have run a small business, by choice. During those twenty years, I have had a sick husband, been widowed, had four children – so I had some reasons that becoming the next Oracle was not my priority. However, I have made a profit every year, some years more than others, and have increased and decreased my number of employees as necessary.
The more articles I read on small business in general and women-owned businesses in particular, the more I wonder how many of those organizations talking about helping small business owners create jobs include people who have actually run a small business.
There seems to be a great concern about the disparity in access to venture capital. Now, that may be a concern for some small businesses but most of the people I know own consulting companies, hair salons, restaurants, retail stores or manufacture products like t-shirts. They are not attractive to VCs because they are not going to have exponential growth.
Many of these small business owners, like me and my friends, are going to be in business for ten, twenty years or more, and pay corporate taxes, payroll taxes and everything else our accountant says we have to pony up every few months.
What about jobs?
I think everyone trying to create jobs through small business should read the insightful article Andrew Grove, Intel-cofounder, wrote on this subject. Those high-flying tech firms create a lot of jobs – overseas ! One problem with the VC-find-the-next-Apple approach, of course, is that those jobs may help investors but they don’t help the U.S. unemployment rate. Many, many of the high tech, high ROI jobs end up in China and India. (Seriously, read Grove’s article. It’s great.)
Twenty years ago, my business partners and I decided against outsourcing because we did not want to employ fewer Americans and pay someone in another country a sub-minimum wage so we could be richer. I know that sounds un-American, but part of our motivation in founding a business, which still derives much of its revenue from work on reservations, was to make life better for people. Obviously, we are privately owned, so we can make those choices.
The other thing I don’t need that every agency and company seems to want to sell me is a business plan. I have a business plan. Like most companies, the gist of it is to have revenues exceed expenses. Okay, it is a little more than that, BUT – after 20 years most of the business owners I know are not kept from hiring from lack of a plan. In fact, their plan is to add workers to meet the demand. It is certainly NOT to take out loans (guaranteed or not) so we can expand and hire more workers.
If anyone really seriously wanted to help small business create jobs they would make it easier for them to get business.
I had to laugh. Several times, representatives from the same “small business services” company have called me telling me,
“We’ll help you get YOUR money from the federal government. After all, it’s YOUR money.”
and then went on to promise me we could get on the GSA schedule and agencies would be falling over themselves to just pull our name up and order a million dollars of consulting services from us. I told their representative that’s not the way it works and he assured me it was and they had done that for lots of companies. I told him to email me the name of one. I’m still waiting.
I am not sure where the stimulus money went. I see some signs that the roads are being upgraded with Recovery Act funds, so that is a good thing. I don’t actually know anyone who got any of that $200 million that went to the NIH in grants, although I know a lot of people who applied, but that all went to universities any way.
I may actually bite the bullet and complete the section 8(a) application this year, although it grates on me to do it. The time I spend on that will take away from billable hours so it will actually COST me money. I’m still debating on it.
Don’t get the idea that we’re sitting around here whining. We have work enough to keep the people we have employed and I am now looking for new contracts. We’ve already turned down a few over the last year, which may sound inconsistent, but it’s not.
We submitted one proposal in May, a second in June. We have too much current work to take time away to do a proposal this month. I’ll submit one or two in August and September, depending on how tired I feel.
Taking a six-month or shorter contract that takes up all of your time and keeps you from bidding on multi-year contracts is not good business. Just bidding on everything that comes down the pike isn’t too bright, either. We look for a match between our capabilities and what the client needs, for areas we can really do excellent work. That way, they are happy and come back to us again and again. After a quarter-century in business, we DO kind of know what we are doing.
I hear a lot about tax breaks for small business. Well, we pay a hell of a lot of taxes and that would be nice. Even though we will probably be exempt from the requirement to provide health care, I have always offered that as an option to employees and our costs may go up a little. Taxes and health care costs are not what keep me from adding employees.
It seems like the people aiming to help small businesses are sincere. However, it’s like the old cliche that when the only tool you have is a hammer, every problem is a nail. Because most of these organizations have people who know how to write business plans, fill out loan applications, apply for certifications of some status or another and lobby on Capitol Hill, that’s what they see as the way to help small businesses.
Most people who have been in business for decades don’t need some consultant to help them develop a business plan before they can add jobs. If their business has been around a long time, they already have a line of credit. I’m not sure what they need is tax cuts or worse health care coverage for their employees.
What they need is work.
I’m surprised I have to explain this to you.
Jul
6
Are you a programmer or aren’t you?
Filed Under Dr. De Mars General Life Ramblings, Software, Technology | Leave a Comment
So, I am writing a paper on how you know you (or someone else) is a “real” programmer. That is, they don’t fit in that “new user” box any more. But how do you make that decision?
Is it like pornography, you just know it when you look at it? (Not that I ever personally looked at any of course, but I have heard you can find it on the Internet if you try really hard.)
Yesterday, Rob Meekings made a comment about design decisions. That is certainly a distinction, when you get to the point that you are actually thinking that way. For example, I often will merge everything together in one long dataset, a habit that makes those who love SQL and the star schema just cringe. The REASON I do this is that most of the people I work with are researchers using very powerful computers with datasets of a few thousand observations, or, at most, a few hundred thousand. Even on a desktop, an analysis with SAS, Stata or SPSS takes seconds. It isn’t worth taking an extra hour or two to make a program run in one second instead of two. It also may make the program more difficult for the user to maintain him/herself.
HOWEVER, when I am running a program that runs against a 100GB dataset and can take hours to run because the researcher cannot use a supercomputer, e.g., due to security classification, I’ll spend a good bit of time trying to make it run as efficiently as possible.
If there isn’t a pressing reason not to do it, I’d recommend someone with a large dataset considering running it on a cluster and take advantage of parallel processing capabilities. This means changing your code slightly to run on a different OS, often Linux or some other Unix version.
I do a lot of “throw away programming”, that’s not to say it’s garbage. Sometimes I think my work is quite good, in fact, but it’s not production code that runs every day to produce reports on 500 different stores. When I DO write production code, I do several things differently. One is that I make good use of %include statements. For example, if there is a footnote that is going to be in every single output that says, “Funding provided by National Science Foundation Rural Systemic Initiative Grant #1234-2010″ and several more lines about the university, address for contact, etc., I am going to have a small file that I just include. Yes, I could copy and paste it or have that as a template for when I create a new program. BUT what happens when we get another grant and we want to recognize both funding agencies in everything we publish?
My point, and you may be surprised by this point to find that I do, in fact, have one, is that a distinction between novice and non-novice programmers is that they have the luxury of thinking about a design because they know more than one way to do something.