Data-Driven Decision-Making

Great By Choice by Jim Collins is a terrific book. Wonderful, timely, and very readable and accessible!

It is also very entertaining for the numerous anecdotes in it.

The idea I find the most useful for me is being "empirical." By that, the author apparently means that your decision must have data to back it up, that you must have proof, reason other than and outside of your feelings, hunches, gut feels etc for any decision you make.

He points out that great leaders can afford to be bold and to take frightfully big risks because they base their decisions on their own personal observations, not on second-hand research or heresy etc. e.g. (My examples) Steve Jobs's decision to run their own Apple stores, and (again) his decisions to enter the lacklustre tablet market with iPad. If you want to be bold, daring, decisive (and at the same time, responsible and not suicidal,) you need solid proofs, concrete data, undeniable facts, repeatable experimental results etc.

Does it mean Data-Driven Decision-making?

The author doesn't say much about this. But I would like to point out that whatever data, whatever amount and type of data, in itself, won't give you a decision. Data in whatever form or amount will not decide for you. Given the same data, two different committees can reach two diametrically opposite decisions.

If certain data does give you a decision, then you are not making a decision, but following an order, a recommendation, or some other person's decision.

Data Prophet : Nate Silver

I'm summarizing an article Data Prophet in Wired October 2012. It's an interview with Nate Silver, a statistician, a data scientist.

He pointed out a number of problems with our typical predictive models.

Out of Sample Problems

Here we get no data for the phenomenon because we did not collect such data in the first place. e.g. If your model excludes real estate as an industry, whatever rise, fall, crisis happens in real estate, you will have no data for them. And your economic prediction will also miss problmes and effects arising from real estate

Over Fitting Problem

Here, you're mistaking mere coincidences or co-occurrences for patterns. e.g. You observed that whenever ice cream sales go up, murder cases go up too. Then, you assumed that there was a pattern.
 If you curb ice cream sales, will you prevent some murder cases then?

Nate advises that instead of looking for ideas, we should accept and see what is really there in the data.

Problem of Over-reacting to New Data

Maybe your model is too flexible or sensitive to new data or new types of data.

Here, I can give a personal experience. Recently I read in Newsweek about life-after-death experience of a neurosurgeon. He said that he got to a strange place and met strange beings, that there our senses were fused such that he could see a sound and hear a color at the same time etc. I believed every word of his. That is, a person like him who has lived like he has lived his life, can get to a place like that.

Nat Silver wisely warns that more is worse if we don't have a good framework, that we'd better stick to basic models if we are prone to over-reaction to new data. Wise poker players, stock investors, soldiers all now this.

I asked myself what if I got there myself. I would hear-see a sound-smell-sight. Then, I would see-hear it dissolve and another sense arise .... dissolve and arise .... dissolve and arise .... dissolve and arise ....I hope my basic, simplistic model will work there too.

The Need to Test in the Right Environment

An example of the problem situation may go like this: You created your model with data from New Mexico. You then tested the model in New Mexico again.

According  to Nate Silver, the right environment is" an environment where you don’t know what’s going to happen."