Decision Making Bias: Insensitivity to Sample Size
Learn and practice avoiding the Insensitivity to Sample Size bias with exercises.
What is the Insensitivity to Sample Size bias?
The concept in one sentence:
Not taking into account the sample size (number of observations, subjects, data, etc.) when coming to a conclusion.
The concept in one quote:
The exaggerated faith in small samples is only one example of a more general illusion – we pay more attention to the content of messages than to information about their reliability, and as a result end up with a view of the world around us that is simpler and more coherent than the data justify. Jumping to conclusions is a safer sport in the world of our imagination than it is in reality.
The benefit of avoiding the bias:
Not jumping to the wrong conclusion based on a limited amount of data.
An advert states that “4 out of 5 dentists recommend this toothpaste”.
Is the toothpaste good?
Not enough information
Not enough information—we don’t know how many dentists were involved.
Assume 5 or 10 dentists were surveyed to make this kind of statement. The size of the sample cannot justify the generalization of the overall population of dentists.
A certain town is served by two hospitals. In the larger hospital about 45 babies are born each day, and in the smaller hospital about 15 babies are born each day. As you know, about 50% of all babies are boys. However, the exact percentage varies from day to day. Sometimes it may be higher than 50%, sometimes lower.
For a period of 1 year, each hospital recorded the days on which more than 60% of the babies born were boys.
Which hospital do you think recorded more such days?
The larger hospital
The smaller hospital
About the same (that is, within 5% of each other)
Howard Wainer and Harris L. Zwerling demonstrated that kidney cancer rates are lowest in counties that are mostly rural, sparsely populated, and located in traditionally Republican states in the Midwest, the South, and the West, but that they are also highest in counties that are mostly rural, sparsely populated, and located in traditionally Republican states in the Midwest, the South, and the West.
What is happening?
Is poker or investing a game of luck?
An investor sees an advert for a new investment fund, boasting of having generated 15% annualized returns since its inception.
Is the new investment fund good?
Not enough information
Alice and Bob both have a bag of 50 balls (red and green balls).
One of the bag has mostly red balls.
Alice draws 5 balls from her bag of balls, and finds that 4 are red and 1 is green.
Bob draws 20 balls from his bag of balls, and finds that 12 are red and 8 are green.
Who has better evidence that their bag is predominantly red?
The Mozart Effect
A study suggested that playing classical music to babies and young children might make them smarter. The findings spawned a whole cottage industry of books, CD and videos.
Without considering how many babies and young children took part in the study, can you trust the findings?
Many researchers have sought the secret of successful education by identifying the most successful schools in the hope of discovering what distinguishes them from others. One of the conclusions of this research is that the most successful schools, on average, are small.
This encouraged the Gates Foundation to make a substantial investment in the creation of small schools, sometimes by splitting large schools into smaller units.
Should we invest more in the creation of small schools?
Consider three possible sequences of Head (H) or Tail (T) coin flips:
Are the sequences equally likely?
Answer to Exercise 1
The smaller hospital.
Most individuals choose 3, expecting the two hospitals to record a similar number of days on which 60 percent or more of the babies board are boys. People seem to have some basic idea of how unusual it is to have 60 percent of a random event occurring in a specific direction. However, statistics tells us that we are much more likely to observe 60 percent of male babies in a smaller sample than in a larger sample.” This effect is easy to understand. Think about which is more likely: getting more than 60 percent heads in three flips of coin or getting more than 60 percent heads in 3,000 flips.
Answer to Exercise 2
While various environmental and economic reasons could be advanced for these facts, Wainer and Zwerlig argue that this is an artifact of sample size. Because of the small sample size, the incidence of a certain kind of cancer in small rural counties is more likely to be further from the mean, in one direction or another.
Answer to Exercise 3
The answer is, not at all. But sample sizes matter. On any given day a good investor or a good poker player can lose money. Any stock investment can turn out to be a loser no matter how large the edge appears. Same for a poker hand. One poker tournament isn’t very different from a coin-flipping contest and neither is six months of investment results.
On that basis luck plays a role. But over time – over thousands of hands against a variety of players and over hundreds of investments in a variety of market environments – skill wins out.
Answer to Exercise 4
Not enough information.
If the fund has not been investing for very long, the results could be due to short-term anomalies and have little to do with the fund’s actual investment methodology.
Answer to Exercise 5
If we pick a small sample, we run a greater risk of the small sample being unusual just by chance. Therefore the sample of 20 actually provides much stronger evidence.
Extreme outcomes (both high and low) are more likely to be found in small than in large samples.
Answer to Exercise 6
The study by psychologist Frances Rauscher was based upon observations of just 36 college students. In just one test students who had listened to Mozart “seemed” to show a significant improvement in their performance in an IQ test. This was picked up by the media and various organizations involved in promoting music. However, in 2007 a review of relevant studies by the Ministry of Education and Research in Germany concluded that the phenomenon was “nonexistent”.
Answer to Exercise 7
This probably makes intuitive sense to you. It is easy to construct a causal story that explains how small schools are able to provide superior education and thus produce high-achieving scholars by giving them more personal attention and encouragement than they could get in larger schools. Unfortunately, the causal analysis is pointless because the facts are wrong. If the statisticians who reported to the Gates Foundation had asked about the characteristics of the worst schools, they would have found that bad schools also tend to be smaller than average. The truth is that small schools are not better on average; they are simply more variable. If anything, say Wainer and Zwerling, large schools tend to produce better results, especially in higher grades where a variety of curricular options is valuable.
Answer to the Bonus Exercise
The intuitive answer—“of course not!”—is false. Because the events are independent and because the outcomes head (H) and tail (T) are (approximately) equally likely, then any possible sequence of six coin flips is as likely as any other. Even now that you know this conclusion is true, it remains counterintuitive, because only the third sequence appears random. As expected, HTHHTH is judged much more likely than the other two sequences. We are pattern seekers, believers in a coherent world, in which regularities (such as a sequence of six heads) appear not by accident but as a result of mechanical causality or of someone’s intention. We do not expect to see regularity produced by a random process, and when we detect what appears to be a rule, we quickly reject the idea that the process is truly random. Random processes produce many sequences that convince people that the process is not random after all.
Messages and feedback are welcome via email.