Flaws in Measuring Happiness

Can your happiness be measured by a number? For many the emotion of happiness is as subjective as the individual. Yet, we see plenty of grocery store check-out magazines touting new studies showing you how to improve your happiness in just a few simple steps. Amazingly it is always 10 or fewer, and what could be simpler to do than 10 things! However, what we have learned recently is that many of those happiness studies fell victim to a data dupe.

First, let’s be fair measuring happiness in a scientific sort of way is very difficult. It involves subjectivity. There is no proper lab test, and the phrase “individual results may vary” could not be overapplied to this very human experience. Putting a number to “how happy are you” today could be different tomorrow for little-known reasons. Further, the top level of happiness for one person could be very different from another. To put this to a test, try asking a large group of people the question “What is the happiest you have ever been” and observe the range of answers. Our life experiences and stages of life help calibrate our measures of happiness – it becomes relative to our prior experiences and observations of others. The point here is not to dig too deep into the psychology of happiness, rather the challenges of bringing a scientific statistical measure to the term “happy”, which is where the data dupe begins.

Chances of Being Wrong
In most academic studies the scientist is attempting to show a relationship between one thing and another. Heating water and its resulting temperature are simple cause-and-effect measurements – heat causes water to get hot. It is solid science and part of the scientific process many of us learned in middle school where we test an idea (hypothesis) and measure if it is true. The statistics part is when several observations of the test are measured and compared to the result. And when the results confirm the cause-and-effect of the original idea, a ‘statistically significant’ conclusion can be made that A causes B, that heat causes water to be hot. But what are the chances that the conclusion is wrong?

Data can be flawed and messy as we often point out in our book Data Duped: How to Avoid Being Hoodwinked by Misinformation. What if the observations of the test were just lucky numbers? What if there was no relation between A and B and the test simply, and conveniently lead to the false conclusion? Data scientists thought about that too and they solved this with another measure of the probability they were wrong and in an act of non-creativity called it the probability-value or P-Value for short.

Flawed Happiness
The P-value is the academic gold standard of scientific papers. It says to the readers that given a reasonable number of observations the conclusions are true, with a small chance they are completely wrong. Usually, that small chance is about 5%. This sounds great until you get to messy, difficult to interpret data, like the measure of happiness.

It should all make us pause the next time we see simple answers to complex problems.

Researchers Elizabeth Dunn and Dunigan Folk of the University of British Columbia recently discovered that many studies of happiness are flawed. The flaws in the research came mostly from researchers manipulating the data after their studies concluded in an attempt to reduce their chances of being wrong (P-value). A technique referred to as ‘cherry picking’ data. Another flaw given the range of subjectivity in happiness was the small number of tests. With fewer tests sometimes it is easier to make false conclusions… and be misled by the data.

Dunn and Dunigan have opened new scrutiny not only of how to measure happiness but maybe of all academic studies that stand on statistical evidence when using a small number of test cases. It should all make us pause the next time we see simple answers to complex problems. Complex problems like losing weight, being a better you, and solving happiness require a lot more than you can get at a grocery store checkout.

~

BONUS: What (Maybe) Makes You Happy
According to the 494 peer reviewed studies compiled by Dunn and Dunnigan…