Bad Surveys

Yet another survey concludes that people are pretty dull when it comes to finances. This time it’s the Teachers Insurance and Annuity Association (TIAA) Institute who asked just over a thousand Americans 28 financial questions. The respondents didn’t do very well. But sometimes, it’s the designers of the study who are dull.

A Wall Street Journal article quotes one of the survey’s 28 questions:

There’s a 50/50 chance that Malik’s car will need engine repairs within the next six months which would cost $1,000. At the same time, there is a 10% chance that he will need to replace the air conditioning unit in his house, which would cost $4,000. Which poses the greater financial risk for Malik?

Anyone mathematically inclined sees instantly that the expected cost is $500 for the engine and $400 for the air conditioner. But the question is which potential repair “poses the greater financial risk for Malik?”

In the field of assessing threats and vulnerabilities, “risk” is defined as the product of probability and the amount of loss. This is the same as the expected value of the loss. Based on this definition, we would choose the engine as the greater risk.

In finance, we usually use standard deviation as the measure of “risk.” For the engine the standard deviation is $500, and for the air conditioner the standard deviation is $1200. Few people would do this exact calculation, but they may understand it intuitively. A 10% chance of a $4000 cost seems riskier than a 50% chance of a $1000 cost, and the math backs up this feeling. Based on this definition we would choose the air conditioner as the greater risk.

In case the argument based on standard deviation isn’t compelling enough, imagine that we replace the potential air conditioner cost with a 0.1% chance of losing $400,000. This is still an expected loss of $400. However, faced with a 50% chance of losing $1000 and a 0.1% chance of losing $400,000, reasonable people would focus more on the potential huge loss.

But the word “risk” isn’t owned by any one technical field. The everyday use of “risk” is imprecise and doesn’t conform exactly to either of the technical definitions.

Some people might look at this question and decide that it’s reasonable to be able to absorb a $1000 loss into their short-term finances, but $4000 would put them into a cycle of high-interest debt and digging out would take time. In this scenario, the air conditioner is the greater risk.

Another way of looking at this question is that engines will cost $1000 per year in repairs and air conditioners will cost $4000 every 5 years or so. So, engines are more expensive, and even though the word “risk” isn’t a good fit, a person who thinks about the question this way would choose the engine as more risky.

As it turns out, the survey designers think the correct answer is that the engine is riskier because its expected cost is higher. I wonder how many knowledgeable respondents understood expected cost but chose the air conditioner anyway because of their view of what “risk” means. I could easily have chosen the air conditioner had I participated in the study.

I agree that most people know too little about personal finances, but in this case, the study designers seem unable to ask clear questions.