Don’t Be Afraid of Poor Analytics

Article main image
Dec 7, 2018

If you’ve taken any analytics or statistics courses, you’ll have been bombarded with cautions about how analytics can go wrong. For example:

  • Random outcomes appear meaningful to the untrained eye.
  • Small sample sizes give misleading results.
  • Without good control groups, you can’t be sure what caused an outcome.
  • Correlation is not causation.

We could go on at some length. In fact, sometimes it feels like statistics is all about showing that whatever analysis you did, your conclusions are sadly mistaken.

In the world of business analytics, we shouldn’t be afraid of poor analytics. The reasons are straightforward:

We need to start somewhere. If we discourage beginners from doing analytics, they will never become experts.

Some analytics is, more often than not, better than no analytics. Let’s imagine a restaurant needs to know if their chef’s special is any good. They ask the first three customers who try it. One likes it and the other two feel it’s mediocre. That’s a small sample size (n=3) but it’s all we’ve got. It could be leading us to the wrong conclusion, but it’s still a sound business decision to take the special off the menu rather than risk alienating customers.

These two reasons are not incidental. It is hard for people to do any analytics so if we allow barriers to get in the way they will give up. Furthermore, it is very rare in business to have clean, randomized control groups with large sample sizes. If we set a high bar, we will not have better analytics, we’ll have no analytics.

I hesitate to say all of this because, of course, we want people to do as high-quality an analysis as possible. We don’t want sloppy work nor do we want them to be blind to the weaknesses of their analyses. I hope I’m not seen as making an argument for poor work; I’m trying to make an argument for doing the best work possible given limited time, data and expertise.

Peter Navin, the SVP of employee experience at Grand Rounds and author of The CMO of People says, “Measure what you want to become.” By that, he means it’s better to give a poor measure of something important rather than a great measure of something trivial.

If the restaurant had a dashboard, they should have a place to report on the quality of the chef’s special. Their existing measure may not be great, but if it’s important, then let’s work with some data rather than no data; and let’s aspire to the analytics level that we want to become.