Five Steps Towards Delivering Better Analytic Outcomes

Mexico shows a whopping 44% infection to testing ratio for every 1 Million of its population. But, there is the issue of causality here: Are we seeing more cases because we are testing more people randomly? Or are we only testing the individuals mostly from high risk population cohorts? Or is it a mix of the two and to what degree and how do we separate out the effects? These questions and their implications are largely ignored as well in the media. It is a bit of a red herring to conclude that this number will be the same for every subsequent 1 Million as we do not know a lot about the people tested or how the testing is even done. However, we can look at a relatively large proportion of tested individuals showing a positive result as one possible indication of the likelihood of accelerated transmission across the population, and hence, a cause for concern.  
 
But the larger point is, as we add more data, more nuances keep getting added that were absent when the initial conclusions based on one global metric was considered.  And, while that is a good thing to note, in this simple example, it is also sobering given that most attention-grabbing headlines seem to be making conclusions based simply on that one global metric (too many news sources to enumerate). To repeat, none of this is to make the point that we should not take COVID-19 seriously. Far from it. Even one death or debilitation is one too many. However, it is dangerous as well to make decisions without fully looking at the context of more than one data point or source. This has the effect of misallocating precious resources that could be used for optimal treatments, rehabilitation policies, preventive care, economic relief and more. The implications of bad analyses have real world consequences for us all. And I am not even going to address here, the unseemly case of data being willfully suppressed by vested interests when accounting for national statistics.

5 best practices towards better analytical reasoning

So, the obvious next questions are, how do we not fall into this analysis trap and what steps could we take to be a bit smarter in how we look at data. There are many things we should do, and this is not meant to be an exhaustive list. But here are a few things that have worked for me. Some of these, no doubt, are more common sensical than anything else but I have lost track of the numbers of times these have been more honored in the breach than I thought possible:
 

1. Understand the question that the analysis is answering and see if you can come up with logical follow ups

   In the COVID-19 case we are trying to understand the true prevalence of positive tests, the severity of community spread, and the likely impact to a population for a finite future. Let’s take the non-COVID-19 example of customer churn.  It is one thing to simply find out how much churn has occurred in the last year. It is another thing entirely to ask who churned and when; if the churn was different across customer segments; if there are unique segment level differences in common variables that could account for these varying churn metrics; if there are seasonal differences, and so on. Asking a composite set of questions where each subsequent question is tightly related to the previous conclusions makes a big difference to the kinds and qualities of decisions taken.
    

2. Understand for yourself how your measures are defined

   Can we all be sure that a test for the virus is administered exactly the same way in each of the geographies that we are looking at, particularly at the early stages of this hitherto unseen pandemic? If tests are widely different then how do we interpret the numbers per million in each population? What do the metrics even mean when they come out of different measurements? As a  non-COVID-19 example, let us take the case of Net Promoter Score (NPS). NPS scores are the difference between the positives and the negative scores from a survey. However, a negative, in some surveys, is defined as anything 6 or lower on a scale of 1-10. Other NPS measurements have different calculations. So, how can we correctly interpret a specific NPS score if we do not know the underlying measurement logic?
    

3. ​Include other contextual data

   Often times data sets are constrained not just in volumes but also in depth and richness.  This is not all that bad if you accept that the insights delivered from the analysis are limited to the data collected.  But we can do better. For example, in the context of COVID-19 testing, we could include another data point such as when region wide lockdowns occurred. Doing so, now enables us to connect infection outcomes to another variable that now widens our ability to make clearer policy prescriptions. Or, conversely, adding more data could eliminate certain pre-conceived notions of causality that were readily assumed to be true. Either way, we get a much tighter analysis. To go beyond healthcare, on the NPS side, again we can add geospatial data, for example, to layer on top of the NPS scores to see if there are geographic variations and then inquire further as to the nature of such differences.  
    

4. Know your data sources

   In one of my recently read works of fiction, “Cloud Atlas” by David Mitchell, I was struck by a brilliant line “He who pays the historian always calls the tune”. Besides, heartily recommending this book to all you book lovers, this line is particularly relevant to how data are sourced. Despite our best efforts, most of us have been in situations where data are often corrupted by certain biases that creep into how they are collected, recorded, and enumerated. For example, a recent survey in Santa Clara County, California, where I live, showed the number of infected to be much higher than official numbers. The sample was obtained via an advertisement on social network for testing volunteers. This ignored the fact that a person at higher risk is more likely to sign up for the free test to get informed of his/her situation. Similarly, on the non-COVID side election polling is always fraught with issues when sampling bias is ignored. Imagine taking an opinion poll of only those people in your social circle and whose political stances are closely aligned with yours. Of course, the result of that poll will likely be different from the underlying actuals. Other types of biases that creep into the data include confirmation bias, model over/under fitting, and the non-inclusion of confounding variables.
    

5. Pick the right analytics for the job

   In this day and age where there are so many analytic tools that have proliferated and the rise of a resume-based culture when picking tools to do analysis, the temptation to go fancy and rogue is all too consuming. Not every problem needs to be tackled with an AI algorithm. While a large bobcat (not the wildcat but the large machine used in construction projects) and a standard handheld shovel can upturn your garden’s soil, the former could be an overkill when we are looking at a 50 square foot area. However, the bobcat is powerful and novel and the temptation to use it and endure the expense of using it (particularly when someone else is paying for it) is too much to resist. But a rational reflection would ensure that we eschew the bobcat in favor of the simple shovel to do the job as it was intended. This is a metaphor for how we choose tools for analytics. In over 90% of the cases, a simpler set of algorithmic choices can do the trick. Making this choice has the crucial benefit of making your analysis accessible to deeper outside scrutiny, wider consumption, and a greater likelihood of being operationalized correctly.  

Calls to action

Without a doubt, I have scratched only the shallow depths of best practices for analytics. There are many more that I can think of. I’d like to also learn from you and would appreciate your additions to this list. But what are some things you could do today, even in this context of the COVID-19 situation, that could make you look at data with a bit more purpose. 
 

• Look at all data that you can get and start looking at their provenance? Who created it, how were the features extracted, who sponsored the study, why is the data including the variables that it is including?
• What is the data purporting to show? Are there alternative explanations for the conclusions given in the analysis?
• Are there other data sources that needs to be added that could shed more light on the current situation?
• Could alternative decisions be made were more data be analyzed?
• Could the insights be used to convince decisionmakers to change course?

 
While I want to wish away this baleful pandemic that has been overstaying its welcome, I know wishes alone don’t count for much. I wish for more of us to look at data critically, even with a degree of cynicism, and keep asking all the hard questions that a lot of you have been asking. All this could lead us to insights that are more reflective of our current exigencies and, more crucially, being able to operationalize those insights in a manner that delivers on a maximal good. Stay safe and healthy.