Google Flu Trends — a project to use Big Data around Google searches to predict flu trends faster than the CDC — was the poster child for the glory of Big Data right up until it “failed spectacularly” in 2013. What happened?
It began as a research experiment, followed by a paper in none other than Nature:
The paper demonstrated that search data, if properly tuned to the flu tracking information from the Centers for Disease Control and Prevention, could produce accurate estimates of flu prevalence two weeks earlier than the CDC’s data—turning the digital refuse of people’s searches into potentially life-saving insights.
And then, GFT failed—and failed spectacularly—missing at the peak of the 2013 flu season by 140 percent. When Google quietly euthanized the program, called Google Flu Trends (GFT), it turned the poster child of big data into the poster child of the foibles of big data. (Source)
Researchers writing at Wired this year go back to postmortem the program, and find, they claim, that the problem was not Big Data per se, but “Big Data Hubris”. For example, Google did not make their algorithms transparent, which led to them missing problems around seasonal terms:
But while Google’s efforts in projecting the flu were well meaning, they were remarkably opaque in terms of method and data — making it dangerous to rely on Google Flu Trends for any decision-making at all.
For example, Google’s algorithm was quite vulnerable to overfitting to seasonal terms unrelated to the flu, like “high school basketball.” With millions of search terms being fit to the CDC’s data, there were bound to be searches that were strongly correlated by pure chance, and these terms were unlikely to be driven by actual flu cases or predictive of future trends. Google also did not take into account changes in search behavior over time. After the introduction of GFT, Google introduced its suggested search feature as well as a number of new health-based add-ons to help people more effectively find the information they need. While this is great for those using Google, it also makes some search terms more prevalent, throwing off GFT’s tracking.
These problems could have been easily spotted (and perhaps corrected) had GFT not been a black box alogrithm — flu researchers at the CDC are nothing if not experts in understanding spurious correlation and seasonal confounding. But the nature of the project was that only a few people could see into the black box, and for the most part they had facile understandings of the issues involved.