The hidden problem with Google trends data

Have you ever tried using Google Trends for market research or machine learning?

You jump into Google Trends, type something like YouTube, and think:

“Wow this is incredibly granular. I can get data every 15–16 minutes. This is perfect for machine learning.”

So you download a month of data…..And suddenly… it’s daily.

Not ideal, but still workable.

Then you try a year.

Now everything is aggregated weekly.

Instead of 365 daily data points, you’ve got 52. That’s nowhere near enough for most time series models.

So how do we turn Google Trends data into something granular enough for machine learning?

That’s what this post is about.

We know data science is hard enough when working with metrics. Customer churn, ad clicks, cart abandons. Even the stuff collected internally is patchy with data quality leaving a lot to be desired. But they’re numbers. They behave. To a certain extent.

You know what doesn’t? People. Particularly when the pesky emotions of love are involved. You can’t turn ‘chemistry’ into a float value. 

 I tried to build a linear regression model to predict romance thinking it would be a fun Valentine’s experiment and an interesting feature engineering challenge. Instead I found inconsistencies, bias and impossible variables. Today we’re going to learn just how important domain knowledge is to data science. Welcome to my data science nightmare. 

Ever wondered how Google maps works out which route you should take to avoid the roadworks? Or how LinkedIn has an eerie habit of identifying the person you bumped into in the office earlier as a person you may know? Or how on earth Amazon thinks it can make deliveries possible via drones?

Today we’re going into the algorithms of just how I managed to plan Bart’s ideal trick or treating route around Springfield.