Being unable to get direct feedback on their campaign must be the worst nightmare of anyone involved with ad agencies. With recent changes to Google’s data policy, many webmasters lost the insight on keywords that run traffic to their websites. Why did it happen in the first place and is it really such a bad thing?

It all started in early 2012 – people began to notice a strange behavior of their analytics reports. There was a mysterious new element in the list of keywords that their visitors used to Google up their page. Depending on the software used (Analytics, Piwik, etc.), it came up as (not provided), (keyword not specified), or something smilar.

At first, most people ignored it, as it only accounted for a slight percentage of their search engine traffic. But as time passed, what once was a small fraction, grew to larger proportions, ultimately covering more than a half of their visits.

What does a “not provided” keyword mean?

In order to understand what happened, we first have to look at some of the technicalities that run the Web. All the pages you visit with your browser are powered by a protocol called HTTP. That’s why most addresses begin with http://, even though you might not realize it, as your computer fills that in for you automatically.

HTTP is a language used by the parties involved in delivering a website to your screen: your web browser (Chrome, Firefox, Safari, etc.), and the web server you are accessing using the website’s address. It is not to be confused with HTML, which describes how a website should be rendered once it’s been received by your client.

A part of the HTTP protocol governs what happens, when you click on a link while browsing the net. You might not be aware of that, but this new request that your computer is sending to the other page you just decided to visit bears information about the previous site you were looking at – the one that displayed the link you clicked. This information is called HTTP Referrer, and has been a part of the World Wide Web since the early days.

Imagine you arrived at my blog, because you clicked on a link mentioned in an online article at Reddit. What your browser sent to our server would be something like: “Hello, frostseo.com, please show me the blog article about Keywords Not Provided. Oh, and by the way – I was sent here by reddit.com”. Now, let’s say it wasn’t Reddit, but Google search results that brought you to my door. Try googling for any phrase of your choice using Private Browsing: if you look closely, you’ll see that your search keywords are mentioned in the URL of the search results page.

That’s how webmasters used to track which phrases are popular amongst the people that Googled them up. They simply looked at the Referring URLs and extracted the information about the original search queries. It was a simple and effective technique, and the statistics obtained as a result were priceless to anyone involved in online businesses.

But then something changed. Suddenly, some of the Google Referring URLs became scrambled and the information about the keywords was no longer available. Those visits were then reported as the infamous Keyword “Not Provided” in analytics reports.

Social Networking & Privacy Policies 

As it turned out, it was a direct consequence of changes to Google Privacy Policy introduced few months before. Back then, Google had just decided to jump on the social networking wagon, and was in the stage of refurbishing its products portfolio. Some of them, which didn’t comply with the long term vision of the search engine mogul, had to be wiped off, in certain cases causing a very loud cry of disappointment from their communities (R.I.P. Reader). Others began a long transformation which was supposed to prepare them for future integration with G+, something we experienced later in 2013 with YouTube.

One of the major steps in the process was to unify all those different services under one Google account. This required a serious revamp to the Terms & Conditions, which affected a global Privacy Policy for all Google services, including the search engine. As the relentless expansion of G+ continued to grow, so did the number of people who were constantly logged in to their Google accounts. And every time they search for something with Google, their search phrases remained private and therefore – not shared with the websites they were seeking.

Whether this truly was a step to protect user privacy or a deliberate measure to obscure certain bits of information is still a matter of ongoing debates. I personally think that with the growing focus on monetization of Google’s services, somebody finally asked why should they pass on the revenue by giving up for free something of incredible value.

And indeed it is very precious data – its importance can be confirmed by anyone even remotely involved with the ad business. No matter if it’s leaflets on a sidewalk or TV commercials, to monitor how your audience responds to various parts of your campaign has always been the key to optimizing its costs and efficiency.

What can be done?

Since the very first days when the “not provided” keywords appeared on the scene, people began to wonder how to work around it. And it’s not a small community we’re talking here – there are billions of Internet users these days, and most of them use Google. It’s now been a few years since the keywords were obscured and so far the only result of the global hype to crack it were countless articles stating the obvious and sharing the same ideas.

Why? The answer is quite simple: the original keywords have not been encrypted in the Referrer URL. They’re not hidden, encoded or obfuscated. They are simply gone. When you click on a search result link, you are not taken directly to the website it shows. You first go to a proxy page, which collects the statistical information about your visit, and it’s only this page that redirects you to your target. Obviously, the URL of this proxy does not mention any keywords, and so there is nothing to extract from the HTTP Referrer – apart from the fact, that this is a visit which originated at Google search result page.

Understanding that fact should point you in the right direction when it comes to handling search engine statistics. Instead of trying to analyze the structure of Google’s links which can be changed any day and won’t get you the keywords anyway, focus on the users. Once a visitor comes to your website, he becomes a way more valuable asset than just a statistic. You can learn a lot about your customer’s needs and preferences from simply watching their behavior as they are browsing through your website. There are two major areas of interest for you to pay attention to:

What made them come to your website? Something must have attracted them to click on a link pointing to your domain. If you provide a high quality and unique content, you should be able to tell a lot by simply looking at the most popular landing URLs (i.e. the starting points of visits to your website). Rather than clogging every bit of content on your homepage, use browser-friendly URLs and arrange your website accordingly. Then, run smart SEO & PPC campaigns, that focus on delivering content matching your customers demands, rather than simply boosting your position for most popular phrases.

What are they looking for once they get there? You don’t need complex Analytics filters in order to monitor your visitors preferences. Before you spend hours looking at the charts, make sure that your offer is presented in a clear and easy to understand fashion. You can spend millions on ad campaigns and advanced log tracking, but at the end of the day, if you don’t provide your clients with what they are after, there won’t be many benefits from your actions to your business. If you’re interested in Free Website Optimization, we have SEO packages which have it included. You have to improve your website not with Google robots in mind, but human beings.

Nurturing these two aspects should make your page easy to read for both your customers and your analytics software. It will give you an easy insight as to what people are into by following their paths, starting with the door they used to enter.

The difference between optimizing your website vs. pumping money into campaigns solely based on keywords popularity is conversion ratio, tightly related to ROI. Think about running a high street shop. You can spend infinite amounts on money trying to get people in, but it’s what they do once they’re inside that really affects your income. Realizing that fact leads to a surprising conclusion – by keeping the keywords to themselves, Google might just have done us all a big favor.