Contents
Background
If you’ve ever done much research into SEO (Search Engine Optimization) you most likely have found that finding accurate data isn’t always the easiest to come across. Search Engines aren’t the most forthcoming with data about what search queries generated traffic to your website, or how well your website ranks for various queries. Some of this information once was available in analytics tools, however going back as far as 2011 this data started reporting as “not provided” as search engines started to encrypt their search traffic data.
Google and Bing do provide Webmaster Tools – free website access to information about a website you own in regards to crawling and search engine traffic. In these tools they do provide some information about what search queries drove traffic to your website and the position in the search engine results page that your website was listed. While this is great information there are two limitations to it, particularly in regards to Google:
- Not all of the search traffic is displayed in Webmaster Tools
- Only 3 months of data is provided in Webmaster Tools
In mid 2015 Google opened up API access to the search traffic data of Webmaster Tools. (Bing has provided API access for some time now.) With this API access came access to more data – 5,000 rows as opposed to 1,000 provided in their Webmaster Tools.
This paved the path for me to put a dream of mine together – have all of this Search Engine data together in one location for long term historical analysis. Now to be honest I have done this in the past, utilizing Excel and what information was available at the time. But the process was slow, manual and cumbersome. I did have some automation built to pull data from the old Google Webmaster Tools reports, but it was unreliable. So putting the past in the past, as soon as the API access became available I started to build out my dream.
Organic Search Analytics Is Born!
I am happy to say I have built out my dream into a tool that I call Organic Search Analytics. Well at the time of writing this, I’ve built the basics for it, but the tool will continue to evolve. The primary goals of the project are:
- Capture search analytics data from major Search Engines (where provided)
- Store the captured data in a database long term
- Generate reports utilizing the captured data
- Provide the code to others to utilize for free so as to benefit the SEO community
And my primary goals have been achieved!
Organic Search Analytics is provided as a free codebase that others can copy to their own webservers and utilize. Organic Search Analytics is available on Github from this URL.
https://github.com/PromInc/organic-search-analytics/
How To Get Started
For many SEOs I know there is one part of the above sentence that scared them – “their own webservers”. I am aware that many SEOs may not have the web developer background that I do and thus this appears to be a hurdle and thus they have written off this tool already. But wait just a second before you jump ship – it’ll be in your benefit. If you understand this from the technology standpoint that I see, and take into consideration that I’m a website developer and not a desktop software developer, you’d understand like I do that there isn’t an alternative option that I could have built. That said, it’s actually easier than you think to build a webserver – on you own desktop/laptop computer! What this does is keeps your data readily accessible on your local machine you use daily and keeps your data secure from other website hackers that try to sniff for information, all while allowing the tools to work! A webserver can be setup many ways, but the quickest and simplest way is to use XAMPP or MAMP – these are desktop softwares that do all of the hard/confusing work for you. A simple install and the webserver limitation is a thing of the past. I encourage you to read the Organic Search Analytics installation guide for more information.
Screenshots
Here are a few screenshots of what the Organic Search Analytics web application looks like. Note: these screenshots are from version 2.2.0
The Future
Organic Search Analytics is not a complete product at the time of writing this post. I’m not sure if it ever will be. As with any web application, changes, iterations, enhancements, new features, etc. will continue to be added. Likewise, I choose to host the codebase on Github partially because it allows for community access and collaboration. This means other users can leave feedback about what could be useful enhancements to the tool, discussion on how to best implement certain features, or even assist in coding.
Overall, the goal of this tool is to become the new goto Webmaster Tools that a SEO could/should use on a daily basis. This means it needs to capture far more data from the major Search Engines than it is doing today. But the reason/motivation behind making this the primary is that it holds long term data which will allow for historical analysis. Also, pairing up data from more than one Search Engine at a time will prove to be valuable for analysis.
