{"id":1221,"date":"2015-12-18T10:52:17","date_gmt":"2015-12-18T04:52:17","guid":{"rendered":"http:\/\/promincproductions.com\/blog\/?p=1221"},"modified":"2015-12-18T10:54:51","modified_gmt":"2015-12-18T04:54:51","slug":"organic-search-analytics-capture-your-seo-data-from-search-engines","status":"publish","type":"post","link":"https:\/\/promincproductions.com\/blog\/organic-search-analytics-capture-your-seo-data-from-search-engines\/","title":{"rendered":"Organic Search Analytics &#8211; Capture Your SEO Data From Search Engines"},"content":{"rendered":"<h2>Background<\/h2>\n<p>If you&#8217;ve ever done much research into SEO (Search Engine Optimization) you most likely have found\u00a0that finding accurate data isn&#8217;t always the easiest to come across. \u00a0Search Engines aren&#8217;t the most forthcoming with data about what search queries generated traffic to your website, or how well your website ranks for various queries. \u00a0Some of this information once was available in analytics tools, however going back as far as 2011 this data started reporting as\u00a0&#8220;not provided&#8221; as search engines started to encrypt their search traffic data.<\/p>\n<p>Google and Bing do provide Webmaster Tools &#8211; free website access to information about a website you own in regards to crawling and search engine traffic. \u00a0In these tools they do provide some information about what search queries drove traffic to your website and the position in the search engine results page that your website was listed. \u00a0While this is great information there are two limitations to it, particularly in regards to Google:<\/p>\n<ul>\n<li>Not all of the search traffic is displayed in Webmaster Tools<\/li>\n<li>Only 3 months of data is provided in Webmaster Tools<\/li>\n<\/ul>\n<p>In mid 2015 Google opened up API access to the search traffic data of Webmaster Tools. \u00a0(Bing has provided API access for some time now.) \u00a0With this API access came access to more data &#8211; 5,000 rows as opposed to 1,000 provided in their Webmaster Tools.<\/p>\n<p>This paved the path for me to put a dream of mine together &#8211; have all of this Search Engine data together in one location for long term historical analysis. \u00a0Now to be honest I have done this in the past, utilizing Excel and what information was available at the time. \u00a0But the process was slow, manual and cumbersome. \u00a0I did have some automation built to pull data from the old Google Webmaster Tools reports, but it was unreliable. \u00a0So putting the past in the past, as soon as the API access became available I started to build out my dream.<\/p>\n<h2>Organic Search Analytics Is Born!<\/h2>\n<p>I am happy to say I have built out my dream into a tool that I call\u00a0<strong>Organic Search Analytics<\/strong>. \u00a0Well at the time of writing this, I&#8217;ve built the basics for it, but the tool will continue to\u00a0evolve. \u00a0The primary goals\u00a0of the project are:<\/p>\n<ul>\n<li>Capture search analytics data from major Search Engines (where provided)<\/li>\n<li>Store the captured data in a database long term<\/li>\n<li>Generate reports utilizing the captured data<\/li>\n<li>Provide the code to others to utilize for free so as to benefit the SEO community<\/li>\n<\/ul>\n<p>And my primary\u00a0goals have been achieved!<\/p>\n<p><strong>Organic Search Analytics<\/strong> is provided as a free codebase that others can copy to their own webservers and utilize. \u00a0<strong>Organic Search Analytics<\/strong> is available on Github from this URL.<\/p>\n<p><a href=\"https:\/\/github.com\/PromInc\/organic-search-analytics\/\" target=\"_blank\" data-lasso-id=\"574\" rel=\"noopener\">https:\/\/github.com\/PromInc\/organic-search-analytics\/<\/a><\/p>\n<h3>How To Get Started<\/h3>\n<p>For many SEOs I know there is one part of the above sentence that scared them &#8211; <em>&#8220;their own webservers&#8221;<\/em>. \u00a0I am aware that many SEOs may not have the web developer background that I do and thus this appears to be a hurdle and thus they have written off this tool already. \u00a0But wait just a second before you jump ship &#8211; it&#8217;ll be in your benefit. \u00a0If you understand this from the technology standpoint that I see, and take into consideration that I&#8217;m\u00a0a website developer and not a desktop software developer, you&#8217;d understand like I do that there isn&#8217;t an alternative option that I could have built. \u00a0That said, it&#8217;s actually easier than you think to build a webserver &#8211; on you own desktop\/laptop computer! \u00a0What this does is keeps your data readily accessible on your local machine you use daily and keeps your data secure from other website hackers that try to sniff for information, all while allowing the tools to work! \u00a0A webserver can be setup many ways, but the quickest and simplest way is to use XAMPP or MAMP &#8211; these are desktop softwares that do all of the hard\/confusing work for you. \u00a0A simple install and the webserver limitation is a thing of the past. \u00a0I encourage you to read the\u00a0<a href=\"https:\/\/github.com\/PromInc\/organic-search-analytics\/wiki\/Installation\" target=\"_blank\" data-lasso-id=\"575\" rel=\"noopener\"><strong>Organic Search Analytics<\/strong> installation guide<\/a> for more information.<\/p>\n<h2>Screenshots<\/h2>\n<p>Here are a few screenshots of what the\u00a0<strong>Organic Search Analytics<\/strong> web application looks like. \u00a0Note: these screenshots are from version 2.2.0<\/p>\n<p><figure id=\"attachment_1216\" aria-describedby=\"caption-attachment-1216\" style=\"width: 500px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytics-configuration.png\" rel=\"attachment wp-att-26\" data-lasso-id=\"576\" data-rel=\"lightbox-gallery-Ja4nElJG\" data-rl_title=\"\" data-rl_caption=\"\"><img decoding=\"async\" class=\"wp-image-1216 size-medium\" title=\"\" src=\"https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytics-configuration-500x273.png\" alt=\"organic search analytics configuration\" width=\"500\" height=\"273\" srcset=\"https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytics-configuration-500x273.png 500w, https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytics-configuration-768x419.png 768w, https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytics-configuration-1024x558.png 1024w, https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytics-configuration-150x82.png 150w, https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytics-configuration-600x327.png 600w, https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytics-configuration.png 1436w\" sizes=\"(max-width: 500px) 100vw, 500px\" \/><\/a><figcaption id=\"caption-attachment-1216\" class=\"wp-caption-text\">Configuration Page. Typically once this information is saved you won&#8217;t need to re-visit this page. But this is where your credentials get entered to allow the tool to work.<\/figcaption><\/figure> <figure id=\"attachment_1217\" aria-describedby=\"caption-attachment-1217\" style=\"width: 500px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytics-data-capture.png\" rel=\"attachment wp-att-27\" data-lasso-id=\"577\" data-rel=\"lightbox-gallery-Ja4nElJG\" data-rl_title=\"\" data-rl_caption=\"\"><img decoding=\"async\" class=\"wp-image-1217 size-medium\" title=\"\" src=\"https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytics-data-capture-500x327.png\" alt=\"organic search analytics data capture\" width=\"500\" height=\"327\" srcset=\"https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytics-data-capture-500x327.png 500w, https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytics-data-capture-768x503.png 768w, https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytics-data-capture-1024x670.png 1024w, https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytics-data-capture-150x98.png 150w, https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytics-data-capture-600x393.png 600w, https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytics-data-capture.png 1436w\" sizes=\"(max-width: 500px) 100vw, 500px\" \/><\/a><figcaption id=\"caption-attachment-1217\" class=\"wp-caption-text\">Data Capture Page. Once setup, a list of missing reports will be displayed here. This allows the user to import the days of data that are missing. A single button click can import all days of missing data as well.<\/figcaption><\/figure> <figure id=\"attachment_1218\" aria-describedby=\"caption-attachment-1218\" style=\"width: 500px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytics-settings.png\" rel=\"attachment wp-att-28\" data-lasso-id=\"578\" data-rel=\"lightbox-gallery-Ja4nElJG\" data-rl_title=\"\" data-rl_caption=\"\"><img decoding=\"async\" class=\"wp-image-1218 size-medium\" title=\"\" src=\"https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytics-settings-500x333.png\" alt=\"organic search analytics settings\" width=\"500\" height=\"333\" srcset=\"https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytics-settings-500x333.png 500w, https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytics-settings-768x511.png 768w, https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytics-settings-1024x682.png 1024w, https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytics-settings-150x100.png 150w, https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytics-settings-600x399.png 600w, https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytics-settings.png 1436w\" sizes=\"(max-width: 500px) 100vw, 500px\" \/><\/a><figcaption id=\"caption-attachment-1218\" class=\"wp-caption-text\">Settings Page. This is where a user can configure what data to capture from which search engine for each domain. As new features get added this page will have more options added.<\/figcaption><\/figure> <figure id=\"attachment_1219\" aria-describedby=\"caption-attachment-1219\" style=\"width: 500px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytis-report.png\" rel=\"attachment wp-att-29\" data-lasso-id=\"579\" data-rel=\"lightbox-gallery-Ja4nElJG\" data-rl_title=\"\" data-rl_caption=\"\"><img decoding=\"async\" class=\"wp-image-1219 size-medium\" title=\"\" src=\"https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytis-report-500x257.png\" alt=\"organic search analytis report\" width=\"500\" height=\"257\" srcset=\"https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytis-report-500x257.png 500w, https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytis-report-768x395.png 768w, https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytis-report-1024x527.png 1024w, https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytis-report-150x77.png 150w, https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytis-report-600x309.png 600w, https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytis-report.png 1436w\" sizes=\"(max-width: 500px) 100vw, 500px\" \/><\/a><figcaption id=\"caption-attachment-1219\" class=\"wp-caption-text\">Report Page. This is an example of what a report page will look like. A graph that can be zoomed in on or saved via right-click is provided as well as a chart of data.<\/figcaption><\/figure><\/p>\n<h2>The Future<\/h2>\n<p><strong>Organic Search Analytics<\/strong> is not a complete product at the time of writing this post. \u00a0I&#8217;m not sure if it ever will be. \u00a0As with any web application, changes, iterations, enhancements, new features, etc. will continue to be added. \u00a0Likewise, I choose to host the codebase on Github partially because it allows for community access and collaboration. \u00a0This means other users can leave feedback about what could be useful enhancements to the tool, discussion on how to best implement certain features, or even assist in coding.<\/p>\n<p>Overall, the goal of this tool is to become the new goto Webmaster Tools that a SEO could\/should use on a daily basis. \u00a0This means it needs to capture far more data from the major Search Engines than it is doing today. \u00a0But the reason\/motivation behind making this the primary is that it holds long term data which will allow for historical analysis. \u00a0Also, pairing up data from more than one Search Engine at a time will prove to be valuable for analysis.<\/p>","protected":false},"excerpt":{"rendered":"<p>Background If you&#8217;ve ever done much research into SEO (Search Engine Optimization) you most likely have found\u00a0that finding [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":1219,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"wprm-recipe-roundup-name":"","wprm-recipe-roundup-description":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[142,49,143],"tags":[],"class_list":["post-1221","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-projects","category-seo","category-web-applications"],"jetpack_featured_media_url":"https:\/\/promincproductions.com\/blog\/wp-content\/uploads\/2015\/12\/organic-search-analytis-report.png","jetpack_shortlink":"https:\/\/wp.me\/p4BbcR-jH","jetpack_sharing_enabled":true,"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/promincproductions.com\/blog\/wp-json\/wp\/v2\/posts\/1221","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/promincproductions.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/promincproductions.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/promincproductions.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/promincproductions.com\/blog\/wp-json\/wp\/v2\/comments?post=1221"}],"version-history":[{"count":3,"href":"https:\/\/promincproductions.com\/blog\/wp-json\/wp\/v2\/posts\/1221\/revisions"}],"predecessor-version":[{"id":1256,"href":"https:\/\/promincproductions.com\/blog\/wp-json\/wp\/v2\/posts\/1221\/revisions\/1256"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/promincproductions.com\/blog\/wp-json\/wp\/v2\/media\/1219"}],"wp:attachment":[{"href":"https:\/\/promincproductions.com\/blog\/wp-json\/wp\/v2\/media?parent=1221"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/promincproductions.com\/blog\/wp-json\/wp\/v2\/categories?post=1221"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/promincproductions.com\/blog\/wp-json\/wp\/v2\/tags?post=1221"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}