Brian SEO, Web Dev

SEO Tool: Bulk HTTP Header Response Checker

I was working in Google Webmaster Tools, looking at a (sadly) large list of 404 errors.  I found that there was an issue with the URL to a category, and thus the subsequent pages in that category ended up with bad URLs as well.  So I corrected the category level URL, tested a few URLs and all seemed good.  But Google was telling me that I had 320 URLs with issues…  Before I checked the “Marked as Fixed” button on all of the URLs, I wanted to verify that all 320 issues were fixed.

That’s a lot of clicking and page checking…

There has to be a better way, right?  So I did a bit of Googleing for a bulk link checking tool.  I came up with nothing that would do the trick.  Sad.  And odd.

The Solution

Never the less, I did find a solution to my problem – just build it.  It’s a pretty simple task really, right?  I just need a bulk list of URLs, reach out to the server and check the header response, and that’s it, right?  Well the answer is yes – the hiccup in that though is I learned that PHP errors when you attempt multiple get_headers requests (despite them having multiple functions to do this).  So rather than using a PHP loop, I had my PHP loop call AJAX requests that will each do the PHP call individually.

And to not be greedy and keep the solution to my self, here it is!  Feel free to use as you need.  It’s not pretty, and I’m aware I could do some polishing up around the edges, but it does work.  So good enough, right?

Use the Bulk HTTP Request Response Tool
Check the HTTP Header Response from a large list of URLs at once. No need to load the page, click page by page, etc. This will give you a quick snapshot of which URLs need further investigation and which can be ignored.
Check the HTTP Header Response from a large list of URLs at once. No need to load the page, click page by page, etc. This will give you a quick snapshot of which URLs need further investigation and which can be ignored.

Usage Notes

  • URLs need to be complete – starting with http://
  • Separate URLs with a semi-colon (;) and no spaces.  This results in one really long string of URLs.
  • Like I said, I used this for checking my Google Webmaster Tools URL Errors report.  So I downloaded the data to Excel, then used Excel and/or a text editor to create the long string of URLs separated by the semi-colon.

Update

A few enhancements have been made since this post was written:

  • You can separate URLs using multiple delimiter methods and select which it is via a radio button.  Options include a New Line (carriage return), comma, semi-colon, or tab.
  • The tool now adds http:// to the beginning of a URL if it is not present
  • Another counter was added to show how many need to be checked.  If you are processing a large number of URLs this will allow you to know when the tool has completed.