What Googlebot “Sees” On Your Website
Google recently released a new tool via “Google Webmaster Tools” within the “labs” section. The new tool is “Fetch as Googlebot.” Again, google has proven to ROCK as they give away these tools to us web goobs who care deeply about providing our customers with websites that deliver great/relevent search results. Now you can see exactly what Googlebot sees. (which is uber important) Within this tool you can also use “Malware details,” which shows any nasty malware hiding in your code.
To use Fetch as Googlebot, you simply log into your Webmaster Tools > Labs > Try Fetch as Googlebot > Sites (choose site) enter URL. Done. From there Googlebot will fetch your results.. usually within miniutes. (depending on the amount of content) Simply refresh that page to see the progress.)
Click the Success link once it’s been processed to see the results.
What is different between fetching as Googlebot from simply viewing the source code?
* #1, can see the HTTP header information :: normally is not contained in the source code itself (since that information is coming from the server)
* #2, Users can now tell if the server is returning any of the page information differently than what has been coded within each page.
* #3) Users can check if the server is returning something different to Googlebot than what other users see. This tool uses the same user-agent and IP range as Googlebot when it crawls the web, so if the server is configured conditionally for user agent or IP address (also referred to as “cloaking”), users will see what’s served to Google.
* #4) Now users can test changes (specifically elements such as redirects) all done in real-time.
Note that the tool doesn’t provide the content that Google is able to extract from the page. For example, if the page contains JavaScript, the raw JavaScript code will be seen as it’s contained on the page, not the rendered view visible in the browser. This means users can not use this tool to determine if Google is able to access the content in rich markup.
Cloaking?
This tool can help you determine if the pages are being cloaked to Google. This may be useful if you’re taking over a project and the original code is not yours. It can also help uncover if your site has been hacked. Using this feature will site owners verify that there are no hidden links in the page that Google actually fetches.”
How do I test redirects?
If you’ve implemented redirects, you can use this tool to test how Googlebot will interpret those redirects without waiting for those pages to be crawled. this is particularly useful when testing correctly implemented 301 redirects.
(Important: Fetch as Googlebot as of now has a 100kb per page limit of info it crawls. again… unlike Googlebot’s normal indexing of sites. Google is wants your feedback if their users find the size restriction is too limiting.)
