What's the inside scoop on Google Sitemaps?
Dave, I just read about Google having some sort of sitemap feature that lets you ensure that their spider sees all of your pages. I kinda wonder whether this is a good thing or not. What have you been hearing about it?
I admit, I haven't had much time to dig into exactly what it is and what it means, but I am lucky that my colleague Michael Motherwell, head of the Austrailian WMS Consulting, shared his own thoughts on this new Google feature. Here's what he had to say:
The goal of SiteMaps is simple: save costs, improve their product, generate buzz.
As background, the single hardest part of a search engine is the crawl scheduler. Try to think of a programme that permanently runs, has to schedule 6 billion documents for crawling, with some crawled every five to ten minutes, some once a month and many inbetween, with multiple rules for NOT crawling pages and a list to crawl that grows every second haphazardly, all utilising thousands of servers and, well, my head spins.
The key to better crawling has always been...
I admit, I haven't had much time to dig into exactly what it is and what it means, but I am lucky that my colleague Michael Motherwell, head of the Austrailian WMS Consulting, shared his own thoughts on this new Google feature. Here's what he had to say:
The goal of SiteMaps is simple: save costs, improve their product, generate buzz.
As background, the single hardest part of a search engine is the crawl scheduler. Try to think of a programme that permanently runs, has to schedule 6 billion documents for crawling, with some crawled every five to ten minutes, some once a month and many inbetween, with multiple rules for NOT crawling pages and a list to crawl that grows every second haphazardly, all utilising thousands of servers and, well, my head spins.
The key to better crawling has always been...
posted by Dave Taylor at
11:56 PM
