Ask Dave Taylor: Tech and Business


What's the inside scoop on Google Sitemaps?

Dave, I just read about Google having some sort of sitemap feature that lets you ensure that their spider sees all of your pages. I kinda wonder whether this is a good thing or not. What have you been hearing about it?

I admit, I haven't had much time to dig into exactly what it is and what it means, but I am lucky that my colleague Michael Motherwell, head of the Austrailian WMS Consulting, shared his own thoughts on this new Google feature. Here's what he had to say:

The goal of SiteMaps is simple: save costs, improve their product, generate buzz.

As background, the single hardest part of a search engine is the crawl scheduler. Try to think of a programme that permanently runs, has to schedule 6 billion documents for crawling, with some crawled every five to ten minutes, some once a month and many inbetween, with multiple rules for NOT crawling pages and a list to crawl that grows every second haphazardly, all utilising thousands of servers and, well, my head spins.

The key to better crawling has always been...