’s Spider: Better and Faster Than Google?

By Scott


Have you heard of the “Google Sandbox Theory?” Essentially, the theory is that Google penalizes new sites by taking a while to properly index them, and then temporarily boosts them in the rankings only to push them artificially low afterwards (presumably this feeds into their ranking algorithm).

We recently launched a new website for a client that should not have had any trouble getting into the Google index. It’s all “static,” there are no programming tricks, and there is a good sitemap to help the spider get through all the site pages.

In several months, Google has yet to index the site at all! Yahoo has pulled a couple of pages. But the new search engine has indexed the whole site!


We have another client who has an e-commerce website (it’s dynamic, not static, which means the “pages” of the site are built on the fly, and it’s probably a complicated site for the search engine spiders to index). The site has been around for many years, and is very popular in its product category. No aggressive SEO tactics are used on the site that would get it penalized in any search engine.

Google has not indexed the full site yet, but the new MSN search engine has indexed over 800 pages of the site!


Based on very anecdotal evidence,’s search engine seems to index new websites faster, and is better at spidering complicated e-commerce websites. I’ll keep my eye out to see if this becomes a trend, or if the two examples cited above are isolated incidents.