Shopping.com’s Spider Discovered

by comparisonengines, on October 3rd, 2006

This opens up a big can of worms, but I’m very proud of SDC for releasing this.

http://www0.shopping.com/bot.html

As part of our ongoing efforts to improve the buying experience for shoppers online, Shopping.com is experimenting with new ways to collect and aggregate data through web crawling. At this point we do not plan on integrating inventory from our web crawling index with inventory that we receive from our merchants.

More later…oh, and did you see Become’s new design?

Update: I was alerted to SDC’s spider by Vic Berggren who runs a myriad of sites. He discovered the crawler on Interlink Communications Systems, but Vic is also involved with New Balance Tampa and the excellent ShoeStock blog. In his first month writing the blog, he had 3,600 unique visitors and 5,500 page views.

Tags: Comparison Engines


paulobrien said

So what is this saying?
How often will sdcresearchlabs-testbot access my web pages?
For most sites, sdcresearchlabs-testbot shouldn’t access your site more than once every few seconds on average.
They’ll crawl stores every few seconds… or not more than that…. so they aren’t answering the question…
If not daily, we’ll have problems

By the way, I just wrapped up my 3 part series about optimizing CSEs and summarizing some new CSE research. Check it out
http://seobrien.blogspot.com

October 3rd, 2006 at 7:05 pm |

Vic Berggren said

Thanks for the mention(s) Brian!

I can analyze the crawl and see if a pattern emerges regarding frequency.

What’s interesting is that the site they crawled does not have an SDC PPC agreement. So, they were able to find the site (somehow) and locate the products.

Do you suspect that we may begin to see un_paid_for data at the bottom of a standard price comparison grid… sorta like ‘web results’?

October 4th, 2006 at 3:34 am |

Vic Berggren said

OK, they definitely crawled again today… roughly 1200 pages. Smart crawler too. It does not deviate too far from the product page at all and when it does its after the other category bits or a Data Sheet/Tech Spec for some of our Networking Products.

It’s also multi-threaded; grabbing between 8-10 pages per group.

October 4th, 2006 at 5:07 pm |