The War between Search Engine and Website

Websits vs web super powers

Posted by Ryan on 2011-11-12

Yesterday Bill Slawski wrote a post titled “The Value of Mutant Augmented Search Results”, which tells us about how possibly search engines analyze the structure of a website and reflect it on the search results.

I expressed my worry in my comment, as I think if a search engine wants users to do more on their website, that means users do less on yours, and eventually, your users become theirs.

I can’t remember since when, but to me, “Don’t be evil” kind of sounds like saying “I didn’t eat Jimmy’s Halloween candy, it is not me!”, when nobody asks.

Google is a web super power now, because maybe 60% to 70% of your site’s traffic comes from it. When they do something that harms your interests, it is not easy for you to fight back, on and off line.

So is there a way to protect yourself?

Well I can only say that there was a way, and a live case just happened around 2 weeks ago here in China.

The search engine is, newly developed by Alibaba and focused on product search and price-comparison. The websites that fight against it are major Chinese B2C websites, including, and

Their strategy is quite simple – killing the search engine in the bud, by blocking its crawlers with robots.txt.

You may think their strategy is questionable: isn’t another traffic source a good thing for a website?

Let’s do a small risk-benefit analysis here:

Possible benefit:

A new traffic source. Sounds great, huh?

But does it equal more visitors, and more buyers? Not necessarily.

Potential risks:

The two major traffic sources for the big B2C websites, especially the Chinese ones, are 1. Direct traffic; 2. Navigational search from search engines. So a new product search engine may mean part of the users that used to go to your website directly or indirectly from a navigational search will go to the product search engine and compare prices first. Will they go to your website after comparing prices? Maybe only if your offers are cheaper.

That is the first risk. The second risk is that in terms of prices, most B2C are not the match of Taobao, the Chinese version of eBay which also belongs to Alibaba, because the prices of the B2C sites include free shipping while Taobao’s small sellers don’t offer that. This means most users of Etao, after comparing prices, will probably go to Etao’s sister site

So their decision of killing a search engine in the bud is quite wise. They don’t want the new search engine to grow into another giant that is too big to cope with.

A SEO joke here:

After people found blocked Etao’s web crawler, Mr Wu Hao, the spokesman of Etao, PUBLICLY claimed: “Technically, 360Buy is not able to completely block Etao. ”

Oh I’m so proud of you, you big web scraper!

Related posts:

Stay up to date