What is Baiduspider?

Baidu's official web crawler

Posted by David on 2011-01-19

1. What is Baiduspider?

“Baiduspider” is the official name of Baidu’s web crawling spider. It crawls web pages and returns updates to the Baidu index.

2. What are the user-agents of Baiduspider?

Baidu’s Products User-agents
Baidu Web/Mobile Search Baiduspider (the same for web and mobile crawlers)
Baidu Image Search Baiduspider-image
Baidu Video Search Baiduspider-video
Baidu News Search Baiduspider-news
Baidu Bookmark Search Baiduspider-favo
Baidu Business Search Baiduspider-ads
Baidu Union Search Baiduspider-cpro

3. I don’t want Baiduspider to index my website, what should I do?

You may have a website that doesn’t target the Chinese audience so being crawled by Baiduspider is a waste of your bandwidth. Good news for you – Baiduspider obeys robots.txt, so some simple commands in your robots.txt file can help you out. For example, you don’t want your site to be indexed by Baiduspider, you can use the following:

a. To block all spiders from Baidu:

User-agent: Baiduspider
Disallow: /

b. To block Baidu Video spiders:

User-agent: Baiduspider-video
Disallow: /

4. How can I know if someone is faking Baiduspider to crawl my website?

a. On Linux:

You can resolve IP addresses to hostname, to check if the hostname format is “*.baidu.com”. If not, it is a fake Baiduspider

b. On Windows:

Start – Run – input “tracert xxx.xxx.xxx.xxx (the IP address)”, then check if the hostname is in the format of “*.baidu.com”

5. How does Baiduspider work?

When Baiduspider comes to a web page, it: 1. crawls the web page and put it in storage; 2. adds the links on your page into its list to check later. This is no different than other search engine bots’ crawling activities, such as Googlebot. Baiduspider sets the crawling frequency based on the server load so usually it doesn’t cause any load problem to the server.

6. Does Baiduspider prefer servers located in China?

Baiduspider’s access to your website is very similar with a real visitor. If a visitor based in China has fast access to the website, so does Baiduspider.

I am sure this post doesn’t cover all the questions about Baidu and Baiduspider you may have, feel free to leave a comment, 🙂

At the mean time, you may also directly contact Baidu via Email at [email protected], if you have any indexation issues on your site.

Related posts:

baidu seo guide

Stay up to date