What is Baiduspider?
Update on 2011-3-31: new Baidu user-agent
1. What is Baiduspider?
“Baiduspider” is the official name of Baidu’s web crawling spider. It crawls web pages and returns updates to the Baidu index.
2. What are the user-agents of Baiduspider?
Baiduspider user-agent list Product user-agent Baidu Web Search Baiduspider Baidu Mobile Search Baiduspider-mobile Baidu Image Search Baiduspider-image Baidu Video Search Baiduspider-video Baidu News Search Baiduspider-news Baidu Bookmark Search Baiduspider-favo Baidu Business Search Baiduspider-ads Baidu Union Search Baiduspider-cpro
3. I don’t want Baiduspider to index my website, what should I do?
You may have a website that doesn’t target the Chinese audience so being crawled by Baiduspider is a waste of your bandwidth. Good news for you – Baiduspider obeys robots.txt, so some simple commands in your robots.txt file can help you out. For example, you don’t want your site to be indexed by Baiduspider, you can use the following:
a. To block all spiders from Baidu:
b. To block Baidu Video spiders:
c. To block all Baidu spiders but Baidu Mobile Search:
4. How can I know if someone is faking Baiduspider to crawl my website?
a. On Linux:
You can resolve IP addresses to hostname, to check if the hostname format is “*.baidu.com”. If not, it is a fake Baiduspider
b. On Windows:
Start – Run – input “tracert xxx.xxx.xxx.xxx (the IP address)”, then check if the hostname is in the format of “*.baidu.com”
5. How does Baiduspider work?
When Baiduspider comes to a web page, it: 1. crawls the web page and put it in storage; 2. adds the links on your page into its list to check later. This is no different than other search engine bots’ crawling activities, such as Googlebot. Baiduspider sets the crawling frequency based on the server load so usually it doesn’t cause any load problem to the server.
6. Does Baiduspider prefer servers located in China?
Baiduspider’s access to your website is very similar with a real visitor. If a visitor based in China has fast access to the website, the does Baiduspider.
I am sure this post doesn’t cover all the questions about Baidu and Baiduspider you may have, feel free to leave a comment,
At the mean time, you may also directly contact Baidu via Email: email@example.com, if you have any indexation issues on your site.Date: January 19th | Topic: Baidu Search Engine Optimization | Author: David