, a search engine and its corresponding User-Agent
Netscape noble baby bot
so, what are the current search engine and its corresponding User-Agent? Here, I list some, for reference.
Looksmart Web Pages Slurp
Yahoo Web Pages noble baby bot
Robots.txt file is a file website, it is to see the search engine spiders. The search engine spider crawling we >
AOL Search Slurp
search engine User-Agent
The basic concept of
Baby baby bot
webmaster work is beautifully designed website, show the rich and colorful content for the public website. Of course, we also hope that careful design of the website to obtain the ideal place, which requires us to study the search engine ranking rules, the maximum opportunity to show to the customer. However, many types of search engines, sometimes we rank to a search engine is very good, but can not get the same ranking in the search engine, the reason is not the same rules of various search engines. Therefore, people have to replicate the same content to cope with the different search engine ranking rules. However, once the search engine found station there are a large number of "clone" page will give punishment, not included in the repeat of the page. On the other hand, our website belongs to the personal file, do not want to expose in the search engine. At this time, robot.txt is in order to solve these two problems.