木公的互联网观察

    本文记录了世界较著名的Robots.txt列表所需设置的搜索蜘蛛名称。如何设置哪个目录不被搜索引擎收录的操作方法请参照设置；当然必须通过Robots.txt；下列为比较出名的搜索引擎蜘蛛名称：

Google搜索引擎蜘蛛统称:Googlebot
Google=Google.com
Googlebot/1.0=Google.com
Googlebot/2.1= Google.com
Googlebot/Test=Google.com
googlebot@googlebot.com=Google.com
Googlebot=Google.com
Googlebot-Image/1.0=Google.com Image Bot

百度搜索引擎蜘蛛统称：baiduspider
Baiduspider+=Baidu.com
BaiDuSpider=Baidu.com

Yahoo搜索引擎蜘蛛统称：Yahoo Slurp
Inktomi Search=Yahoo.com
Inktomi=Yahoo.com
Mozilla/5.0 (compatible; Yahoo! Slurp;)=Yahoo.com
Slurp.so/1.0= Yahoo.com
Slurp/2.0j=Yahoo.com
Slurp/2.0-KiteHourly=Yahoo.com
Slurp/2.0-OwlWeekly=Yahoo.com
Slurp/3.0-AU=Yahoo.com
slurp@inktomi.com=Yahoo.com
slurp@inktomi= Yahoo.com
Slurp=Yahoo.com
spider@aeneid.com=Yahoo.com
www.inktomisearch.com=Yahoo.com
YahooSeeker/CafeKelsa=Yahoo.com

MSN搜索引擎蜘蛛统称：Msnbot
MSN=MSN.com
MSNBOT/0.1=MSN.com
msnbot=MSN.com
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)=MSN.com
Altavista搜索引擎蜘蛛统称：Scooter
Scooter/1.0 scooter@pa.dec.com=Altavista.com
Scooter/1.0=Altavista.com
Scooter/1.1 (custom)=Altavista.com
Scooter/2.0 G.R.A.B. V1.1.0 =Altavista.com
Scooter/2.0 G.R.A.B. X2.0=Altavista.com
Scooter/3.3.QA.pczukor=Altavista.com
Scooter/3.3.vscooter=Altavista.com
Scooter/3.3_SF=Altavista.com
Scooter/3.3=Altavista.com
Scooter_bh0-3.0.3=Altavista.com
Scooter_trk3-3.0.3=Altavista.com
Scooter=Altavista.com
Scooter2_Mercator_3-1.0=Altavista.com
Scooter2_Mercator_x-x.0=Altavista.com
Scooter-3.0.EU=Altavista.com
Scooter-3.0.FS=Altavista.com
Scooter-3.0.HD=Altavista.com
Scooter-3.0.VNS=Altavista.com
Scooter-3.0QI=Altavista.com
Scooter-3.2.BT=Altavista.com
Scooter-3.2.DIL=Altavista.com
Scooter-3.2.EX=Altavista.com
Scooter-3.2.JT=Altavista.com
Scooter-3.2.NIV=Altavista.com
Scooter-3.2.SF0=Altavista.com
Scooter-3.2.snippet=Altavista.com
Scooter-3.2=Altavista.com
Scooter-3.3dev=Altavista.com
Scooter-ARS-1.1=Altavista.com
Scooter-ARS-1.1-ih=Altavista.com
scooter-venus-3.0.vns=Altavista.com
Scooter-W3.1.2=Altavista.com
Scooter-W3-1.0=Altavista.com
AltaVista-Intranet=Altavista.co.uk

Lycos搜索引擎蜘蛛统称： Lycos_Spider_(T-Rex)
Lycos_Spider_(T-Rex)/3.0=Lycos.com
Lycos_Spider_(T-Rex)=Lycos.com
lycos=Lycos.com

Alltheweb搜索引擎蜘蛛统称： FAST-WebCrawler/
Fast Crawler Gold Edition=Alltheweb.com
FAST Data Search Crawler=Alltheweb.com
FAST Enterprise Crawler 6 (Experimental)=Alltheweb.com
FAST Enterprise Crawler 6 used by FAST (FAST)=Alltheweb.com
FAST Enterprise Crawler 6 used by FAST=Alltheweb.com
FAST Enterprise Crawler 6=Alltheweb.com
FAST Enterprise Crawler/6 (www.fastsearch.com)=Alltheweb.com
FAST Enterprise Crawler/6=Alltheweb.com
FAST FirstPage retriever (compatible; MSIE 5.5; Mozilla/4.0)=Alltheweb.com
FAST FirstPage retriever=Alltheweb.com
Fast PartnerSite Crawler=Alltheweb.com
FastCrawler 3.0.1 (crawler@1klik.dk)=Alltheweb.com
FastCrawler 3.0.1=Alltheweb.com
FAST-WebCrawler/2.2.10=Altheweb.com
FAST-WebCrawler/2.2.6=Altheweb.com
FAST-WebCrawler/2.2.7=Altheweb.com
FAST-WebCrawler/2.2.8=Altheweb.com
FAST-WebCrawler/3.2 test=Altheweb.com
FAST-WebCrawler/3.2=Altheweb.com
FAST-WebCrawler/3.3=Altheweb.com
FAST-WebCrawler/3.4/Nirvana=Altheweb.com
FAST-WebCrawler/3.4/PartnerSite=Altheweb.com
FAST-WebCrawler/3.5=Altheweb.com
FAST-WebCrawler/3.6/FirstPage=Altheweb.com
FAST-WebCrawler/3.6=Altheweb.com
FAST-WebCrawler/3.7/FirstPage=Altheweb.com
FAST-WebCrawler/3.7=Altheweb.com
FAST-WebCrawler/3.8/Fresh=Altheweb.com
FAST-WebCrawler/3.8=Altheweb.com
FAST-WebCrawler/3.x Multimedia (mm dash crawler at fast dot no)=Altheweb.com
FAST-WebCrawler/3.x Multimedia=Altheweb.com
FAST-WebCrawler=Alltheweb.com
crawler@fast.no=Alltheweb.com

INKTOMI搜索引擎蜘蛛统称：Slurp
BSDSeek/1.0=Inktomi.com

如需要参考的可以参照本文：
User-agent(用户代理设置):(蜘蛛名称)
拒绝：(文件名字)

User-agent: Black Hole
Disallow: /
User-agent: Titan
Disallow: /
User-agent: WebStripper (蜘蛛名称)
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)

作者：木公 的互联网观察 => Blank
备注：未经许可 谢绝任何形势转载...

Tags: WebStripper, 搜索引擎, 蜘蛛