常见蜘蛛：
Baiduspider
YodaoBot
YoudaoBot
Googlebot
Googlebot-Image
Mediapartners-Google
Sogou-Test-Spider
Twiceler
Yahoo! Slurp
Yahoo! Slurp China
Yahoo!-AdCrawler
Yahoo ContentMatch Crawler
Sosospider
CollapsarWEB qihoobot
NaverBot
sogou spider
sogou in spider
Sogou web spider
Sogou News Spider
Sogou Orion spider
Sogou head spider
SurveyBot
Yanga WorldSearch Bot v
baiduspider-mobile-gate
discobot
ia_archiver
msnbot
msnbot-media
360Spider
bingbot
YisouSpider
EasouSpider
JikeSpider
EtaoSpider
YandexBot
AhrefsBot
ezooms.bot