(Facebot|Pinterestbot|LinkedInBot|Twitterbot|Qwantify|DataForSeoBot|AhrefsBot|SemrushBot|MJ12bot|DotBot|WebMeUp|NetEstate|Datanyze|Ubermetrics-Technologies|Technologies|Ezooms|CCBot|AlphaBot|LexxeBot|ProCogBot|AlexaCrawler|Iaskspider|Applebot|NaverBot|Gigabot|Cliqzbot|Adidxbot|Archive.org_bot|SeznamBot|Slurp|DuckDuckBot|Exabot|YandexBot|Googlebot)

(AhrefsBot|AwarioBot|BLEXBot|Barkrowler|CensysInspect|Criteo|DataForSeoBot|DigExt|DnyzBot|DotBot|ExtLinksBot|Ezooms|FlightDeckReports|Go-http-client|Grapeshot|Heritrix|HttpClient|HubSpot|InternetMeasurement|Knowledge|Linguee|MJ12bot|MauiBot|MegaIndex|RepoLookoutBot|SemrushBot|SurdotlyBot|Web-Crawler|WellKnownBot|Yellowbrandprotectionbot|ZoominfoBot|axios|fasthttp|github|libcurl|paloaltonetworks|python|seokicks|serpstatbot|webprosbot|amazonbot|Amazonbot)


(ubuntu|Censys|censys|CensysInspect|Java|ubuntu|python-requests|ChatGLM|GPTBot|openai|gptbot|iaskspider|internet-measurement|msray-plus|i686|ArchLinux|CrOS|nb-NO;|en-US|es-ES|Googlebot|GoogleOther|Adsbot|AhrefsBot|amazonbot|Apache-HttpClient|ArchLinux|AwarioBot|axios|facebook|fasthttp|FlightDeckReports|FreeBSD|MSIE|RepoLookoutBot|Trident|YandexBot)


(HTTrack|Apache-HttpClient|harvest|audit|dirbuster|pangolin|nmap|sqln|hydra|Parser|libwww|BBBike|sqlmap|w3af|owasp|Nikto|fimap|havij|zmeu|BabyKrokodil|netsparker|httperf| SF/)





--------有种无耻是替竞争对手（反向攻击）、站长：无用蜘蛛的站点提交申请--------


有重复，请自行删选过滤。懂的自然知道干嘛用的~