【免费分享】屏蔽AI蜘蛛和防止网站文章采集方法

迪巴拉 · 发表于 2024-10-15 19:29:38

马上注册，免费下载更多dz插件网资源。

您需要登录才可以下载或查看，没有账号？立即注册

×

我从最经济实惠，简单粗暴开始说；不说废话，直接开整。

方法一：域名DNS托管到cloudflare，一键屏蔽AI爬虫
【免费分享】屏蔽AI蜘蛛和防止网站文章采集方法域名,爬虫,重启,托管,防火墙

如果访问不了cloudflare，那就需要自己搞定梯子。
（国内域名几乎不影响访问速度，有些人会觉得使用国内DNS速度快，其实速度差不多）

方法二：宝塔防火墙设置屏蔽AI爬虫（我用的是破解版宝塔，免费版不知道能不能设置）

AmazonbotClaudeBotPetalBotgptbotAhrefsSemrushImagesiftTeomaia_archivertwicelerMSNBotScrubbyRobozillaGigabotyahoo-mmcrawleryahoo-blogs/v3.9psbotScrapySemrushBotAhrefsBotApplebotAspiegelBotDotBotDataForSeoBotjavaMJ12botpythonseoCensys

复制代码

方法三：复制下面的代码，保存为robots.txt，上传到网站根目录

User-agent: AhrefsDisallow: /User-agent: SemrushDisallow: /User-agent: ImagesiftDisallow: /User-agent: AmazonbotDisallow: /User-agent: gptbotDisallow: /User-agent: ClaudeBotDisallow: /User-agent: PetalBotDisallow: /User-agent: BaiduspiderDisallow: User-agent: SosospiderDisallow: User-agent: sogou spiderDisallow: User-agent: YodaoBotDisallow: User-agent: GooglebotDisallow: User-agent: BingbotDisallow: User-agent: SlurpDisallow: User-agent: TeomaDisallow: /User-agent: ia_archiverDisallow: /User-agent: twicelerDisallow: /User-agent: MSNBotDisallow: /User-agent: ScrubbyDisallow: /User-agent: RobozillaDisallow: /User-agent: GigabotDisallow: /User-agent: googlebot-imageDisallow: User-agent: googlebot-mobileDisallow: User-agent: yahoo-mmcrawlerDisallow: /User-agent: yahoo-blogs/v3.9Disallow: /User-agent: psbotDisallow: User-agent: dotbotDisallow: /

复制代码

方法四：防止网站被采集（宝塔配置文件保存以下代码）

#禁止Scrapy等工具的抓取if ($http_user_agent ~* (Scrapy|Curl|HttpClient|crawl|curb|git|Wtrace)) { return 403;}#禁止指定UA及UA为空的访问if ($http_user_agent ~* "CheckMarkNetwork|Synapse|Nimbostratus-Bot|Dark|scraper|LMAO|Hakai|Gemini|Wappalyzer|masscan|crawler4j|Mappy|Center|eright|aiohttp|MauiBot|Crawler|researchscan|Dispatch|AlphaBot|Census|ips-agent|NetcraftSurveyAgent|ToutiaoSpider|EasyHttp|Iframely|sysscan|fasthttp|muhstik|DeuSu|mstshash|HTTP_Request|ExtLinksBot|package|SafeDNSBot|CPython|SiteExplorer|SSH|MegaIndex|BUbiNG|CCBot|NetTrack|Digincore|aiHitBot|SurdotlyBot|null|SemrushBot|Test|Copied|ltx71|Nmap|DotBot|AdsBot|InetURL|Pcore-HTTP|PocketParser|Wotbox|newspaper|DnyzBot|redback|PiplBot|SMTBot|WinHTTP|Auto Spider 1.0|GrabNet|TurnitinBot|Go-Ahead-Got-It|Download Demon|Go!Zilla|GetWeb!|GetRight|libwww-perl|Cliqzbot|MailChimp|SMTBot|Dataprovider|XoviBot|linkdexbot|SeznamBot|Qwantify|spbot|evc-batch|zgrab|Go-http-client|FeedDemon|Jullo|Feedly|YandexBot|oBot|FlightDeckReports|Linguee Bot|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|EasouSpider|LinkpadBot|Ezooms|^$" ) { return 403; }#禁止非GET|HEAD|POST方式的抓取if ($request_method !~ ^(GET|HEAD|POST)$) { return 403;}

复制代码

添加完毕后保存，重启nginx即可，这样这些蜘蛛或工具扫描网站的时候就会提示403禁止访问。
注意：如果你网站使用火车头采集发布，使用以上代码会返回403错误，发布不了的。如果想使用火车头采集发布，请使用下面的代码：

#禁止Scrapy等工具的抓取if ($http_user_agent ~* (Scrapy|Curl|HttpClient|crawl|curb|git|Wtrace)) { return 403;}#禁止指定UA及UA为空的访问if ($http_user_agent ~* "CheckMarkNetwork|Synapse|Nimbostratus-Bot|Dark|scraper|LMAO|Hakai|Gemini|Wappalyzer|masscan|crawler4j|Mappy|Center|eright|aiohttp|MauiBot|Crawler|researchscan|Dispatch|AlphaBot|Census|ips-agent|NetcraftSurveyAgent|ToutiaoSpider|EasyHttp|Iframely|sysscan|fasthttp|muhstik|DeuSu|mstshash|HTTP_Request|ExtLinksBot|package|SafeDNSBot|CPython|SiteExplorer|SSH|MegaIndex|BUbiNG|CCBot|NetTrack|Digincore|aiHitBot|SurdotlyBot|null|SemrushBot|Test|Copied|ltx71|Nmap|DotBot|AdsBot|InetURL|Pcore-HTTP|PocketParser|Wotbox|newspaper|DnyzBot|redback|PiplBot|SMTBot|WinHTTP|Auto Spider 1.0|GrabNet|TurnitinBot|Go-Ahead-Got-It|Download Demon|Go!Zilla|GetWeb!|GetRight|libwww-perl|Cliqzbot|MailChimp|SMTBot|Dataprovider|XoviBot|linkdexbot|SeznamBot|Qwantify|spbot|evc-batch|zgrab|Go-http-client|FeedDemon|Jullo|Feedly|YandexBot|oBot|FlightDeckReports|Linguee Bot|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|EasouSpider|LinkpadBot|Ezooms ) { return 403; }#禁止非GET|HEAD|POST方式的抓取if ($request_method !~ ^(GET|HEAD|POST)$) { return 403;}

复制代码

设置完了可以用模拟爬去来看看有没有误伤了好蜘蛛，说明：以上屏蔽的蜘蛛名不包括以下常见的6大蜘蛛名：百度蜘蛛：Baiduspider谷歌蜘蛛：Googlebot必应蜘蛛：bingbot搜狗蜘蛛：Sogou web spider360蜘蛛：360Spider神马蜘蛛：YisouSpider爬虫常见的User-Agent如下：

FeedDemon 内容采集BOT/0.1 (BOT for JCE) sql注入CrawlDaddy sql注入Java 内容采集Jullo 内容采集Feedly 内容采集UniversalFeedParser 内容采集ApacheBench cc攻击器Swiftbot 无用爬虫YandexBot 无用爬虫AhrefsBot 无用爬虫jikeSpider 无用爬虫MJ12bot 无用爬虫ZmEu phpmyadmin 漏洞扫描WinHttp 采集cc攻击EasouSpider 无用爬虫HttpClient tcp攻击Microsoft URL Control 扫描YYSpider 无用爬虫jaunty wordpress爆破扫描器oBot 无用爬虫Python-urllib 内容采集Indy Library 扫描FlightDeckReports Bot 无用爬虫Linguee Bot 无用爬虫

复制代码

©DZ插件网所发布的一切资源仅限用于学习和研究目的；不得将上述内容用于商业或者非法用途，否则，一切后果请用户自负。
网站部分内容来源于网络，版权争议与本站无关。您必须在下载后的24个小时之内，从您的电脑中彻底删除上述内容。如果您喜欢该程序，请支持正版软件，得到更好的正版服务。
您在本站任何的赞助购买、下载、查阅、回复等行为等均表示接受并同意签订《DZ插件网免责声明协议》。
如有侵权请邮件与我们联系处理: discuzaddons@vip.qq.com 并出示相关证明以便删除。敬请谅解！