人工智能时代，怎么才能屏蔽语言大模型？
2025/03/14 08:31:08
网站不间断的被语言模型claude爬取，导致系统资源一直很高，robots跟本不管用。
- 18.222.5.173 - - [14/Mar/2025:01:16:52 +0800] ＂GET /Spider/?url=%2Fzheli%2F15058.html HTTP/2.0＂ 301 162 ＂-＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂18.222.5.173 - - [14/Mar/2025:01:16:53 +0800] ＂GET /Spider/?url=%2Fzheli%2F15058.html HTTP/2.0＂ 200 20 ＂https://bm.cn/Spider/?url=%2Fzheli%2F15058.html＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂18.217.118.121 - - [14/Mar/2025:01:16:57 +0800] ＂GET /lizhizhichang/14027.html HTTP/2.0＂ 301 162 ＂-＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂18.217.118.121 - - [14/Mar/2025:01:16:59 +0800] ＂GET /lizhizhichang/14027.html HTTP/2.0＂ 200 14218 ＂https://bm.cn/lizhizhichang/14027.html＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂3.129.14.196 - - [14/Mar/2025:01:17:10 +0800] ＂GET /zheli/15754.html HTTP/2.0＂ 301 162 ＂-＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂3.129.14.196 - - [14/Mar/2025:01:17:12 +0800] ＂GET /zheli/15754.html HTTP/2.0＂ 200 6412 ＂https://bm.cn/zheli/15754.html＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂18.217.194.145 - - [14/Mar/2025:01:17:18 +0800] ＂GET /Spider/?url=%2Flizhigushi%2F11859.html HTTP/2.0＂ 301 162 ＂-＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂18.217.194.145 - - [14/Mar/2025:01:17:19 +0800] ＂GET /Spider/?url=%2Flizhigushi%2F11859.html HTTP/2.0＂ 200 20 ＂https://bm.cn/Spider/?url=%2Flizhigushi%2F11859.html＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂18.222.24.148 - - [14/Mar/2025:01:17:25 +0800] ＂GET /?p=%2FDo%2Fvisits%2Fid%2F13700 HTTP/2.0＂ 301 162 ＂-＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂18.222.24.148 - - [14/Mar/2025:01:17:25 +0800] ＂GET /?p=%2FDo%2Fvisits%2Fid%2F13700 HTTP/2.0＂ 200 27 ＂https://bm.cn/?p=%2FDo%2Fvisits%2Fid%2F13700＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂52.14.217.35 - - [14/Mar/2025:01:17:42 +0800] ＂GET /Spider/?url=%2Fzheli%2F14575.html HTTP/2.0＂ 301 162 ＂-＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂52.14.217.35 - - [14/Mar/2025:01:17:42 +0800] ＂GET /Spider/?url=%2Fzheli%2F14575.html HTTP/2.0＂ 200 20 ＂https://bm.cn/Spider/?url=%2Fzheli%2F14575.html＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂52.15.33.65 - - [14/Mar/2025:01:18:41 +0800] ＂GET /zheli/15446.html HTTP/2.0＂ 301 162 ＂-＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂52.15.33.65 - - [14/Mar/2025:01:18:42 +0800] ＂GET /zheli/15446.html HTTP/2.0＂ 200 6486 ＂https://bm.cn/zheli/15446.html＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂3.15.33.175 - - [14/Mar/2025:01:19:00 +0800] ＂GET /?p=%2FDo%2Fvisits%2Fid%2F15151 HTTP/2.0＂ 301 162 ＂-＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂3.15.33.175 - - [14/Mar/2025:01:19:00 +0800] ＂GET /Spider/?url=%2Fzheli%2F15151.html HTTP/2.0＂ 301 162 ＂-＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂3.15.33.175 - - [14/Mar/2025:01:19:01 +0800] ＂GET /Spider/?url=%2Fzheli%2F15151.html HTTP/2.0＂ 200 20 ＂https://bm.cn/Spider/?url=%2Fzheli%2F15151.html＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂3.15.33.175 - - [14/Mar/2025:01:19:01 +0800] ＂GET /?p=%2FDo%2Fvisits%2Fid%2F15151 HTTP/2.0＂ 200 27 ＂https://bm.cn/?p=%2FDo%2Fvisits%2Fid%2F15151＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂3.142.119.0 - - [14/Mar/2025:01:19:24 +0800] ＂GET /?p=%2FDo%2Fvisits%2Fid%2F11859 HTTP/2.0＂ 301 162 ＂-＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂3.142.119.0 - - [14/Mar/2025:01:19:25 +0800] ＂GET /?p=%2FDo%2Fvisits%2Fid%2F11859 HTTP/2.0＂ 200 27 ＂https://bm.cn/?p=%2FDo%2Fvisits%2Fid%2F11859＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂3.144.138.160 - - [14/Mar/2025:01:19:59 +0800] ＂GET /Spider/?url=%2Flizhirenwu%2F13822.html HTTP/2.0＂ 301 162 ＂-＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂3.144.138.160 - - [14/Mar/2025:01:20:00 +0800] ＂GET /Spider/?url=%2Flizhirenwu%2F13822.html HTTP/2.0＂ 200 20 ＂https://bm.cn/Spider/?url=%2Flizhirenwu%2F13822.html＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂3.18.103.72 - - [14/Mar/2025:01:20:00 +0800] ＂GET /?p=%2FDo%2Fvisits%2Fid%2F16923 HTTP/2.0＂ 301 162 ＂-＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂3.18.103.72 - - [14/Mar/2025:01:20:01 +0800] ＂GET /?p=%2FDo%2Fvisits%2Fid%2F16923 HTTP/2.0＂ 200 27 ＂https://bm.cn/?p=%2FDo%2Fvisits%2Fid%2F16923＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂18.119.10.178 - - [14/Mar/2025:01:20:22 +0800] ＂GET /?p=%2FDo%2Fvisits%2Fid%2F12272 HTTP/2.0＂ 301 162 ＂-＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂18.119.10.178 - - [14/Mar/2025:01:20:23 +0800] ＂GET /?p=%2FDo%2Fvisits%2Fid%2F12272 HTTP/2.0＂ 200 27 ＂https://bm.cn/?p=%2FDo%2Fvisits%2Fid%2F12272＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂3.15.163.5 - - [14/Mar/2025:01:20:27 +0800] ＂GET /Spider/?url=%2Fmingyan%2F11834.html HTTP/2.0＂ 301 162 ＂-＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂3.15.163.5 - - [14/Mar/2025:01:20:27 +0800] ＂GET /Spider/?url=%2Fmingyan%2F11834.html HTTP/2.0＂ 200 20 ＂https://bm.cn/Spider/?url=%2Fmingyan%2F11834.html＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂52.14.172.185 - - [14/Mar/2025:01:21:21 +0800] ＂GET /lizhigushi/9903.html HTTP/2.0＂ 301 162 ＂-＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂52.14.172.185 - - [14/Mar/2025:01:21:23 +0800] ＂GET /lizhigushi/9903.html HTTP/2.0＂ 200 6363 ＂https://bm.cn/lizhigushi/9903.html＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂3.145.68.208 - - [14/Mar/2025:01:21:59 +0800] ＂GET /lizhirenwu/12951.html HTTP/2.0＂ 301 162 ＂-＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂3.138.121.63 - - [14/Mar/2025:01:21:59 +0800] ＂GET /zheli/15750.html HTTP/2.0＂ 301 162 ＂-＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂3.145.68.208 - - [14/Mar/2025:01:22:01 +0800] ＂GET /lizhirenwu/12951.html HTTP/2.0＂ 200 8768 ＂https://bm.cn/lizhirenwu/12951.html＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂3.138.121.63 - - [14/Mar/2025:01:22:01 +0800] ＂GET /zheli/15750.html HTTP/2.0＂ 200 7679 ＂https://bm.cn/zheli/15750.html＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂3.144.190.95 - - [14/Mar/2025:01:24:33 +0800] ＂GET /zheli/14880.html HTTP/2.0＂ 301 162 ＂-＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂3.144.190.95 - - [14/Mar/2025:01:24:35 +0800] ＂GET /zheli/14880.html HTTP/2.0＂ 200 6297 ＂https://bm.cn/zheli/14880.html＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂3.145.129.10 - - [14/Mar/2025:01:32:26 +0800] ＂GET /?p=%2FDo%2Fvisits%2Fid%2F9480 HTTP/2.0＂ 301 162 ＂-＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂3.145.129.10 - - [14/Mar/2025:01:32:27 +0800] ＂GET /?p=%2FDo%2Fvisits%2Fid%2F9480 HTTP/2.0＂ 200 27 ＂https://bm.cn/?p=%2FDo%2Fvisits%2Fid%2F9480＂ ＂Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)＂