机器人每天访问网站 1k+ 次 [已关闭]

问题描述 投票:0回答:2

我很难确定是什么导致我的网站加载速度极慢,我找到了一些东西,但谷歌档案没有提供正确的答案,甚至没有提供解释。 在我的原始访问日志中,我发现了有关不同机器人访问我的网站的多个记录,这是一个示例:

202.46.53.40 - - [31/Dec/2016:03:30:51 +0100] "GET /en/home/184-2016-hyperlite-motive-wakeboard.html HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36"
202.46.54.27 - - [31/Dec/2016:03:30:52 +0100] "GET /en/home/184-2016-hyperlite-motive-wakeboard.html HTTP/1.1" 301 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36"
202.46.56.210 - - [31/Dec/2016:03:30:53 +0100] "GET /en/home/184-2016-hyperlite-motive-wakeboard.html HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36"
202.46.56.114 - - [31/Dec/2016:03:30:54 +0100] "GET /en/wakeboards/184-2016-hyperlite-motive-wakeboard.html HTTP/1.1" 200 140041 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36"
180.76.15.154 - - [31/Dec/2016:03:31:26 +0100] "GET /en/26-sup HTTP/1.1" 406 73864 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
157.55.39.40 - - [31/Dec/2016:03:31:50 +0100] "GET /en/helmets/57-2015-mystic-mk8-helmet-mint.html HTTP/1.1" 302 - "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.39.40 - - [31/Dec/2016:03:31:55 +0100] "GET /en/helmets/57-2015-mystic-mk8-helmet-mint.html HTTP/1.1" 301 - "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
77.75.77.95 - - [31/Dec/2016:03:34:03 +0100] "GET /robots.txt HTTP/1.1" 404 57839 "-" "Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)"
77.75.77.95 - - [31/Dec/2016:03:34:05 +0100] "GET /en/31-bags HTTP/1.1" 301 - "-" "Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)"
163.172.66.143 - - [31/Dec/2016:03:43:36 +0100] "GET /en/13-rokavice HTTP/1.1" 302 - "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)"
202.46.54.134 - - [31/Dec/2016:04:04:20 +0100] "GET /en/accessories/169-plavutke-pro-ii.html HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36"
202.46.54.102 - - [31/Dec/2016:04:04:21 +0100] "GET /en/accessories/169-plavutke-pro-ii.html HTTP/1.1" 301 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36"
202.46.48.140 - - [31/Dec/2016:04:04:22 +0100] "GET /en/accessories/169-plavutke-pro-ii.html HTTP/1.1" 200 110602 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36"
180.76.15.10 - - [31/Dec/2016:04:04:55 +0100] "GET /en/56-kiteboarding-gear HTTP/1.1" 406 62988 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
66.249.76.47 - - [31/Dec/2016:04:25:33 +0100] "GET /380/komplet-oceanrodeo-razor-fst8-advenced-performance-kite.jpg HTTP/1.1" 200 126044 "-" "Googlebot-Image/1.0"
112.210.233.49 - - [31/Dec/2016:04:29:17 +0100] "POST /modules/sendtoafriend/sendtoafriend_ajax.php?rand=1472104141118 HTTP/1.1" 500 - "https://proadrenalin.si/modules/sendtoafriend/sendtoafriend_ajax.php?rand=1472104141118" "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1)"
66.249.76.78 - - [31/Dec/2016:04:33:09 +0100] "POST /modules/leocustomajax/leoajax.php?rand=1482019200024 HTTP/1.1" 200 14 "https://www.proadrenalin.si/en/20-wakeboards?p=3" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

是否有可能是这次访问导致了页面加载缓慢的问题? 12 月 31 日我有 1342 个请求,1 月 1 日有 1222 个请求,1 月 2 日有 2374 个请求,1 月 4 日有 2391 个请求......这种情况每天都在发生。 Webshop 由 Prestashop 运行,据我检查,该平台没有造成任何导致页面加载缓慢的问题。大多数模块都被禁用、删除,只有需要(启用)的模块在服务器上,兑现功能已打开,当发生变化时重新编译..

任何提示、阅读链接、可能的解决方案......都会非常有用,因为目前我生活在噩梦中......

bots prestashop robots.txt
2个回答
2
投票

您可以找到访问您商店的机器人的 IP 模式,然后使用 .htaccess 文件阻止这些 IP。

访问以下 URL 了解更多详细信息:

如何使用 .htaccess 文件阻止 IP 地址范围


0
投票

我有同样的问题,目前正在尝试通过 robots.txt 阻止该机器人的解决方案,如下所示:

User-agent: SeznamBot
Disallow: /

取自官方来源https://napoveda.seznam.cz/en/full-text-search/crawling-control/

© www.soinside.com 2019 - 2024. All rights reserved.