如何做搜索引擎蜘蛛日志分析
搜索引擎蜘蛛日志文件是一种非常强大但未被站长充分利用的文件，分析它可以获取有关每个搜索引擎如何爬取网站内容的相关信息点，及查看搜索引擎蜘蛛在一段时间内的行为。
| IP地址(8) | 服务器名称 | 所属国家 |
|---|---|---|
| 3.38.135.32 | ec2-3-38-135-32.ap-northeast-2.compute.amazonaws.com | KR |
| 54.180.143.100 | ec2-54-180-143-100.ap-northeast-2.compute.amazonaws.com | KR |
| 54.93.112.237 | ec2-54-93-112-237.eu-central-1.compute.amazonaws.com | DE |
| 3.39.229.197 | ec2-3-39-229-197.ap-northeast-2.compute.amazonaws.com | KR |
| 3.69.242.206 | ec2-3-69-242-206.eu-central-1.compute.amazonaws.com | DE |
| 18.118.30.81 | ec2-18-118-30-81.us-east-2.compute.amazonaws.com | US |
| 13.212.243.26 | ec2-13-212-243-26.ap-southeast-1.compute.amazonaws.com | SG |
| 3.14.133.71 | ec2-3-14-133-71.us-east-2.compute.amazonaws.com | US |
| 13.125.211.144 | ec2-13-125-211-144.ap-northeast-2.compute.amazonaws.com | KR |
| 3.120.175.84 | ec2-3-120-175-84.eu-central-1.compute.amazonaws.com | DE |
| 18.117.10.207 | ec2-18-117-10-207.us-east-2.compute.amazonaws.com | US |
| 18.189.178.77 | ec2-18-189-178-77.us-east-2.compute.amazonaws.com | US |
| 18.192.206.115 | ec2-18-192-206-115.eu-central-1.compute.amazonaws.com | DE |
| 52.24.186.221 | ec2-52-24-186-221.us-west-2.compute.amazonaws.com | US |
| 54.93.177.28 | ec2-54-93-177-28.eu-central-1.compute.amazonaws.com | DE |
| 35.87.241.217 | ec2-35-87-241-217.us-west-2.compute.amazonaws.com | US |
| 3.73.86.229 | ec2-3-73-86-229.eu-central-1.compute.amazonaws.com | DE |
| 18.193.254.30 | ec2-18-193-254-30.eu-central-1.compute.amazonaws.com | ? |
| 54.201.104.171 | ec2-54-201-104-171.us-west-2.compute.amazonaws.com | ? |
| 3.67.39.57 | ec2-3-67-39-57.eu-central-1.compute.amazonaws.com | DE |
| 18.194.242.103 | ec2-18-194-242-103.eu-central-1.compute.amazonaws.com | DE |
| 52.15.235.142 | ec2-52-15-235-142.us-east-2.compute.amazonaws.com | US |
| 52.215.190.48 | ec2-52-215-190-48.eu-west-1.compute.amazonaws.com | IE |
| 54.183.177.59 | ec2-54-183-177-59.us-west-1.compute.amazonaws.com | US |
| 18.231.141.117 | ec2-18-231-141-117.sa-east-1.compute.amazonaws.com | BR |
| 13.125.43.53 | ec2-13-125-43-53.ap-northeast-2.compute.amazonaws.com | KR |
| 3.8.233.236 | ec2-3-8-233-236.eu-west-2.compute.amazonaws.com | GB |
| 34.254.98.129 | ec2-34-254-98-129.eu-west-1.compute.amazonaws.com | IE |
| 13.229.120.26 | ec2-13-229-120-26.ap-southeast-1.compute.amazonaws.com | SG |
| 3.253.84.110 | ec2-3-253-84-110.eu-west-1.compute.amazonaws.com | IE |
| 3.38.106.29 | ec2-3-38-106-29.ap-northeast-2.compute.amazonaws.com | KR |
| 18.119.108.223 | ec2-18-119-108-223.us-east-2.compute.amazonaws.com | US |
| 18.119.166.235 | ec2-18-119-166-235.us-east-2.compute.amazonaws.com | US |
| 18.119.129.146 | ec2-18-119-129-146.us-east-2.compute.amazonaws.com | US |
| 35.86.209.2 | ec2-35-86-209-2.us-west-2.compute.amazonaws.com | US |
| 35.92.156.181 | ec2-35-92-156-181.us-west-2.compute.amazonaws.com | US |
| 52.40.185.41 | ec2-52-40-185-41.us-west-2.compute.amazonaws.com | US |
| 3.73.51.233 | ec2-3-73-51-233.eu-central-1.compute.amazonaws.com | DE |
| 3.127.27.112 | ec2-3-127-27-112.eu-central-1.compute.amazonaws.com | ? |
| 52.78.71.100 | ec2-52-78-71-100.ap-northeast-2.compute.amazonaws.com | KR |
| 18.157.80.92 | ec2-18-157-80-92.eu-central-1.compute.amazonaws.com | DE |
| 54.180.140.71 | ec2-54-180-140-71.ap-northeast-2.compute.amazonaws.com | KR |
| 18.191.5.200 | ec2-18-191-5-200.us-east-2.compute.amazonaws.com | US |
| 35.86.243.48 | ec2-35-86-243-48.us-west-2.compute.amazonaws.com | US |
| 3.144.6.124 | ec2-3-144-6-124.us-east-2.compute.amazonaws.com | US |
| 18.119.136.60 | ec2-18-119-136-60.us-east-2.compute.amazonaws.com | US |
| 35.85.145.137 | ec2-35-85-145-137.us-west-2.compute.amazonaws.com | US |
| 18.224.251.111 | ec2-18-224-251-111.us-east-2.compute.amazonaws.com | US |
| 3.137.158.241 | ec2-3-137-158-241.us-east-2.compute.amazonaws.com | US |
| 3.73.32.157 | ec2-3-73-32-157.eu-central-1.compute.amazonaws.com | ? |
| 18.224.51.243 | ec2-18-224-51-243.us-east-2.compute.amazonaws.com | US |
| 3.71.99.156 | ec2-3-71-99-156.eu-central-1.compute.amazonaws.com | DE |
| 3.141.45.168 | ec2-3-141-45-168.us-east-2.compute.amazonaws.com | US |
| 35.158.137.156 | ec2-35-158-137-156.eu-central-1.compute.amazonaws.com | DE |
| 3.136.11.206 | ec2-3-136-11-206.us-east-2.compute.amazonaws.com | US |
| 3.78.222.168 | ec2-3-78-222-168.eu-central-1.compute.amazonaws.com | DE |
| 34.218.231.231 | ec2-34-218-231-231.us-west-2.compute.amazonaws.com | US |
| 3.137.181.118 | ec2-3-137-181-118.us-east-2.compute.amazonaws.com | US |
| 3.144.9.2 | ec2-3-144-9-2.us-east-2.compute.amazonaws.com | US |
| 18.116.14.165 | ec2-18-116-14-165.us-east-2.compute.amazonaws.com | US |
| 18.192.48.66 | ec2-18-192-48-66.eu-central-1.compute.amazonaws.com | DE |
| 3.37.175.237 | ec2-3-37-175-237.ap-northeast-2.compute.amazonaws.com | KR |
| 18.195.214.235 | ec2-18-195-214-235.eu-central-1.compute.amazonaws.com | DE |
| 18.217.91.238 | ec2-18-217-91-238.us-east-2.compute.amazonaws.com | US |
| 3.76.215.10 | ec2-3-76-215-10.eu-central-1.compute.amazonaws.com | DE |
| 52.79.35.145 | ec2-52-79-35-145.ap-northeast-2.compute.amazonaws.com | KR |
| 3.141.194.81 | ec2-3-141-194-81.us-east-2.compute.amazonaws.com | US |
| 18.116.163.140 | ec2-18-116-163-140.us-east-2.compute.amazonaws.com | US |
| 3.70.170.114 | ec2-3-70-170-114.eu-central-1.compute.amazonaws.com | DE |
| 18.197.158.97 | ec2-18-197-158-97.eu-central-1.compute.amazonaws.com | DE |
| 52.28.226.221 | ec2-52-28-226-221.eu-central-1.compute.amazonaws.com | DE |
| 3.34.125.62 | ec2-3-34-125-62.ap-northeast-2.compute.amazonaws.com | KR |
| 35.91.38.89 | ec2-35-91-38-89.us-west-2.compute.amazonaws.com | US |
| 3.123.6.72 | ec2-3-123-6-72.eu-central-1.compute.amazonaws.com | DE |
| 35.81.166.216 | ec2-35-81-166-216.us-west-2.compute.amazonaws.com | US |
| IP地址(8) | 服务器名称 | 所属国家 |
|---|---|---|
| 18.194.242.103 | ec2-18-194-242-103.eu-central-1.compute.amazonaws.com | DE |
| 52.15.235.142 | ec2-52-15-235-142.us-east-2.compute.amazonaws.com | US |
| 52.215.190.48 | ec2-52-215-190-48.eu-west-1.compute.amazonaws.com | IE |
| 54.183.177.59 | ec2-54-183-177-59.us-west-1.compute.amazonaws.com | US |
| 18.231.141.117 | ec2-18-231-141-117.sa-east-1.compute.amazonaws.com | BR |
| 13.125.43.53 | ec2-13-125-43-53.ap-northeast-2.compute.amazonaws.com | KR |
| 3.8.233.236 | ec2-3-8-233-236.eu-west-2.compute.amazonaws.com | GB |
| 34.254.98.129 | ec2-34-254-98-129.eu-west-1.compute.amazonaws.com | IE |
对于未知蜘蛛或者爬虫。它的用途对网站来说可能是好的，也可能是坏的，这取决于它是什么。所以说，这需要站长进一步分析判断这些尚不明确的爬虫行为，再作最终决定。 但，根据以往的经验，未声明行为目的及未命名的蜘蛛爬虫，通常都有不可告人的秘密，我们理应对其行为进行控制，比如拦截。
您可以通过在网站的 robots.txt 中设置用户代理访问规则来屏蔽 CheckMarkNetwork 或限制其访问权限。我们建议安装 Spider Analyser 插件，以检查它是否真正遵循这些规则。
# robots.txt # 下列代码一般情况可以拦截该代理 User-agent: CheckMarkNetwork Disallow: /
您无需手动执行此操作，可通过我们的 Wordpress 插件 Spider Analyser 来拦截不必要的蜘蛛或者爬虫。
(工作日 10:00 - 18:30 为您服务)
