如何做搜索引擎蜘蛛日志分析
搜索引擎蜘蛛日志文件是一种非常强大但未被站长充分利用的文件,分析它可以获取有关每个搜索引擎如何爬取网站内容的相关信息点,及查看搜索引擎蜘蛛在一段时间内的行为。
| IP地址(131) | 服务器名称 | 所属国家 |
|---|---|---|
| 23.22.255.95 | ec2-23-22-255-95.compute-1.amazonaws.com | US |
| 3.228.110.6 | ec2-3-228-110-6.compute-1.amazonaws.com | US |
| 35.171.93.237 | ec2-35-171-93-237.compute-1.amazonaws.com | US |
| 3.228.230.254 | ec2-3-228-230-254.compute-1.amazonaws.com | US |
| 54.224.65.6 | ec2-54-224-65-6.compute-1.amazonaws.com | US |
| 52.45.60.239 | ec2-52-45-60-239.compute-1.amazonaws.com | US |
| 3.209.1.82 | ec2-3-209-1-82.compute-1.amazonaws.com | US |
| 54.225.107.146 | ec2-54-225-107-146.compute-1.amazonaws.com | US |
| 34.195.147.166 | ec2-34-195-147-166.compute-1.amazonaws.com | US |
| 52.70.241.236 | ec2-52-70-241-236.compute-1.amazonaws.com | US |
| IP地址(31) | 服务器名称 | 所属国家 |
|---|---|---|
| 100.25.85.101 | ec2-100-25-85-101.compute-1.amazonaws.com | US |
| 54.224.65.6 | ec2-54-224-65-6.compute-1.amazonaws.com | US |
| 54.225.107.146 | ec2-54-225-107-146.compute-1.amazonaws.com | US |
| 52.70.241.236 | ec2-52-70-241-236.compute-1.amazonaws.com | US |
| 3.209.1.82 | ec2-3-209-1-82.compute-1.amazonaws.com | US |
| 35.171.93.237 | ec2-35-171-93-237.compute-1.amazonaws.com | US |
| 23.21.106.87 | ec2-23-21-106-87.compute-1.amazonaws.com | ? |
| 23.22.255.95 | ec2-23-22-255-95.compute-1.amazonaws.com | US |
| 52.45.60.239 | ec2-52-45-60-239.compute-1.amazonaws.com | US |
| 34.195.147.166 | ec2-34-195-147-166.compute-1.amazonaws.com | US |
| IP地址(21) | 服务器名称 | 所属国家 |
|---|---|---|
| 34.195.11.192 | ec2-34-195-11-192.compute-1.amazonaws.com | US |
| 54.225.107.146 | ec2-54-225-107-146.compute-1.amazonaws.com | US |
| 3.228.110.6 | ec2-3-228-110-6.compute-1.amazonaws.com | US |
| 100.25.85.101 | ec2-100-25-85-101.compute-1.amazonaws.com | US |
| 54.157.240.233 | ec2-54-157-240-233.compute-1.amazonaws.com | US |
| 54.164.10.107 | ec2-54-164-10-107.compute-1.amazonaws.com | US |
| 34.196.176.7 | ec2-34-196-176-7.compute-1.amazonaws.com | US |
| 23.22.255.95 | ec2-23-22-255-95.compute-1.amazonaws.com | US |
| 18.204.244.43 | ec2-18-204-244-43.compute-1.amazonaws.com | US |
| 52.70.241.236 | ec2-52-70-241-236.compute-1.amazonaws.com | US |
| IP地址(65) | 服务器名称 | 所属国家 |
|---|---|---|
| 54.224.65.6 | ec2-54-224-65-6.compute-1.amazonaws.com | US |
| 52.70.241.236 | ec2-52-70-241-236.compute-1.amazonaws.com | US |
| 52.45.60.239 | ec2-52-45-60-239.compute-1.amazonaws.com | US |
| 35.170.86.252 | ec2-35-170-86-252.compute-1.amazonaws.com | US |
| 23.22.255.95 | ec2-23-22-255-95.compute-1.amazonaws.com | US |
| 23.21.106.87 | ec2-23-21-106-87.compute-1.amazonaws.com | US |
| 54.225.107.146 | ec2-54-225-107-146.compute-1.amazonaws.com | US |
| 3.228.230.254 | ec2-3-228-230-254.compute-1.amazonaws.com | US |
| 34.195.147.166 | ec2-34-195-147-166.compute-1.amazonaws.com | US |
| 54.243.107.230 | ec2-54-243-107-230.compute-1.amazonaws.com | US |
| 3.233.252.114 | ec2-3-233-252-114.compute-1.amazonaws.com | US |
| 100.25.85.101 | ec2-100-25-85-101.compute-1.amazonaws.com | US |
| 44.208.63.44 | ec2-44-208-63-44.compute-1.amazonaws.com | US |
| 3.228.110.6 | ec2-3-228-110-6.compute-1.amazonaws.com | US |
| 35.171.93.237 | ec2-35-171-93-237.compute-1.amazonaws.com | US |
| 34.196.176.7 | ec2-34-196-176-7.compute-1.amazonaws.com | US |
| 3.209.1.82 | ec2-3-209-1-82.compute-1.amazonaws.com | US |
| 34.195.11.192 | ec2-34-195-11-192.compute-1.amazonaws.com | US |
| 54.157.240.233 | ec2-54-157-240-233.compute-1.amazonaws.com | US |
| 54.164.10.107 | ec2-54-164-10-107.compute-1.amazonaws.com | US |
| 18.204.244.43 | ec2-18-204-244-43.compute-1.amazonaws.com | US |
| 3.93.182.229 | ec2-3-93-182-229.compute-1.amazonaws.com | US |
| 54.90.240.76 | ec2-54-90-240-76.compute-1.amazonaws.com | US |
| 34.203.202.84 | ec2-34-203-202-84.compute-1.amazonaws.com | US |
| 3.83.164.208 | ec2-3-83-164-208.compute-1.amazonaws.com | US |
| 54.198.48.124 | ec2-54-198-48-124.compute-1.amazonaws.com | US |
| 3.93.68.213 | ec2-3-93-68-213.compute-1.amazonaws.com | US |
| 3.91.179.124 | ec2-3-91-179-124.compute-1.amazonaws.com | US |
| 54.172.143.223 | ec2-54-172-143-223.compute-1.amazonaws.com | US |
| 52.86.244.199 | ec2-52-86-244-199.compute-1.amazonaws.com | US |
| 34.227.224.23 | 34.227.224.23 | US |
| 54.145.95.28 | ec2-54-145-95-28.compute-1.amazonaws.com | US |
| 54.87.91.179 | ec2-54-87-91-179.compute-1.amazonaws.com | US |
| 54.208.13.120 | ip-54-208-13-120.core.woorank.com | US |
| 44.208.85.114 | ec2-44-208-85-114.compute-1.amazonaws.com | US |
| 54.147.154.129 | ec2-54-147-154-129.compute-1.amazonaws.com | US |
| 52.45.72.243 | ec2-52-45-72-243.compute-1.amazonaws.com | US |
| 54.165.180.233 | 54.165.180.233 | US |
| 18.215.29.61 | ec2-18-215-29-61.compute-1.amazonaws.com | US |
| 3.230.219.226 | ec2-3-230-219-226.compute-1.amazonaws.com | US |
| 54.174.175.150 | ip-54-174-175-150.core.woorank.com | US |
| 107.21.221.94 | ec2-107-21-221-94.compute-1.amazonaws.com | US |
| 54.165.77.65 | ec2-54-165-77-65.compute-1.amazonaws.com | US |
| 54.91.214.198 | ec2-54-91-214-198.compute-1.amazonaws.com | US |
| 54.157.214.118 | ec2-54-157-214-118.compute-1.amazonaws.com | US |
| 44.201.209.50 | ec2-44-201-209-50.compute-1.amazonaws.com | US |
| 107.22.141.52 | ec2-107-22-141-52.compute-1.amazonaws.com | US |
| 185.209.161.11 | . | NL |
| 3.227.234.100 | ec2-3-227-234-100.compute-1.amazonaws.com | US |
| 35.168.36.210 | ec2-35-168-36-210.compute-1.amazonaws.com | US |
| IP地址(65) | 服务器名称 | 所属国家 |
|---|---|---|
| 3.93.182.229 | ec2-3-93-182-229.compute-1.amazonaws.com | US |
| 54.90.240.76 | ec2-54-90-240-76.compute-1.amazonaws.com | US |
| 34.203.202.84 | ec2-34-203-202-84.compute-1.amazonaws.com | US |
| 3.83.164.208 | ec2-3-83-164-208.compute-1.amazonaws.com | US |
| 54.198.48.124 | ec2-54-198-48-124.compute-1.amazonaws.com | US |
| 3.93.68.213 | ec2-3-93-68-213.compute-1.amazonaws.com | US |
| 3.91.179.124 | ec2-3-91-179-124.compute-1.amazonaws.com | US |
| 54.172.143.223 | ec2-54-172-143-223.compute-1.amazonaws.com | US |
| 52.86.244.199 | ec2-52-86-244-199.compute-1.amazonaws.com | US |
| 34.227.224.23 | 34.227.224.23 | US |
| IP地址(1) | 服务器名称 | 所属国家 |
|---|---|---|
| 54.91.181.80 | ec2-54-91-181-80.compute-1.amazonaws.com | US |
| IP地址(1) | 服务器名称 | 所属国家 |
|---|---|---|
| 54.210.193.108 | ec2-54-210-193-108.compute-1.amazonaws.com | US |
| IP地址(63) | 服务器名称 | 所属国家 |
|---|---|---|
| 54.165.145.137 | ec2-54-165-145-137.compute-1.amazonaws.com | US |
| 34.203.216.168 | ec2-34-203-216-168.compute-1.amazonaws.com | US |
| 34.230.27.176 | ec2-34-230-27-176.compute-1.amazonaws.com | US |
| 52.90.111.91 | ec2-52-90-111-91.compute-1.amazonaws.com | US |
| 18.232.151.94 | ec2-18-232-151-94.compute-1.amazonaws.com | US |
| 34.207.142.152 | ec2-34-207-142-152.compute-1.amazonaws.com | US |
| 54.85.162.172 | ec2-54-85-162-172.compute-1.amazonaws.com | US |
| 52.207.172.103 | ec2-52-207-172-103.compute-1.amazonaws.com | US |
| 34.203.247.29 | ec2-34-203-247-29.compute-1.amazonaws.com | US |
| 54.234.40.32 | ec2-54-234-40-32.compute-1.amazonaws.com | US |
| IP地址(10) | 服务器名称 | 所属国家 |
|---|---|---|
| 54.174.241.203 | ec2-54-174-241-203.compute-1.amazonaws.com | US |
| 54.175.247.210 | ec2-54-175-247-210.compute-1.amazonaws.com | US |
| 34.230.49.3 | ec2-34-230-49-3.compute-1.amazonaws.com | US |
| 52.23.255.59 | ec2-52-23-255-59.compute-1.amazonaws.com | US |
| 52.207.216.60 | ec2-52-207-216-60.compute-1.amazonaws.com | US |
| 18.212.49.15 | ec2-18-212-49-15.compute-1.amazonaws.com | US |
| 34.228.232.112 | ec2-34-228-232-112.compute-1.amazonaws.com | US |
| 52.206.163.222 | ec2-52-206-163-222.compute-1.amazonaws.com | US |
| 54.84.247.202 | ec2-54-84-247-202.compute-1.amazonaws.com | US |
| 54.86.99.216 | ec2-54-86-99-216.compute-1.amazonaws.com | US |
| IP地址(4) | 服务器名称 | 所属国家 |
|---|---|---|
| 54.175.247.210 | ec2-54-175-247-210.compute-1.amazonaws.com | US |
| 18.206.120.14 | ec2-18-206-120-14.compute-1.amazonaws.com | US |
| 52.207.216.60 | ec2-52-207-216-60.compute-1.amazonaws.com | US |
| 34.230.49.3 | ec2-34-230-49-3.compute-1.amazonaws.com | US |
| IP地址(3) | 服务器名称 | 所属国家 |
|---|---|---|
| 18.206.120.14 | ec2-18-206-120-14.compute-1.amazonaws.com | US |
| 52.23.255.59 | ec2-52-23-255-59.compute-1.amazonaws.com | US |
| 34.235.128.34 | ec2-34-235-128-34.compute-1.amazonaws.com | US |
| IP地址(2) | 服务器名称 | 所属国家 |
|---|---|---|
| 54.161.198.183 | ec2-54-161-198-183.compute-1.amazonaws.com | US |
| 54.91.181.80 | ec2-54-91-181-80.compute-1.amazonaws.com | US |
| 54.210.193.108 | ec2-54-210-193-108.compute-1.amazonaws.com | US |
| 54.165.145.137 | ec2-54-165-145-137.compute-1.amazonaws.com | US |
| 34.203.216.168 | ec2-34-203-216-168.compute-1.amazonaws.com | US |
| 34.230.27.176 | ec2-34-230-27-176.compute-1.amazonaws.com | US |
| 52.90.111.91 | ec2-52-90-111-91.compute-1.amazonaws.com | US |
| 18.232.151.94 | ec2-18-232-151-94.compute-1.amazonaws.com | US |
| 34.207.142.152 | ec2-34-207-142-152.compute-1.amazonaws.com | US |
| 54.85.162.172 | ec2-54-85-162-172.compute-1.amazonaws.com | US |
| 52.207.172.103 | ec2-52-207-172-103.compute-1.amazonaws.com | US |
| 34.203.247.29 | ec2-34-203-247-29.compute-1.amazonaws.com | US |
| 54.234.40.32 | ec2-54-234-40-32.compute-1.amazonaws.com | US |
| 54.174.241.203 | ec2-54-174-241-203.compute-1.amazonaws.com | US |
| 54.175.247.210 | ec2-54-175-247-210.compute-1.amazonaws.com | US |
| 34.230.49.3 | ec2-34-230-49-3.compute-1.amazonaws.com | US |
| 52.23.255.59 | ec2-52-23-255-59.compute-1.amazonaws.com | US |
| 52.207.216.60 | ec2-52-207-216-60.compute-1.amazonaws.com | US |
| 18.212.49.15 | ec2-18-212-49-15.compute-1.amazonaws.com | US |
| 34.228.232.112 | ec2-34-228-232-112.compute-1.amazonaws.com | US |
| 52.206.163.222 | ec2-52-206-163-222.compute-1.amazonaws.com | US |
| 54.84.247.202 | ec2-54-84-247-202.compute-1.amazonaws.com | US |
| 54.86.99.216 | ec2-54-86-99-216.compute-1.amazonaws.com | US |
| 18.206.120.14 | ec2-18-206-120-14.compute-1.amazonaws.com | US |
| 34.235.128.34 | ec2-34-235-128-34.compute-1.amazonaws.com | US |
| 54.167.158.55 | ec2-54-167-158-55.compute-1.amazonaws.com | US |
| 34.235.143.139 | ec2-34-235-143-139.compute-1.amazonaws.com | US |
| 54.164.98.220 | ec2-54-164-98-220.compute-1.amazonaws.com | US |
| 54.173.209.55 | ec2-54-173-209-55.compute-1.amazonaws.com | US |
| IP地址(2) | 服务器名称 | 所属国家 |
|---|---|---|
| 54.167.158.55 | ec2-54-167-158-55.compute-1.amazonaws.com | US |
| 52.206.163.222 | ec2-52-206-163-222.compute-1.amazonaws.com | US |
| IP地址(2) | 服务器名称 | 所属国家 |
|---|---|---|
| 34.235.143.139 | ec2-34-235-143-139.compute-1.amazonaws.com | US |
| 54.164.98.220 | ec2-54-164-98-220.compute-1.amazonaws.com | US |
| IP地址(34) | 服务器名称 | 所属国家 |
|---|---|---|
| 54.85.198.156 | ip-54-85-198-156.core.woorank.com | US |
| 54.173.113.175 | ip-54-173-113-175.core.woorank.com | US |
| 52.5.69.31 | ip-52-5-69-31.core.woorank.com | US |
| 54.152.77.211 | ip-54-152-77-211.core.woorank.com | US |
| 52.7.32.143 | ip-52-7-32-143.core.woorank.com | US |
| 54.84.162.141 | ip-54-84-162-141.core.woorank.com | US |
| 52.5.133.46 | ip-52-5-133-46.core.woorank.com | US |
| 52.0.86.232 | ip-52-0-86-232.core.woorank.com | US |
| 54.173.9.10 | ip-54-173-9-10.core.woorank.com | US |
| 54.174.179.157 | ip-54-174-179-157.core.woorank.com | US |
| IP地址(2) | 服务器名称 | 所属国家 |
|---|---|---|
| 54.84.162.141 | ip-54-84-162-141.core.woorank.com | US |
| 52.0.86.232 | ip-52-0-86-232.core.woorank.com | US |
一般不需要拦截,尤其是如果你自己也受益于搜索引擎优化服务。不过,如果你担心服务器资源占用等问题,且您都不使用这些工具,当然也可以选择拦截它们。
您可以通过在网站的 robots.txt 中设置用户代理访问规则来屏蔽 woobot 或限制其访问权限。我们建议安装 Spider Analyser 插件,以检查它是否真正遵循这些规则。
# robots.txt # 下列代码一般情况可以拦截该代理 User-agent: woobot Disallow: /
您无需手动执行此操作,可通过我们的 Wordpress 插件 Spider Analyser 来拦截不必要的蜘蛛或者爬虫。
(工作日 10:00 - 18:30 为您服务)