如何做搜索引擎蜘蛛日志分析
搜索引擎蜘蛛日志文件是一种非常强大但未被站长充分利用的文件，分析它可以获取有关每个搜索引擎如何爬取网站内容的相关信息点，及查看搜索引擎蜘蛛在一段时间内的行为。
| IP地址(1) | 服务器名称 | 所属国家 |
|---|---|---|
| 51.77.135.89 | ns31066279.ip-51-77-135.eu | FR |
| IP地址(1) | 服务器名称 | 所属国家 |
|---|---|---|
| 104.238.188.207 | ? | FR |
| IP地址(53) | 服务器名称 | 所属国家 |
|---|---|---|
| 202.143.148.61 | 202.143.148.61 | TH |
| 23.160.208.247 | relay13f.tor.ian.sh | US |
| 18.221.254.150 | ec2-18-221-254-150.us-east-2.compute.amazonaws.com | US |
| 80.82.70.118 | rnd.group-ib.ru | SC |
| 63.123.238.8 | 63.123.238.8 | US |
| 38.70.11.33 | 38.70.11.33 | US |
| 51.77.135.89 | ns31066279.ip-51-77-135.eu | FR |
| 104.238.188.207 | ? | FR |
| 65.214.45.217 | ? | US |
| 65.214.45.213 | ? | US |
| 65.214.45.143 | 65.214.45.143 | US |
| 65.214.45.148 | 65.214.45.148 | US |
| 66.235.124.57 | crawler5106.ask.com | US |
| 66.235.116.44 | crawler9044.ask.com | US |
| 66.235.124.81 | crawler5129.ask.com | US |
| 66.235.124.119 | crawler5141.ask.com | US |
| 66.235.124.196 | crawler6196.ask.com | US |
| 66.235.124.58 | crawler5107.ask.com | US |
| 66.235.124.59 | crawler5108.ask.com | US |
| 66.235.124.10 | crawler5010.ask.com | US |
| 66.235.124.98 | crawler5198.ask.com | US |
| 66.235.124.124 | crawler6124.ask.com | US |
| 66.235.124.2 | crawler5002.ask.com | US |
| 66.235.124.68 | crawler5116.ask.com | US |
| 104.234.204.134 | 104.234.204.134 | CA |
| 66.235.124.100 | crawler5200.ask.com | US |
| 66.235.124.99 | crawler5199.ask.com | US |
| 66.235.124.125 | crawler6125.ask.com | US |
| 66.235.124.202 | crawler6202.ask.com | US |
| 3.69.48.48 | ec2-3-69-48-48.eu-central-1.compute.amazonaws.com | DE |
| 54.167.12.189 | ec2-54-167-12-189.compute-1.amazonaws.com | US |
| 66.235.124.3 | crawler5003.ask.com | US |
| 66.235.124.73 | crawler5121.ask.com | US |
| 66.235.124.102 | crawler5202.ask.com | US |
| 66.235.124.127 | crawler6127.ask.com | US |
| 66.235.124.4 | crawler5004.ask.com | US |
| 66.235.124.74 | crawler5122.ask.com | US |
| 44.201.70.77 | ec2-44-201-70-77.compute-1.amazonaws.com | US |
| 66.235.124.55 | crawler5104.ask.com | US |
| 66.235.112.5 | tn-fw.ask.com | US |
| 66.235.124.109 | crawler5209.ask.com | US |
| 66.235.124.8 | crawler5008.ask.com | US |
| 66.235.124.192 | crawler6192.ask.com | US |
| 66.235.124.56 | crawler5105.ask.com | US |
| 159.223.138.47 | 159.223.138.47 | US |
| 66.235.124.1 | crawler5001.ask.com | US |
| 66.235.124.12 | crawler5012.ask.com | US |
| 66.235.124.89 | crawler5137.ask.com | US |
| 66.235.124.197 | crawler6197.ask.com | US |
| 54.160.206.72 | ec2-54-160-206-72.compute-1.amazonaws.com | US |
| 66.235.124.7 | crawler5007.ask.com | US |
| 66.235.124.101 | crawler5201.ask.com | US |
| 66.235.124.126 | crawler6126.ask.com | US |
| 54.183.113.139 | ec2-54-183-113-139.us-west-1.compute.amazonaws.com | US |
| 66.235.116.43 | crawler9043.ask.com | US |
| 66.235.124.80 | crawler5128.ask.com | US |
| 66.235.124.113 | crawler5213.ask.com | US |
| 66.235.124.193 | crawler6193.ask.com | US |
| 35.83.250.11 | ec2-35-83-250-11.us-west-2.compute.amazonaws.com | US |
| 66.235.124.54 | crawler5103.ask.com | US |
| 78.153.140.151 | hostglobal.plus | GB |
| 66.235.124.103 | crawler5203.ask.com | US |
| 66.235.124.13 | crawler5013.ask.com | US |
| 66.235.124.5 | crawler5005.ask.com | US |
| 45.200.148.16 | 45.200.148.16 | ID |
| 178.215.236.240 | 178.215.236.240 | FR |
| 66.235.124.78 | crawler5126.ask.com | US |
| 66.235.124.104 | crawler5204.ask.com | US |
| 66.235.124.139 | crawler6139.ask.com | US |
| 66.235.124.79 | crawler5127.ask.com | US |
| 18.246.40.49 | ec2-18-246-40-49.us-west-2.compute.amazonaws.com | US |
| 66.235.124.105 | crawler5205.ask.com | US |
| 66.235.124.15 | crawler5015.ask.com | US |
| 100.25.24.228 | ec2-100-25-24-228.compute-1.amazonaws.com | US |
| 54.169.152.253 | ec2-54-169-152-253.ap-southeast-1.compute.amazonaws.com | SG |
| IP地址(53) | 服务器名称 | 所属国家 |
|---|---|---|
| 65.214.45.217 | ? | US |
| 65.214.45.213 | ? | US |
| 65.214.45.143 | 65.214.45.143 | US |
| 65.214.45.148 | 65.214.45.148 | US |
| 66.235.116.44 | crawler9044.ask.com | US |
| 66.235.124.89 | crawler5137.ask.com | US |
| 66.235.124.58 | crawler5107.ask.com | US |
| 66.235.124.7 | crawler5007.ask.com | US |
| 66.235.124.103 | crawler5203.ask.com | US |
| 66.235.124.125 | crawler6125.ask.com | US |
对于未知蜘蛛或者爬虫。它的用途对网站来说可能是好的，也可能是坏的，这取决于它是什么。所以说，这需要站长进一步分析判断这些尚不明确的爬虫行为，再作最终决定。 但，根据以往的经验，未声明行为目的及未命名的蜘蛛爬虫，通常都有不可告人的秘密，我们理应对其行为进行控制，比如拦截。
您可以通过在网站的 robots.txt 中设置用户代理访问规则来屏蔽 Ask Jeeves/Teoma 或限制其访问权限。我们建议安装 Spider Analyser 插件，以检查它是否真正遵循这些规则。
# robots.txt # 下列代码一般情况可以拦截该代理 User-agent: Ask Jeeves/Teoma Disallow: /
您无需手动执行此操作，可通过我们的 Wordpress 插件 Spider Analyser 来拦截不必要的蜘蛛或者爬虫。
