如何做搜索引擎蜘蛛日志分析
搜索引擎蜘蛛日志文件是一种非常强大但未被站长充分利用的文件,分析它可以获取有关每个搜索引擎如何爬取网站内容的相关信息点,及查看搜索引擎蜘蛛在一段时间内的行为。
| IP地址(76) | 服务器名称 | 所属国家 |
|---|---|---|
| 51.104.146.225 | 51.104.146.225 | IE |
| 20.56.197.63 | 20.56.197.63 | US |
| 20.197.209.27 | 20.197.209.27 | US |
| 20.72.242.93 | 20.72.242.93 | US |
| 20.43.150.93 | 20.43.150.93 | SG |
| 52.143.242.6 | 52.143.242.6 | US |
| 52.146.59.154 | 52.146.59.154 | US |
| 20.44.222.1 | 20.44.222.1 | SG |
| 52.154.60.82 | 52.154.60.82 | US |
| 40.64.105.247 | 40.64.105.247 | US |
| 51.104.180.53 | 51.104.180.53 | IE |
| 20.71.12.143 | 20.71.12.143 | US |
| 52.146.59.12 | 52.146.59.12 | US |
| 40.64.106.11 | 40.64.106.11 | US |
| 20.73.132.240 | 20.73.132.240 | US |
| 20.73.202.147 | 20.73.202.147 | US |
| 40.89.243.175 | 40.89.243.175 | US |
| 20.53.92.211 | 20.53.92.211 | US |
| 13.89.106.77 | 13.89.106.77 | US |
| 52.146.59.156 | 52.146.59.156 | US |
| 52.143.241.111 | 52.143.241.111 | US |
| 20.99.255.235 | 20.99.255.235 | US |
| 51.104.146.235 | 51.104.146.235 | IE |
| 20.43.150.85 | 20.43.150.85 | SG |
| 52.146.58.236 | 52.146.58.236 | US |
| 20.62.224.44 | 20.62.224.44 | US |
| 20.197.209.11 | 20.197.209.11 | US |
| 51.104.180.47 | 51.104.180.47 | IE |
| 104.211.155.106 | 104.211.155.106 | IN |
| 104.211.155.225 | 104.211.155.225 | IN |
| 51.104.180.26 | 51.104.180.26 | IE |
| 20.53.91.2 | 20.53.91.2 | US |
| 20.56.197.58 | 20.56.197.58 | US |
| 20.226.133.105 | 20.226.133.105 | US |
| 20.207.99.197 | 20.207.99.197 | IN |
| 20.207.97.190 | 20.207.97.190 | IN |
| 40.81.250.205 | 40.81.250.205 | IN |
| 40.88.21.235 | 40.88.21.235 | US |
| 20.191.45.212 | 20.191.45.212 | IE |
| IP地址(128) | 服务器名称 | 所属国家 |
|---|---|---|
| 93.182.76.240 | 93-182-76-240.netonline.net | TR |
| 135.148.89.78 | ip78.ip-135-148-89.us | US |
| 2607:f7a0:5:3:805:4702:a82:58e4 | 2607:f7a0:5:3:805:4702:a82:58e4 | US |
| 156.96.154.175 | ? | US |
| 197.185.96.128 | rain-197-185-96-128.rain.network | ZA |
| 173.245.206.155 | ? | US |
| 41.217.30.222 | 41.217.30.222 | NG |
| 180.251.186.31 | ? | ID |
| 159.242.234.75 | ? | US |
| 160.154.229.110 | ? | CI |
| 162.43.117.82 | sv13241.xserver.jp | JP |
| 35.244.101.12 | 12.101.244.35.bc.googleusercontent.com | AU |
| 188.138.201.133 | 188-138-201-133.starnet.md | ? |
| 185.18.212.186 | cloud20.yourlinuxhost.com | IR |
| 35.202.86.24 | 24.86.202.35.bc.googleusercontent.com | US |
| 203.161.63.181 | server1.pageturner-authors.com | US |
| 34.116.95.186 | 186.95.116.34.bc.googleusercontent.com | AU |
| 123.56.152.17 | 123.56.152.17 | CN |
| 50.63.15.171 | 171.15.63.50.host.secureserver.net | US |
| 216.244.65.162 | cloudcongo.serversfarm.com | US |
| 92.205.52.167 | sh20167.ispgateway.de | FR |
| 87.247.245.131 | gabrovo.footholds.net | GB |
| 104.198.169.217 | 217.169.198.104.bc.googleusercontent.com | US |
| 184.168.114.42 | 42.114.168.184.host.secureserver.net | SG |
| 198.54.125.68 | premium100.web-hosting.com | US |
| 192.95.6.120 | ip120.ip-192-95-6.net | CA |
| 116.90.60.141 | vmres13.web-servers.com.au | AU |
| 162.0.237.182 | server.divineinfoservices.com | US |
| 35.228.201.97 | 97.201.228.35.bc.googleusercontent.com | FI |
| 34.123.236.191 | 191.236.123.34.bc.googleusercontent.com | US |
| 89.117.96.150 | server1.npl-usa-host.com | US |
| 91.206.200.156 | web718.default-host.net | UA |
| 177.221.140.101 | cloud101.americahost.cl | CL |
| 195.110.38.168 | 195.110.38.168 | IR |
| 69.163.181.218 | pdx1-shared-a1-33.dreamhost.com | US |
| 35.193.246.28 | 28.246.193.35.bc.googleusercontent.com | US |
| 107.180.116.234 | _unknown.ip.secureserver.net | US |
| 207.96.149.53 | web2.kpio2.com | CA |
| 78.142.63.55 | cloud.hostingnovapyme22.com | BG |
| 162.240.146.135 | mail.mypmspace.com | US |
| 16.170.232.212 | ec2-16-170-232-212.eu-north-1.compute.amazonaws.com | SE |
| 185.191.76.27 | 185.191.76.27 | IR |
| 169.255.59.77 | lithium.web4africa.net | ZA |
| 34.136.120.175 | 175.120.136.34.bc.googleusercontent.com | US |
| 94.182.172.194 | sht.194.172.182.94.euhosted.com | IR |
| 198.54.120.73 | premium52.web-hosting.com | US |
| 86.110.243.227 | web227.webhouse.sk | SK |
| IP地址(46) | 服务器名称 | 所属国家 |
|---|---|---|
| 20.191.45.212 | 20.191.45.212 | IE |
| 40.88.21.235 | 40.88.21.235 | US |
| 107.21.1.8 | ec2-107-21-1-8.compute-1.amazonaws.com | US |
| 54.208.102.37 | ec2-54-208-102-37.compute-1.amazonaws.com | US |
| 23.21.227.63 | ec2-23-21-227-63.compute-1.amazonaws.com | US |
| 54.208.88.53 | ec2-54-208-88-53.compute-1.amazonaws.com | US |
| 107.23.49.117 | ec2-107-23-49-117.compute-1.amazonaws.com | US |
| 54.208.80.140 | ec2-54-208-80-140.compute-1.amazonaws.com | US |
| 107.23.45.196 | ec2-107-23-45-196.compute-1.amazonaws.com | US |
| 157.7.104.94 | lit736.phy.lolipop.jp | JP |
| 34.67.78.24 | 24.78.67.34.bc.googleusercontent.com | US |
| 118.27.100.212 | www180.onamae.ne.jp | JP |
| 46.4.252.224 | pl7.fakat.net | ? |
| 23.227.137.226 | emerald8.doveserver.com | ? |
| 103.57.222.10 | nethost-1511.inet.vn | VN |
| 101.34.244.228 | 101.34.244.228 | CN |
| 149.202.50.200 | vps-bd0fb58f.vps.ovh.net | FR |
| 45.63.6.107 | vip12.3wns.com | ? |
| 64.31.47.66 | s12.hosterpk.com | US |
| 164.132.171.176 | ns3047943.ip-164-132-171.eu | FR |
| 184.168.114.42 | 42.114.168.184.host.secureserver.net | SG |
| 64.31.43.186 | s11.hosterpk.com | US |
| 57.129.1.90 | mail.vapefully.com | DE |
| 203.161.63.181 | server1.pageturner-authors.com | US |
| 142.93.155.125 | systemsaholic.com | CA |
| 185.7.252.210 | furud.elkdata.ee | EE |
| 89.32.45.46 | xhosting9.xservers.ro | RO |
| 62.171.175.151 | paris.softowebservice.com | DE |
| 64.31.22.34 | s21.hosterpk.com | US |
| 50.62.177.9 | p3plcpnl0751.prod.phx3.secureserver.net | US |
| 35.228.201.97 | 97.201.228.35.bc.googleusercontent.com | FI |
| 173.233.87.66 | webserver79.turnkeywebspace.com | US |
| 92.205.52.167 | sh20167.ispgateway.de | FR |
| 66.85.157.26 | gains.arrowsupercloud.com | US |
| 34.134.180.8 | 8.180.134.34.bc.googleusercontent.com | US |
| 118.27.99.19 | www109.onamae.ne.jp | JP |
| 34.30.130.95 | 95.130.30.34.bc.googleusercontent.com | US |
| 34.71.83.157 | 157.83.71.34.bc.googleusercontent.com | US |
| 118.27.99.23 | www113.conoha.ne.jp | JP |
| 34.41.188.139 | 139.188.41.34.bc.googleusercontent.com | US |
| 86.110.243.30 | u3.webhouse.sk | SK |
| 104.154.171.63 | 63.171.154.104.bc.googleusercontent.com | US |
| 66.29.141.249 | premium277.web-hosting.com | US |
| 34.116.114.90 | 90.114.116.34.bc.googleusercontent.com | AU |
| 116.90.60.141 | vmres13.web-servers.com.au | AU |
| 185.36.231.20 | 20-231-36-185.static.hostiran.name | IR |
| IP地址(9) | 服务器名称 | 所属国家 |
|---|---|---|
| 40.76.173.151 | 40.76.173.151 | US |
| 40.76.162.191 | 40.76.162.191 | US |
| 52.142.24.149 | 52.142.24.149 | US |
| 40.76.163.7 | 40.76.163.7 | US |
| 20.185.79.47 | 20.185.79.47 | US |
| 40.76.163.23 | 40.76.163.23 | US |
| 40.76.162.208 | 40.76.162.208 | US |
| 20.185.79.15 | 20.185.79.15 | US |
| 52.142.26.175 | 52.142.26.175 | US |
| 40.76.162.247 | 40.76.162.247 | US |
| 52.204.97.54 | duckduckbot.duckduckgo.com | US |
| 54.208.100.253 | ec2-54-208-100-253.compute-1.amazonaws.com | US |
| 50.16.241.114 | ec2-50-16-241-114.compute-1.amazonaws.com | US |
| 52.5.190.19 | ec2-52-5-190-19.compute-1.amazonaws.com | US |
| 50.16.241.117 | duckduckbot.duckduckgo.com | US |
| 50.16.241.113 | ec2-50-16-241-113.compute-1.amazonaws.com | US |
| 54.197.234.188 | ec2-54-197-234-188.compute-1.amazonaws.com | US |
| 50.16.247.234 | duckduckbot.duckduckgo.com | US |
| 23.21.227.69 | duckduckbot.duckduckgo.com | US |
| 54.197.242.0 | ec2-54-197-242-0.compute-1.amazonaws.com | US |
| 183.90.253.11 | sv1410.xserver.jp | JP |
| 101.34.244.228 | 101.34.244.228 | CN |
| 72.167.253.29 | 29.253.167.72.host.secureserver.net | US |
| 34.69.224.251 | 251.224.69.34.bc.googleusercontent.com | US |
| 162.0.209.75 | business88.web-hosting.com | US |
| 183.181.81.79 | sv10398.xserver.jp | JP |
| 203.161.63.181 | server1.pageturner-authors.com | US |
| 68.183.229.214 | ns345.naxza.com | SG |
| 50.62.141.175 | 175.141.62.50.host.secureserver.net | US |
| 194.5.188.58 | 194.5.188.58 | IR |
| 202.181.99.12 | www292.sakura.ne.jp | JP |
| 64.31.47.66 | s12.hosterpk.com | US |
| 198.23.62.125 | altar42.supremepanel42.com | US |
| 157.7.104.94 | lit736.phy.lolipop.jp | JP |
| 164.132.171.176 | ns3047943.ip-164-132-171.eu | FR |
| 34.133.34.10 | 10.34.133.34.bc.googleusercontent.com | US |
| 34.173.15.200 | 200.15.173.34.bc.googleusercontent.com | US |
| 57.129.1.90 | mail.vapefully.com | DE |
| 35.244.116.233 | 233.116.244.35.bc.googleusercontent.com | AU |
| 50.62.177.225 | p3plcpnl0956.prod.phx3.secureserver.net | US |
| 34.140.126.92 | 92.126.140.34.bc.googleusercontent.com | BE |
| 50.63.15.171 | 171.15.63.50.host.secureserver.net | US |
| 45.118.146.199 | 45.118.146.199 | VN |
| 89.32.45.46 | xhosting9.xservers.ro | RO |
| 195.110.38.168 | 195.110.38.168 | IR |
| 14.194.247.195 | static-195.247.194.14-tataidc.co.in | IN |
| 118.27.99.23 | www113.conoha.ne.jp | JP |
| 45.158.12.116 | server.cukurovayazilim.com.tr | TR |
| 34.41.188.139 | 139.188.41.34.bc.googleusercontent.com | US |
| 185.182.57.43 | vserver262.axc.nl | GB |
| IP地址(18) | 服务器名称 | 所属国家 |
|---|---|---|
| 52.204.97.54 | duckduckbot.duckduckgo.com | US |
| 54.208.100.253 | duckduckbot.duckduckgo.com | US |
| 50.16.241.114 | ec2-50-16-241-114.compute-1.amazonaws.com | US |
| 52.5.190.19 | duckduckbot.duckduckgo.com | US |
| 50.16.241.117 | duckduckbot.duckduckgo.com | US |
| 50.16.241.113 | ec2-50-16-241-113.compute-1.amazonaws.com | US |
| 54.197.234.188 | ec2-54-197-234-188.compute-1.amazonaws.com | US |
| 50.16.247.234 | duckduckbot.duckduckgo.com | US |
| 23.21.227.69 | duckduckbot.duckduckgo.com | US |
| 54.197.242.0 | ec2-54-197-242-0.compute-1.amazonaws.com | US |
| IP地址(9) | 服务器名称 | 所属国家 |
|---|---|---|
| 52.204.97.54 | duckduckbot.duckduckgo.com | US |
| 40.76.173.151 | 40.76.173.151 | US |
| 54.197.234.188 | ec2-54-197-234-188.compute-1.amazonaws.com | US |
| 50.16.241.117 | duckduckbot.duckduckgo.com | US |
| 54.208.100.253 | ec2-54-208-100-253.compute-1.amazonaws.com | US |
| 52.5.190.19 | ec2-52-5-190-19.compute-1.amazonaws.com | US |
| 50.16.241.114 | ec2-50-16-241-114.compute-1.amazonaws.com | US |
| 50.16.241.113 | ec2-50-16-241-113.compute-1.amazonaws.com | US |
| 50.16.247.234 | duckduckbot.duckduckgo.com | US |
一般不要拦截。搜索引擎爬虫为搜索引擎提供动力,是用户发现您网站的有效途径。事实上,拦截搜索引擎爬虫可能会严重减少网站的自然流量。
您可以通过在网站的 robots.txt 中设置用户代理访问规则来屏蔽 DuckDuckBot 或限制其访问权限。我们建议安装 Spider Analyser 插件,以检查它是否真正遵循这些规则。
# robots.txt # 下列代码一般情况可以拦截该代理 User-agent: DuckDuckBot Disallow: /
您无需手动执行此操作,可通过我们的 Wordpress 插件 Spider Analyser 来拦截不必要的蜘蛛或者爬虫。
DuckDuckBot是DuckDuckGo的网络爬虫。DuckDuckBot的工作是不断改进DuckDuck搜索引擎的搜索结果,为用户提供最好和最安全的搜索体验。它尊重WWW::RobotRules并来自这些IP地址。
20.191.45.212 40.88.21.235 40.76.173.151 40.76.163.7 20.185.79.47 52.142.26.175 20.185.79.15 52.142.24.149 40.76.162.208 40.76.163.23 40.76.162.191 40.76.162.247
(工作日 10:00 - 18:30 为您服务)