如何做搜索引擎蜘蛛日志分析
搜索引擎蜘蛛日志文件是一种非常强大但未被站长充分利用的文件,分析它可以获取有关每个搜索引擎如何爬取网站内容的相关信息点,及查看搜索引擎蜘蛛在一段时间内的行为。
| IP地址(67) | 服务器名称 | 所属国家 |
|---|---|---|
| 35.216.229.155 | 155.229.216.35.bc.googleusercontent.com | CH |
| 35.216.216.224 | 224.216.216.35.bc.googleusercontent.com | CH |
| 35.216.231.87 | 87.231.216.35.bc.googleusercontent.com | CH |
| 35.216.239.19 | 19.239.216.35.bc.googleusercontent.com | CH |
| 35.216.241.78 | 78.241.216.35.bc.googleusercontent.com | CH |
| 35.216.205.216 | 216.205.216.35.bc.googleusercontent.com | CH |
| 35.216.237.60 | 60.237.216.35.bc.googleusercontent.com | CH |
| 35.216.199.51 | 51.199.216.35.bc.googleusercontent.com | CH |
| 35.216.225.253 | 253.225.216.35.bc.googleusercontent.com | CH |
| 35.216.229.26 | 26.229.216.35.bc.googleusercontent.com | CH |
| 179.43.168.146 | hostedby.privatelayer.com | CH |
| 45.148.10.235 | 45.148.10.235 | NL |
| 179.43.163.250 | hostedby.privatelayer.com | CH |
| IP地址(7) | 服务器名称 | 所属国家 |
|---|---|---|
| 179.43.168.146 | hostedby.privatelayer.com | CH |
| 185.7.33.149 | 185.7.33.149 | SE |
| 212.102.41.44 | unn-212-102-41-44.cdn77.com | US |
| 185.162.235.162 | 185.162.235.162 | RU |
| 185.162.235.175 | 185.162.235.175 | RU |
| 45.148.10.235 | 45.148.10.235 | NL |
| 179.43.163.250 | hostedby.privatelayer.com | CH |
| IP地址(14) | 服务器名称 | 所属国家 |
|---|---|---|
| 139.144.150.205 | textgenerator.scan.leakix.org | US |
| 164.92.84.255 | javaslang.scan.leakix.org | US |
| 134.122.89.242 | covertocover.scan.leakix.org | DE |
| 139.59.138.49 | uitemplates.scan.leakix.org | DE |
| 164.90.205.35 | flaskalternative.scan.leakix.org | NL |
| 165.227.146.2 | writethedocs.scan.leakix.org | US |
| 162.243.184.251 | truth.scan.leakix.org | US |
| 139.59.230.191 | devcraft.scan.leakix.org | SG |
| 64.227.126.135 | pharmaceutical.scan.leakix.org | DE |
| 165.22.74.203 | aitechnology.scan.leakix.org | DE |
| 159.65.58.104 | etsy.scan.leakix.org | GB |
| 165.22.120.216 | advise.scan.leakix.org | GB |
| 68.183.64.176 | freelancers.scan.leakix.org | ? |
| 142.93.64.15 | secretkey.scan.leakix.org | US |
| 161.35.190.56 | nojavascript.scan.leakix.org | US |
| 164.90.222.93 | datastorages.scan.leakix.org | DE |
| 138.68.163.10 | ubiquitouslanguage.scan.leakix.org | GB |
| 46.101.103.192 | minimalviableproduct.scan.leakix.org | DE |
| 164.92.192.25 | projectupdate.scan.leakix.org | US |
| 161.35.27.144 | hashtags.scan.leakix.org | DE |
| 206.81.1.88 | marketingtrends.scan.leakix.org | US |
| 178.62.73.12 | connascence.scan.leakix.org | GB |
| 139.144.96.150 | readconcern.scan.leakix.org | AU |
| 192.53.126.23 | darkmagic.scan.leakix.org | US |
| 207.154.225.47 | portability.scan.leakix.org | DE |
| 165.232.76.155 | neopets.scan.leakix.org | DE |
| 104.248.140.11 | quantifiers.scan.leakix.org | DE |
| 128.199.62.55 | startpage.scan.leakix.org | NL |
| 139.144.150.8 | businessvalue.scan.leakix.org | GB |
| 137.184.106.30 | bundler.scan.leakix.org | US |
| 142.93.158.96 | inspiration.scan.leakix.org | CA |
| 137.184.162.65 | c11.scan.leakix.org | CA |
| 144.126.202.105 | 144.126.202.105 | GB |
| 139.144.150.26 | component.scan.leakix.org | US |
| 172.104.102.196 | statisticswithr.scan.leakix.org | JP |
| 207.154.240.169 | nodepty.scan.leakix.org | DE |
| 128.199.61.251 | crystal.scan.leakix.org | NL |
| 172.105.37.32 | developeradvocates.scan.leakix.org | IN |
| 139.59.182.142 | vscodeextensions.scan.leakix.org | GB |
| 138.68.133.118 | win32.scan.leakix.org | GB |
| 74.207.237.46 | office.scan.leakix.org | ? |
| 167.99.184.41 | tasktiger.scan.leakix.org | CA |
| 74.207.237.114 | sentiment.scan.leakix.org | US |
| 167.172.232.142 | fad.scan.leakix.org | US |
| 139.59.65.144 | responsiveimages.scan.leakix.org | IN |
| 162.243.186.177 | datamodel.scan.leakix.org | US |
| 134.122.34.144 | codekata.scan.leakix.org | CA |
| 128.199.195.68 | 128.199.195.68 | SG |
| 137.184.222.107 | elasticsearchhq.scan.leakix.org | US |
| 143.110.218.229 | dockerdesktop.scan.leakix.org | CA |
| 159.223.102.13 | gprc.scan.leakix.org | US |
| 162.243.161.105 | subgrid.scan.leakix.org | US |
| 167.71.48.191 | comic.scan.leakix.org | DE |
| 167.71.185.75 | ittraining.scan.leakix.org | US |
| 142.93.153.3 | httpinterceptor.scan.leakix.org | CA |
| 45.79.83.159 | drush.scan.leakix.org | US |
| 198.199.121.22 | api.scan.leakix.org | US |
| 161.35.155.246 | algorhythm.scan.leakix.org | NL |
| 134.122.63.192 | facepalm.scan.leakix.org | NL |
| 178.128.151.41 | stm.scan.leakix.org | US |
| 139.144.150.45 | platformdeveloper.scan.leakix.org | US |
| 146.190.98.165 | coderdojo.scan.leakix.org | US |
| 138.197.88.136 | vuecli.scan.leakix.org | US |
| 159.65.138.217 | theater.scan.leakix.org | SG |
| 159.203.182.222 | graphqlyoga.scan.leakix.org | US |
| 159.203.63.67 | distance.scan.leakix.org | CA |
| 161.35.176.95 | unittest.scan.leakix.org | US |
| 146.190.64.200 | gophercon.scan.leakix.org | US |
| 143.198.72.96 | testtask.scan.leakix.org | US |
| 147.182.168.210 | codegolf.scan.leakix.org | US |
| 159.223.108.26 | comehelpme.scan.leakix.org | US |
| 167.172.20.95 | freecodecamporg.scan.leakix.org | US |
| 167.99.8.63 | url.scan.leakix.org | US |
| 178.62.3.65 | friends.scan.leakix.org | GB |
| 143.42.118.5 | opensourceproject.scan.leakix.org | US |
| 139.144.150.23 | appusers.scan.leakix.org | US |
| 147.182.130.98 | dashboard.scan.leakix.org | US |
| 45.55.193.222 | challengingtask.scan.leakix.org | US |
| 104.236.193.132 | bulmacss.scan.leakix.org | US |
| 159.203.94.228 | verticalrhythm.scan.leakix.org | US |
| 165.22.108.223 | 165.22.108.223 | SG |
| 159.203.44.43 | playback.scan.leakix.org | CA |
| 143.110.156.182 | asni.scan.leakix.org | US |
| 137.184.150.232 | inversionofcontrol.scan.leakix.org | US |
| 144.126.198.24 | gitmoji.scan.leakix.org | GB |
| 64.227.32.66 | d103188940.scan.leakix.org | GB |
| 157.245.105.107 | f894f8ec11.scan.leakix.org | IN |
| 159.89.12.166 | c5d51acfea.scan.leakix.org | DE |
| 188.166.87.88 | 188.166.87.88 | NL |
| 139.162.141.82 | 139-162-141-82.ip.linodeusercontent.com | DE |
| 138.197.191.87 | ca65e15345.scan.leakix.org | DE |
| 46.101.1.225 | b812f4218d.scan.leakix.org | GB |
| 96.126.110.54 | 96-126-110-54.ip.linodeusercontent.com | US |
| 146.190.242.161 | a28759fb9c.scan.leakix.org | CA |
| 167.99.181.249 | b781e0bb13.scan.leakix.org | CA |
| 139.59.136.184 | c3ee778768.scan.leakix.org | DE |
| 139.59.143.102 | f1096f7e4e.scan.leakix.org | DE |
| 96.126.110.181 | 96-126-110-181.ip.linodeusercontent.com | US |
| 172.105.158.219 | 172-105-158-219.ip.linodeusercontent.com | US |
| 159.89.17.243 | a05c4808ab.scan.leakix.org | DE |
| 164.90.228.79 | fee8d5bfdc.scan.leakix.org | DE |
| 64.226.65.160 | a46db02ec6.scan.leakix.org | DE |
| 165.227.173.41 | d559d155e9.scan.leakix.org | DE |
| 143.110.213.72 | cec2b92574.scan.leakix.org | CA |
| 206.189.225.181 | c8021b81a5.scan.leakix.org | US |
| 159.89.127.165 | b4bcd7c472.scan.leakix.org | CA |
| 46.101.111.185 | ffaffaab3a.scan.leakix.org | DE |
| 207.154.197.113 | bf57ea116e.scan.leakix.org | DE |
| 68.183.180.73 | e86207ecae.scan.leakix.org | SG |
| 64.225.75.246 | b32f2b056d.scan.leakix.org | NL |
| 206.189.2.13 | fb65c10da2.scan.leakix.org | NL |
| 178.128.207.138 | eab9c05722.scan.leakix.org | DE |
| 207.154.212.47 | a4d39b522e.scan.leakix.org | DE |
| 139.162.155.225 | 139-162-155-225.ip.linodeusercontent.com | DE |
| 165.22.34.189 | a36d657dc7.scan.leakix.org | US |
| 157.245.36.108 | a93200c42e.scan.leakix.org | GB |
| 206.81.24.74 | be0f5ba2c6.scan.leakix.org | DE |
| 206.189.19.19 | b37662257c.scan.leakix.org | GB |
| 209.97.180.8 | a0d8574844.scan.leakix.org | GB |
| 68.183.9.16 | fafb352f31.scan.leakix.org | NL |
| 139.162.210.205 | 139-162-210-205.ip.linodeusercontent.com | GB |
| 157.230.19.140 | d9fc35a06b.scan.leakix.org | DE |
| 64.226.78.121 | d12a1cb769.scan.leakix.org | DE |
| 142.93.129.190 | f20a02ce01.scan.leakix.org | NL |
| 165.22.235.3 | d9b49875ee.scan.leakix.org | CA |
| 167.99.210.137 | b2dcc8edcd.scan.leakix.org | NL |
| 206.81.24.227 | b59bc1c6ef.scan.leakix.org | DE |
| 164.92.244.132 | ffc5722872.scan.leakix.org | DE |
| 172.105.16.117 | 172-105-16-117.ip.linodeusercontent.com | CA |
| 146.190.63.48 | cdac169393.scan.leakix.org | US |
| 165.227.39.235 | c53df711d7.scan.leakix.org | CA |
| 165.227.84.14 | d3b29af448.scan.leakix.org | US |
| 134.122.28.88 | e9b8e372f1.scan.leakix.org | US |
| 172.105.16.131 | 172-105-16-131.ip.linodeusercontent.com | CA |
| 206.189.233.36 | a957d52272.scan.leakix.org | US |
| 172.105.16.105 | 172-105-16-105.ip.linodeusercontent.com | CA |
| 209.38.208.202 | ac09637315.scan.leakix.org | DE |
| 188.166.108.93 | a7638675e3.scan.leakix.org | NL |
| 159.65.18.197 | a6efafe308.scan.leakix.org | GB |
| 134.209.25.199 | f952b6ebb7.scan.leakix.org | GB |
| 164.92.107.174 | dd761bf4f4.scan.leakix.org | US |
| 143.244.168.161 | b4ed9564d2.scan.leakix.org | US |
| 23.239.4.252 | 23-239-4-252.ip.linodeusercontent.com | US |
| 172.105.197.17 | 172-105-197-17.ip.linodeusercontent.com | JP |
| 138.68.86.32 | b69efeaf93.scan.leakix.org | DE |
| 159.223.132.86 | f090494790.scan.leakix.org | US |
| 138.68.82.23 | e154df25ea.scan.leakix.org | DE |
| 167.172.158.128 | d5904e1cdf.scan.leakix.org | US |
| 209.38.248.17 | c1fe727412.scan.leakix.org | DE |
| 146.190.63.248 | aaedd5bcff.scan.leakix.org | US |
| 206.81.12.187 | bf99e5305e.scan.leakix.org | US |
| 159.89.174.87 | f8fb72f228.scan.leakix.org | IN |
| 104.237.130.38 | 104-237-130-38.ip.linodeusercontent.com | US |
| 139.59.231.238 | e1b4837f7f.scan.leakix.org | SG |
| 64.227.70.2 | b46c9f9797.scan.leakix.org | NL |
| 142.93.143.8 | a4ac419f3c.scan.leakix.org | NL |
| 139.162.96.14 | 139-162-96-14.ip.linodeusercontent.com | JP |
| 64.23.218.208 | ed93d36780.scan.leakix.org | US |
| 164.90.208.56 | a6b22d76dd.scan.leakix.org | DE |
| 147.182.149.75 | dec04dc34a.scan.leakix.org | CA |
| 128.199.182.152 | cdffb2c5b1.scan.leakix.org | SG |
| 139.162.96.81 | 139-162-96-81.ip.linodeusercontent.com | JP |
| 96.126.110.74 | 96-126-110-74.ip.linodeusercontent.com | US |
| 128.199.182.55 | ea73d34464.scan.leakix.org | SG |
| 157.245.113.227 | dc16f0d67a.scan.leakix.org | US |
| 159.65.144.72 | a579b68427.scan.leakix.org | IN |
| 192.46.211.230 | 192-46-211-230.ip.linodeusercontent.com | IN |
| 159.203.96.42 | c06ac4526f.scan.leakix.org | US |
| 172.105.16.40 | 172-105-16-40.ip.linodeusercontent.com | CA |
| 167.71.81.114 | cb1190adfb.scan.leakix.org | US |
| 142.93.0.66 | b1cb777a43.scan.leakix.org | US |
| 157.245.204.205 | ca1b036c29.scan.leakix.org | SG |
| 23.239.21.238 | 23-239-21-238.ip.linodeusercontent.com | US |
| 143.110.217.244 | a1b2bc4e35.scan.leakix.org | CA |
| 139.59.132.8 | a113ac0491.scan.leakix.org | DE |
| 167.99.182.39 | c8e7cbf86a.scan.leakix.org | CA |
| 146.190.103.103 | d3ecc9518c.scan.leakix.org | SG |
| 147.182.200.94 | a5f4a43e57.scan.leakix.org | US |
| 167.71.175.236 | ca7e79b6df.scan.leakix.org | US |
| 138.68.144.227 | c165c2962c.scan.leakix.org | GB |
| 128.199.182.77 | b1b05ba60a.scan.leakix.org | SG |
| 139.162.101.202 | 139-162-101-202.ip.linodeusercontent.com | JP |
| 206.189.95.232 | e81def74b5.scan.leakix.org | SG |
| IP地址(11) | 服务器名称 | 所属国家 |
|---|---|---|
| 161.35.188.242 | probe-ny002.rand0.leakix.org | US |
| 185.162.235.162 | 185.162.235.162 | NL |
| 185.162.235.175 | 185.162.235.175 | NL |
| 167.71.13.196 | synprobe001.leakix.net | NL |
| 167.99.133.28 | probe-de001.rand0.leakix.org | DE |
| 161.35.86.181 | probe-nl001.rand0.leakix.org | NL |
| 134.122.112.12 | probe-ny001.rand0.leakix.org | US |
| 212.102.41.44 | unn-212-102-41-44.cdn77.com | US |
| 185.7.33.149 | 185.7.33.149 | SE |
| 179.43.168.146 | hostedby.privatelayer.com | CH |
| 45.148.10.235 | 45.148.10.235 | NL |
| IP地址(6) | 服务器名称 | 所属国家 |
|---|---|---|
| 134.122.112.12 | probe-ny001.rand0.leakix.org | US |
| 161.35.86.181 | probe-nl001.rand0.leakix.org | NL |
| 161.35.188.242 | probe-ny002.rand0.leakix.org | US |
| 143.198.136.88 | probe-ca001.rand0.leakix.org | US |
| 167.99.133.28 | probe-de001.rand0.leakix.org | DE |
| 167.71.13.196 | synprobe001.leakix.net | NL |
| IP地址(35) | 服务器名称 | 所属国家 |
|---|---|---|
| 194.195.242.241 | eu-central-scanner-296.scan0.leakix.org | DE |
| 139.177.183.58 | eu-central-scanner-291.scan0.leakix.org | US |
| 139.177.182.20 | eu-central-scanner-277.scan0.leakix.org | US |
| 194.195.245.52 | eu-central-scanner-307.scan0.leakix.org | DE |
| 194.195.246.59 | eu-central-scanner-313.scan0.leakix.org | DE |
| 194.195.243.6 | eu-central-scanner-293.scan0.leakix.org | DE |
| 172.105.95.115 | eu-central-scanner-433.scan0.leakix.org | US |
| 172.105.73.51 | eu-central-scanner-432.scan0.leakix.org | US |
| 172.105.94.4 | eu-central-scanner-423.scan0.leakix.org | US |
| 172.104.230.234 | eu-central-scanner-399.scan0.leakix.org | US |
| IP地址(4) | 服务器名称 | 所属国家 |
|---|---|---|
| 193.218.118.211 | 211.118.218.193.urdn.com.ua | UA |
| 104.244.74.57 | tor1.panhu.xyz | LU |
| 45.154.255.67 | tor-exit-2.keff.org | SE |
| 199.249.230.108 | tor18.quintex.com | US |
| IP地址(3) | 服务器名称 | 所属国家 |
|---|---|---|
| 109.70.100.57 | tor-exit-anonymizer.appliedprivacy.net | AT |
| 188.120.235.117 | drremmiz4.fvds.ru | RU |
| 205.185.127.217 | tor-exit.monoxyde.org | US |
| IP地址(1) | 服务器名称 | 所属国家 |
|---|---|---|
| 212.83.166.62 | as12876.tor.shh.sh | FR |
对于未知蜘蛛或者爬虫。它的用途对网站来说可能是好的,也可能是坏的,这取决于它是什么。所以说,这需要站长进一步分析判断这些尚不明确的爬虫行为,再作最终决定。 但,根据以往的经验,未声明行为目的及未命名的蜘蛛爬虫,通常都有不可告人的秘密,我们理应对其行为进行控制,比如拦截。
您可以通过在网站的 robots.txt 中设置用户代理访问规则来屏蔽 LeakIX bot 或限制其访问权限。我们建议安装 Spider Analyser 插件,以检查它是否真正遵循这些规则。
# robots.txt # 下列代码一般情况可以拦截该代理 User-agent: LeakIX bot Disallow: /
您无需手动执行此操作,可通过我们的 Wordpress 插件 Spider Analyser 来拦截不必要的蜘蛛或者爬虫。
LeakIX是第一个结合了搜索引擎对公共信息的索引和与结果相关的开放报告平台的平台。
我们打算提供一个先发制人的解决方案,在我们索引的最合理的数据上信任个人研究人员和安全公司,提供一个明确的事件报告,我们也帮助确定哪些信息已经/可能受到影响,以及如何解决这个问题。
我们的首要目标是预防,所有的自愿报告都是免费的。
(工作日 10:00 - 18:30 为您服务)