如何做搜尋引擎蜘蛛日誌分析
搜尋引擎蜘蛛日誌檔案是一種非常強大但未被站長充分利用的檔案，分析它可以獲取有關每個搜尋引擎如何爬取網站內容的相關資訊點，及檢視搜尋引擎蜘蛛在一段時間內的行為。
| IP地址(98) | 伺服器名稱 | 所屬國家 |
|---|---|---|
| 44.210.119.82 | ec2-44-210-119-82.compute-1.amazonaws.com | US |
| 44.210.119.78 | ec2-44-210-119-78.compute-1.amazonaws.com | US |
| 3.101.216.99 | ec2-3-101-216-99.us-west-1.compute.amazonaws.com | US |
| 3.101.226.148 | ec2-3-101-226-148.us-west-1.compute.amazonaws.com | US |
| 3.101.216.122 | ec2-3-101-216-122.us-west-1.compute.amazonaws.com | US |
| 3.101.216.65 | ec2-3-101-216-65.us-west-1.compute.amazonaws.com | US |
| 3.101.216.74 | ec2-3-101-216-74.us-west-1.compute.amazonaws.com | US |
| 3.101.226.176 | ec2-3-101-226-176.us-west-1.compute.amazonaws.com | US |
| 207.244.227.143 | vmi922544.contaboserver.net | US |
| 207.244.231.45 | vmi922545.contaboserver.net | US |
| 44.210.119.67 | ec2-44-210-119-67.compute-1.amazonaws.com | US |
| 13.57.251.133 | ec2-13-57-251-133.us-west-1.compute.amazonaws.com | US |
| 3.101.226.154 | ec2-3-101-226-154.us-west-1.compute.amazonaws.com | US |
| 3.101.230.171 | ec2-3-101-230-171.us-west-1.compute.amazonaws.com | US |
| 3.101.230.229 | ec2-3-101-230-229.us-west-1.compute.amazonaws.com | US |
| 3.101.216.123 | ec2-3-101-216-123.us-west-1.compute.amazonaws.com | US |
| 3.101.226.184 | ec2-3-101-226-184.us-west-1.compute.amazonaws.com | US |
| 3.101.230.187 | ec2-3-101-230-187.us-west-1.compute.amazonaws.com | US |
| 3.101.230.154 | ec2-3-101-230-154.us-west-1.compute.amazonaws.com | US |
| 3.101.226.165 | ec2-3-101-226-165.us-west-1.compute.amazonaws.com | US |
| 3.101.230.147 | ec2-3-101-230-147.us-west-1.compute.amazonaws.com | US |
| 3.101.230.233 | ec2-3-101-230-233.us-west-1.compute.amazonaws.com | US |
| 3.101.230.205 | ec2-3-101-230-205.us-west-1.compute.amazonaws.com | US |
| 3.101.230.202 | ec2-3-101-230-202.us-west-1.compute.amazonaws.com | US |
| 3.101.226.170 | ec2-3-101-226-170.us-west-1.compute.amazonaws.com | US |
| 3.101.226.149 | ec2-3-101-226-149.us-west-1.compute.amazonaws.com | US |
| 3.101.226.161 | ec2-3-101-226-161.us-west-1.compute.amazonaws.com | US |
| 3.101.216.71 | ec2-3-101-216-71.us-west-1.compute.amazonaws.com | US |
| 44.210.119.91 | ec2-44-210-119-91.compute-1.amazonaws.com | US |
| 44.210.119.87 | ec2-44-210-119-87.compute-1.amazonaws.com | US |
| 44.210.119.79 | ec2-44-210-119-79.compute-1.amazonaws.com | US |
| 3.101.239.89 | ec2-3-101-239-89.us-west-1.compute.amazonaws.com | US |
| 3.101.240.24 | ec2-3-101-240-24.us-west-1.compute.amazonaws.com | US |
| 44.210.119.76 | ec2-44-210-119-76.compute-1.amazonaws.com | US |
| 44.210.119.94 | ec2-44-210-119-94.compute-1.amazonaws.com | US |
| 44.210.119.89 | ec2-44-210-119-89.compute-1.amazonaws.com | US |
| 3.101.239.92 | ec2-3-101-239-92.us-west-1.compute.amazonaws.com | US |
| 3.101.239.90 | ec2-3-101-239-90.us-west-1.compute.amazonaws.com | US |
| 3.101.239.104 | ec2-3-101-239-104.us-west-1.compute.amazonaws.com | US |
| 3.101.240.65 | ec2-3-101-240-65.us-west-1.compute.amazonaws.com | US |
| 44.210.119.69 | ec2-44-210-119-69.compute-1.amazonaws.com | US |
| 3.101.240.153 | ec2-3-101-240-153.us-west-1.compute.amazonaws.com | US |
| 3.101.230.227 | ec2-3-101-230-227.us-west-1.compute.amazonaws.com | US |
| 3.101.240.108 | ec2-3-101-240-108.us-west-1.compute.amazonaws.com | US |
| 3.101.239.81 | ec2-3-101-239-81.us-west-1.compute.amazonaws.com | US |
| 3.101.226.145 | ec2-3-101-226-145.us-west-1.compute.amazonaws.com | US |
| 3.101.240.106 | ec2-3-101-240-106.us-west-1.compute.amazonaws.com | US |
| 3.101.226.167 | ec2-3-101-226-167.us-west-1.compute.amazonaws.com | US |
| 3.101.226.132 | ec2-3-101-226-132.us-west-1.compute.amazonaws.com | US |
| 44.210.119.84 | ec2-44-210-119-84.compute-1.amazonaws.com | US |
| 3.101.230.196 | ec2-3-101-230-196.us-west-1.compute.amazonaws.com | US |
| 3.101.216.100 | ec2-3-101-216-100.us-west-1.compute.amazonaws.com | US |
| 44.210.119.75 | ec2-44-210-119-75.compute-1.amazonaws.com | US |
一般不攔截。此類爬蟲通常是網站所有者提交掃描請求才會出現。如果攔截，則無法執行相應的掃描動作。
您可以通過在網站的 robots.txt 中設定使用者代理訪問規則來遮蔽 Tenable crawler 或限制其訪問許可權。我們建議安裝 Spider Analyser 外掛，以檢查它是否真正遵循這些規則。
# robots.txt # 下列程式碼一般情況可以攔截該代理 User-agent: Tenable crawler Disallow: /
您無需手動執行此操作，可通過我們的 Wordpress 外掛 Spider Analyser 來攔截不必要的蜘蛛或者爬蟲。
