如何做搜尋引擎蜘蛛日誌分析
搜尋引擎蜘蛛日誌檔案是一種非常強大但未被站長充分利用的檔案，分析它可以獲取有關每個搜尋引擎如何爬取網站內容的相關資訊點，及檢視搜尋引擎蜘蛛在一段時間內的行為。
| IP地址(265) | 伺服器名稱 | 所屬國家 |
|---|---|---|
| 54.227.93.62 | ec2-54-227-93-62.compute-1.amazonaws.com | US |
| 54.90.204.205 | ec2-54-90-204-205.compute-1.amazonaws.com | US |
| 54.226.108.194 | ec2-54-226-108-194.compute-1.amazonaws.com | US |
| 54.91.54.24 | ec2-54-91-54-24.compute-1.amazonaws.com | US |
| 54.175.233.63 | ec2-54-175-233-63.compute-1.amazonaws.com | US |
| 34.229.6.164 | ec2-34-229-6-164.compute-1.amazonaws.com | US |
| 54.210.94.25 | ec2-54-210-94-25.compute-1.amazonaws.com | US |
| 54.235.4.164 | ec2-54-235-4-164.compute-1.amazonaws.com | US |
| 35.172.225.99 | ec2-35-172-225-99.compute-1.amazonaws.com | US |
| 18.208.130.112 | ec2-18-208-130-112.compute-1.amazonaws.com | US |
| 34.228.167.58 | ec2-34-228-167-58.compute-1.amazonaws.com | US |
| 3.91.205.117 | ec2-3-91-205-117.compute-1.amazonaws.com | US |
| 3.90.27.157 | ec2-3-90-27-157.compute-1.amazonaws.com | US |
| 34.207.119.173 | ec2-34-207-119-173.compute-1.amazonaws.com | US |
| 3.208.17.2 | ec2-3-208-17-2.compute-1.amazonaws.com | US |
| 34.201.92.164 | ec2-34-201-92-164.compute-1.amazonaws.com | US |
| 54.89.222.199 | ec2-54-89-222-199.compute-1.amazonaws.com | US |
| 54.146.213.219 | ec2-54-146-213-219.compute-1.amazonaws.com | US |
| 18.232.182.152 | ec2-18-232-182-152.compute-1.amazonaws.com | US |
| 107.21.20.185 | ec2-107-21-20-185.compute-1.amazonaws.com | US |
| 54.85.51.139 | ec2-54-85-51-139.compute-1.amazonaws.com | US |
| 54.91.186.52 | ec2-54-91-186-52.compute-1.amazonaws.com | US |
| 54.221.171.198 | ec2-54-221-171-198.compute-1.amazonaws.com | US |
| 54.221.62.139 | ec2-54-221-62-139.compute-1.amazonaws.com | US |
| 54.221.122.122 | ec2-54-221-122-122.compute-1.amazonaws.com | US |
| 54.205.142.92 | ec2-54-205-142-92.compute-1.amazonaws.com | US |
| 54.91.120.254 | ec2-54-91-120-254.compute-1.amazonaws.com | US |
| 52.4.186.160 | ec2-52-4-186-160.compute-1.amazonaws.com | US |
| 3.91.60.138 | ec2-3-91-60-138.compute-1.amazonaws.com | US |
| 54.209.239.234 | ec2-54-209-239-234.compute-1.amazonaws.com | US |
可以考慮攔截。。爬蟲通常會下載公開的網際網路內容，這些內容預設情況下可以免費訪問。不過，如果你不希望你的內容被用於未經授權的目的，你應該攔截它們。
您可以通過在網站的 robots.txt 中設定使用者代理訪問規則來遮蔽 nbertaupete95 或限制其訪問許可權。我們建議安裝 Spider Analyser 外掛，以檢查它是否真正遵循這些規則。
# robots.txt # 下列程式碼一般情況可以攔截該代理 User-agent: nbertaupete95 Disallow: /
您無需手動執行此操作，可通過我們的 Wordpress 外掛 Spider Analyser 來攔截不必要的蜘蛛或者爬蟲。
