
如何做搜尋引擎蜘蛛日誌分析
搜尋引擎蜘蛛日誌檔案是一種非常強大但未被站長充分利用的檔案,分析它可以獲取有關每個搜尋引擎如何爬取網站內容的相關資訊點,及檢視搜尋引擎蜘蛛在一段時間內的行為。
IP地址(4) | 伺服器名稱 | 所屬國家 |
---|---|---|
74.125.212.9 | google-proxy-74-125-212-9.google.com | US |
64.233.172.5 | google-proxy-64-233-172-5.google.com | US |
74.125.214.41 | google-proxy-74-125-214-41.google.com | US |
66.249.84.74 | google-proxy-66-249-84-74.google.com | US |
64.233.172.8 | google-proxy-64-233-172-8.google.com | US |
66.102.8.9 | google-proxy-66-102-8-9.google.com | US |
66.102.8.7 | google-proxy-66-102-8-7.google.com | US |
66.102.8.5 | google-proxy-66-102-8-5.google.com | US |
66.249.83.45 | google-proxy-66-249-83-45.google.com | US |
74.125.214.39 | google-proxy-74-125-214-39.google.com | US |
64.233.172.11 | google-proxy-64-233-172-11.google.com | US |
66.102.7.119 | google-proxy-66-102-7-119.google.com | US |
66.249.80.187 | google-proxy-66-249-80-187.google.com | US |
IP地址(4) | 伺服器名稱 | 所屬國家 |
---|---|---|
74.125.214.39 | google-proxy-74-125-214-39.google.com | US |
64.233.172.11 | google-proxy-64-233-172-11.google.com | US |
66.102.7.119 | google-proxy-66-102-7-119.google.com | US |
66.249.80.187 | google-proxy-66-249-80-187.google.com | US |
IP地址(68) | 伺服器名稱 | 所屬國家 |
---|---|---|
18.234.215.92 | ec2-18-234-215-92.compute-1.amazonaws.com | US |
54.226.1.15 | ec2-54-226-1-15.compute-1.amazonaws.com | US |
34.224.168.40 | ec2-34-224-168-40.compute-1.amazonaws.com | US |
107.20.4.237 | ec2-107-20-4-237.compute-1.amazonaws.com | US |
54.172.166.97 | ec2-54-172-166-97.compute-1.amazonaws.com | US |
100.26.41.244 | ec2-100-26-41-244.compute-1.amazonaws.com | US |
54.90.249.139 | ec2-54-90-249-139.compute-1.amazonaws.com | US |
54.91.69.59 | ec2-54-91-69-59.compute-1.amazonaws.com | US |
34.229.12.216 | ec2-34-229-12-216.compute-1.amazonaws.com | US |
54.91.107.113 | ec2-54-91-107-113.compute-1.amazonaws.com | US |
52.91.158.198 | ec2-52-91-158-198.compute-1.amazonaws.com | US |
34.201.138.145 | ec2-34-201-138-145.compute-1.amazonaws.com | US |
54.234.235.126 | ec2-54-234-235-126.compute-1.amazonaws.com | US |
3.85.17.197 | ec2-3-85-17-197.compute-1.amazonaws.com | US |
3.91.48.123 | ec2-3-91-48-123.compute-1.amazonaws.com | US |
54.242.121.208 | ec2-54-242-121-208.compute-1.amazonaws.com | US |
54.204.82.128 | ec2-54-204-82-128.compute-1.amazonaws.com | US |
54.234.17.206 | ec2-54-234-17-206.compute-1.amazonaws.com | US |
3.91.98.196 | ec2-3-91-98-196.compute-1.amazonaws.com | US |
52.73.15.57 | ec2-52-73-15-57.compute-1.amazonaws.com | US |
34.235.153.71 | ec2-34-235-153-71.compute-1.amazonaws.com | US |
54.174.225.191 | ec2-54-174-225-191.compute-1.amazonaws.com | US |
54.161.7.19 | ec2-54-161-7-19.compute-1.amazonaws.com | US |
IP地址(68) | 伺服器名稱 | 所屬國家 |
---|---|---|
34.201.138.145 | ec2-34-201-138-145.compute-1.amazonaws.com | US |
54.234.235.126 | ec2-54-234-235-126.compute-1.amazonaws.com | US |
3.85.17.197 | ec2-3-85-17-197.compute-1.amazonaws.com | US |
3.91.48.123 | ec2-3-91-48-123.compute-1.amazonaws.com | US |
54.242.121.208 | ec2-54-242-121-208.compute-1.amazonaws.com | US |
54.204.82.128 | ec2-54-204-82-128.compute-1.amazonaws.com | US |
54.234.17.206 | ec2-54-234-17-206.compute-1.amazonaws.com | US |
3.91.98.196 | ec2-3-91-98-196.compute-1.amazonaws.com | US |
52.73.15.57 | ec2-52-73-15-57.compute-1.amazonaws.com | US |
34.235.153.71 | ec2-34-235-153-71.compute-1.amazonaws.com | US |
可能不需要,尤其是如果你自己也受益於資訊蒐集服務。不過,如果你擔心伺服器資源佔用等問題,可以選擇攔截它們。
您可以通過在網站的 robots.txt 中設定使用者代理訪問規則來遮蔽 BrandVerity 或限制其訪問許可權。我們建議安裝 Spider Analyser 外掛,以檢查它是否真正遵循這些規則。
# robots.txt # 下列程式碼一般情況可以攔截該代理 User-agent: BrandVerity Disallow: /
您無需手動執行此操作,可通過我們的 Wordpress 外掛 Spider Analyser 來攔截不必要的蜘蛛或者爬蟲。