
如何做搜尋引擎蜘蛛日誌分析
搜尋引擎蜘蛛日誌檔案是一種非常強大但未被站長充分利用的檔案,分析它可以獲取有關每個搜尋引擎如何爬取網站內容的相關資訊點,及檢視搜尋引擎蜘蛛在一段時間內的行為。
IP地址(1) | 伺服器名稱 | 所屬國家 |
---|---|---|
199.87.250.224 | h-250-224.scoutjet.com | US |
199.87.254.157 | h-254-157.scoutjet.com | US |
199.87.254.76 | h-254-76.scoutjet.com | US |
199.87.254.57 | h-254-57.scoutjet.com | US |
199.87.254.218 | h-254-218.scoutjet.com | US |
199.87.255.28 | h-255-28.scoutjet.com | US |
199.87.255.61 | h-255-61.scoutjet.com | US |
199.87.254.202 | h-254-202.scoutjet.com | US |
38.99.97.71 | h-97-71.scoutjet.com | US |
199.87.254.237 | h-254-237.scoutjet.com | US |
38.108.180.76 | h-180-76.scoutjet.com | US |
IP地址(1) | 伺服器名稱 | 所屬國家 |
---|---|---|
38.108.180.76 | h-180-76.scoutjet.com | US |
IP地址(274) | 伺服器名稱 | 所屬國家 |
---|---|---|
199.87.252.141 | h-252-141.scoutjet.com | US |
199.87.252.87 | h-252-87.scoutjet.com | US |
199.87.253.40 | h-254-40.scoutjet.com | US |
199.87.252.180 | h-252-180.scoutjet.com | US |
199.87.253.66 | h-254-66.scoutjet.com | US |
199.87.253.76 | h-254-76.scoutjet.com | US |
199.87.252.116 | h-252-116.scoutjet.com | US |
199.87.252.103 | h-252-103.scoutjet.com | US |
199.87.252.56 | h-252-56.scoutjet.com | US |
199.87.253.111 | h-254-111.scoutjet.com | US |
199.87.252.123 | h-252-123.scoutjet.com | US |
199.87.252.83 | h-252-83.scoutjet.com | US |
199.87.250.236 | h-250-236.scoutjet.com | US |
199.87.252.47 | h-252-47.scoutjet.com | US |
199.87.253.95 | h-254-95.scoutjet.com | US |
199.87.252.186 | h-252-186.scoutjet.com | US |
199.87.253.59 | h-254-59.scoutjet.com | US |
199.87.252.153 | h-252-153.scoutjet.com | US |
199.87.253.118 | h-254-118.scoutjet.com | US |
199.87.252.164 | h-252-164.scoutjet.com | US |
199.87.252.25 | h-252-25.scoutjet.com | US |
199.87.253.73 | h-254-73.scoutjet.com | US |
199.87.252.165 | h-252-165.scoutjet.com | US |
199.87.253.131 | h-254-131.scoutjet.com | US |
199.87.252.134 | h-252-134.scoutjet.com | US |
199.87.252.97 | h-252-97.scoutjet.com | US |
199.87.252.101 | h-252-101.scoutjet.com | US |
199.87.252.60 | h-252-60.scoutjet.com | US |
38.99.97.40 | h-97-40.scoutjet.com | US |
199.87.252.112 | h-252-112.scoutjet.com | US |
對於未知蜘蛛或者爬蟲。它的用途對網站來說可能是好的,也可能是壞的,這取決於它是什麼。所以說,這需要站長進一步分析判斷這些尚不明確的爬蟲行為,再作最終決定。 但,根據以往的經驗,未宣告行為目的及未命名的蜘蛛爬蟲,通常都有不可告人的祕密,我們理應對其行為進行控制,比如攔截。
您可以通過在網站的 robots.txt 中設定使用者代理訪問規則來遮蔽 ScoutJet 或限制其訪問許可權。我們建議安裝 Spider Analyser 外掛,以檢查它是否真正遵循這些規則。
# robots.txt # 下列程式碼一般情況可以攔截該代理 User-agent: ScoutJet Disallow: /
您無需手動執行此操作,可通過我們的 Wordpress 外掛 Spider Analyser 來攔截不必要的蜘蛛或者爬蟲。