如何做搜尋引擎蜘蛛日誌分析
搜尋引擎蜘蛛日誌檔案是一種非常強大但未被站長充分利用的檔案，分析它可以獲取有關每個搜尋引擎如何爬取網站內容的相關資訊點，及檢視搜尋引擎蜘蛛在一段時間內的行為。
| IP地址(1) | 伺服器名稱 | 所屬國家 |
|---|---|---|
| 51.77.135.89 | ns31066279.ip-51-77-135.eu | FR |
| IP地址(1) | 伺服器名稱 | 所屬國家 |
|---|---|---|
| 104.238.188.207 | ? | FR |
| IP地址(53) | 伺服器名稱 | 所屬國家 |
|---|---|---|
| 202.143.148.61 | 202.143.148.61 | TH |
| 23.160.208.247 | relay13f.tor.ian.sh | US |
| 18.221.254.150 | ec2-18-221-254-150.us-east-2.compute.amazonaws.com | US |
| 80.82.70.118 | rnd.group-ib.ru | SC |
| 63.123.238.8 | 63.123.238.8 | US |
| 38.70.11.33 | 38.70.11.33 | US |
| 51.77.135.89 | ns31066279.ip-51-77-135.eu | FR |
| 104.238.188.207 | ? | FR |
| 65.214.45.217 | ? | US |
| 65.214.45.213 | ? | US |
| 65.214.45.143 | 65.214.45.143 | US |
| 65.214.45.148 | 65.214.45.148 | US |
| 66.235.124.57 | crawler5106.ask.com | US |
| 66.235.116.44 | crawler9044.ask.com | US |
| 66.235.124.81 | crawler5129.ask.com | US |
| 66.235.124.119 | crawler5141.ask.com | US |
| 66.235.124.196 | crawler6196.ask.com | US |
| 66.235.124.58 | crawler5107.ask.com | US |
| 66.235.124.59 | crawler5108.ask.com | US |
| 66.235.124.10 | crawler5010.ask.com | US |
| 66.235.124.98 | crawler5198.ask.com | US |
| 66.235.124.124 | crawler6124.ask.com | US |
| 66.235.124.2 | crawler5002.ask.com | US |
| 66.235.124.68 | crawler5116.ask.com | US |
| 104.234.204.134 | 104.234.204.134 | CA |
| 66.235.124.100 | crawler5200.ask.com | US |
| 66.235.124.99 | crawler5199.ask.com | US |
| 66.235.124.125 | crawler6125.ask.com | US |
| 66.235.124.202 | crawler6202.ask.com | US |
| 3.69.48.48 | ec2-3-69-48-48.eu-central-1.compute.amazonaws.com | DE |
| 54.167.12.189 | ec2-54-167-12-189.compute-1.amazonaws.com | US |
| 66.235.124.3 | crawler5003.ask.com | US |
| 66.235.124.73 | crawler5121.ask.com | US |
| 66.235.124.102 | crawler5202.ask.com | US |
| 66.235.124.127 | crawler6127.ask.com | US |
| 66.235.124.4 | crawler5004.ask.com | US |
| 66.235.124.74 | crawler5122.ask.com | US |
| 44.201.70.77 | ec2-44-201-70-77.compute-1.amazonaws.com | US |
| 66.235.124.55 | crawler5104.ask.com | US |
| 66.235.112.5 | tn-fw.ask.com | US |
| 66.235.124.109 | crawler5209.ask.com | US |
| 66.235.124.8 | crawler5008.ask.com | US |
| 66.235.124.192 | crawler6192.ask.com | US |
| 66.235.124.56 | crawler5105.ask.com | US |
| 159.223.138.47 | 159.223.138.47 | US |
| 66.235.124.1 | crawler5001.ask.com | US |
| 66.235.124.12 | crawler5012.ask.com | US |
| 66.235.124.89 | crawler5137.ask.com | US |
| 66.235.124.197 | crawler6197.ask.com | US |
| 54.160.206.72 | ec2-54-160-206-72.compute-1.amazonaws.com | US |
| 66.235.124.7 | crawler5007.ask.com | US |
| 66.235.124.101 | crawler5201.ask.com | US |
| 66.235.124.126 | crawler6126.ask.com | US |
| 54.183.113.139 | ec2-54-183-113-139.us-west-1.compute.amazonaws.com | US |
| 66.235.116.43 | crawler9043.ask.com | US |
| 66.235.124.80 | crawler5128.ask.com | US |
| 66.235.124.113 | crawler5213.ask.com | US |
| 66.235.124.193 | crawler6193.ask.com | US |
| 35.83.250.11 | ec2-35-83-250-11.us-west-2.compute.amazonaws.com | US |
| 66.235.124.54 | crawler5103.ask.com | US |
| 78.153.140.151 | hostglobal.plus | GB |
| 66.235.124.103 | crawler5203.ask.com | US |
| 66.235.124.13 | crawler5013.ask.com | US |
| 66.235.124.5 | crawler5005.ask.com | US |
| 45.200.148.16 | 45.200.148.16 | ID |
| 178.215.236.240 | 178.215.236.240 | FR |
| 66.235.124.78 | crawler5126.ask.com | US |
| 66.235.124.104 | crawler5204.ask.com | US |
| 66.235.124.139 | crawler6139.ask.com | US |
| 66.235.124.79 | crawler5127.ask.com | US |
| 18.246.40.49 | ec2-18-246-40-49.us-west-2.compute.amazonaws.com | US |
| 66.235.124.105 | crawler5205.ask.com | US |
| 66.235.124.15 | crawler5015.ask.com | US |
| 100.25.24.228 | ec2-100-25-24-228.compute-1.amazonaws.com | US |
| 54.169.152.253 | ec2-54-169-152-253.ap-southeast-1.compute.amazonaws.com | SG |
| IP地址(53) | 伺服器名稱 | 所屬國家 |
|---|---|---|
| 65.214.45.217 | ? | US |
| 65.214.45.213 | ? | US |
| 65.214.45.143 | 65.214.45.143 | US |
| 65.214.45.148 | 65.214.45.148 | US |
| 66.235.116.44 | crawler9044.ask.com | US |
| 66.235.124.89 | crawler5137.ask.com | US |
| 66.235.124.58 | crawler5107.ask.com | US |
| 66.235.124.7 | crawler5007.ask.com | US |
| 66.235.124.103 | crawler5203.ask.com | US |
| 66.235.124.125 | crawler6125.ask.com | US |
對於未知蜘蛛或者爬蟲。它的用途對網站來說可能是好的，也可能是壞的，這取決於它是什麼。所以說，這需要站長進一步分析判斷這些尚不明確的爬蟲行為，再作最終決定。 但，根據以往的經驗，未宣告行為目的及未命名的蜘蛛爬蟲，通常都有不可告人的祕密，我們理應對其行為進行控制，比如攔截。
您可以通過在網站的 robots.txt 中設定使用者代理訪問規則來遮蔽 Ask Jeeves/Teoma 或限制其訪問許可權。我們建議安裝 Spider Analyser 外掛，以檢查它是否真正遵循這些規則。
# robots.txt # 下列程式碼一般情況可以攔截該代理 User-agent: Ask Jeeves/Teoma Disallow: /
您無需手動執行此操作，可通過我們的 Wordpress 外掛 Spider Analyser 來攔截不必要的蜘蛛或者爬蟲。
