如何做搜尋引擎蜘蛛日誌分析
搜尋引擎蜘蛛日誌檔案是一種非常強大但未被站長充分利用的檔案，分析它可以獲取有關每個搜尋引擎如何爬取網站內容的相關資訊點，及檢視搜尋引擎蜘蛛在一段時間內的行為。
| IP地址(12) | 伺服器名稱 | 所屬國家 |
|---|---|---|
| 35.180.57.200 | ec2-35-180-57-200.eu-west-3.compute.amazonaws.com | FR |
| 15.236.212.64 | ec2-15-236-212-64.eu-west-3.compute.amazonaws.com | FR |
| 13.39.22.8 | ec2-13-39-22-8.eu-west-3.compute.amazonaws.com | FR |
| 13.38.217.138 | ec2-13-38-217-138.eu-west-3.compute.amazonaws.com | FR |
| 35.180.227.60 | ec2-35-180-227-60.eu-west-3.compute.amazonaws.com | ? |
| 35.180.65.131 | ec2-35-180-65-131.eu-west-3.compute.amazonaws.com | FR |
| 35.181.1.5 | ec2-35-181-1-5.eu-west-3.compute.amazonaws.com | FR |
| 13.36.194.243 | ec2-13-36-194-243.eu-west-3.compute.amazonaws.com | FR |
| 13.36.235.103 | ec2-13-36-235-103.eu-west-3.compute.amazonaws.com | FR |
| 15.237.44.217 | ec2-15-237-44-217.eu-west-3.compute.amazonaws.com | FR |
| 15.236.95.9 | ec2-15-236-95-9.eu-west-3.compute.amazonaws.com | FR |
| 13.37.225.0 | ec2-13-37-225-0.eu-west-3.compute.amazonaws.com | FR |
| 15.236.226.8 | ec2-15-236-226-8.eu-west-3.compute.amazonaws.com | FR |
| 13.39.82.127 | ec2-13-39-82-127.eu-west-3.compute.amazonaws.com | FR |
| 35.181.1.163 | ec2-35-181-1-163.eu-west-3.compute.amazonaws.com | FR |
| 13.37.237.40 | ec2-13-37-237-40.eu-west-3.compute.amazonaws.com | FR |
| 35.180.83.26 | ec2-35-180-83-26.eu-west-3.compute.amazonaws.com | FR |
| 13.38.33.35 | ec2-13-38-33-35.eu-west-3.compute.amazonaws.com | FR |
| 13.38.97.37 | ec2-13-38-97-37.eu-west-3.compute.amazonaws.com | FR |
| 13.36.244.84 | ec2-13-36-244-84.eu-west-3.compute.amazonaws.com | FR |
| 15.188.48.190 | ec2-15-188-48-190.eu-west-3.compute.amazonaws.com | FR |
| 13.38.61.26 | ec2-13-38-61-26.eu-west-3.compute.amazonaws.com | FR |
| 13.37.230.128 | ec2-13-37-230-128.eu-west-3.compute.amazonaws.com | FR |
| 15.236.158.224 | ec2-15-236-158-224.eu-west-3.compute.amazonaws.com | FR |
| 35.180.24.214 | ec2-35-180-24-214.eu-west-3.compute.amazonaws.com | FR |
| 35.180.232.25 | ec2-35-180-232-25.eu-west-3.compute.amazonaws.com | FR |
| 15.188.15.34 | ec2-15-188-15-34.eu-west-3.compute.amazonaws.com | FR |
| 13.38.230.94 | ec2-13-38-230-94.eu-west-3.compute.amazonaws.com | FR |
| 213.246.45.63 | frhb60348ds.ikexpress.com | FR |
| 51.159.186.213 | 213-186-159-51.instances.scw.cloud | FR |
| 51.159.144.254 | 254-144-159-51.instances.scw.cloud | FR |
| 51.159.177.141 | 141-177-159-51.instances.scw.cloud | FR |
| 51.159.137.159 | 159-137-159-51.instances.scw.cloud | FR |
| 51.159.152.180 | 180-152-159-51.instances.scw.cloud | FR |
| 51.159.168.160 | 160-168-159-51.instances.scw.cloud | FR |
| 51.159.160.208 | 208-160-159-51.instances.scw.cloud | FR |
| 51.159.187.80 | 80-187-159-51.instances.scw.cloud | FR |
| 51.159.169.115 | 115-169-159-51.instances.scw.cloud | FR |
| 13.37.231.226 | ec2-13-37-231-226.eu-west-3.compute.amazonaws.com | FR |
| 13.37.237.113 | ec2-13-37-237-113.eu-west-3.compute.amazonaws.com | FR |
| 35.180.192.37 | ec2-35-180-192-37.eu-west-3.compute.amazonaws.com | ? |
| 13.38.106.196 | ec2-13-38-106-196.eu-west-3.compute.amazonaws.com | FR |
| 15.188.77.151 | ec2-15-188-77-151.eu-west-3.compute.amazonaws.com | FR |
| 13.38.91.25 | ec2-13-38-91-25.eu-west-3.compute.amazonaws.com | FR |
| 35.181.59.181 | ec2-35-181-59-181.eu-west-3.compute.amazonaws.com | FR |
| 13.38.230.71 | ec2-13-38-230-71.eu-west-3.compute.amazonaws.com | FR |
| 13.37.245.252 | ec2-13-37-245-252.eu-west-3.compute.amazonaws.com | FR |
| 35.180.137.255 | ec2-35-180-137-255.eu-west-3.compute.amazonaws.com | FR |
| 35.180.198.110 | ec2-35-180-198-110.eu-west-3.compute.amazonaws.com | FR |
| 15.237.51.228 | ec2-15-237-51-228.eu-west-3.compute.amazonaws.com | FR |
| 13.38.1.177 | ec2-13-38-1-177.eu-west-3.compute.amazonaws.com | FR |
| 13.38.251.230 | ec2-13-38-251-230.eu-west-3.compute.amazonaws.com | FR |
| 13.38.89.155 | ec2-13-38-89-155.eu-west-3.compute.amazonaws.com | FR |
| 13.38.73.138 | ec2-13-38-73-138.eu-west-3.compute.amazonaws.com | FR |
| 35.180.158.184 | ec2-35-180-158-184.eu-west-3.compute.amazonaws.com | FR |
| 13.38.245.174 | ec2-13-38-245-174.eu-west-3.compute.amazonaws.com | FR |
| 13.39.104.245 | ec2-13-39-104-245.eu-west-3.compute.amazonaws.com | FR |
| 35.181.54.75 | ec2-35-181-54-75.eu-west-3.compute.amazonaws.com | FR |
| 13.37.225.206 | ec2-13-37-225-206.eu-west-3.compute.amazonaws.com | FR |
| 13.37.230.71 | ec2-13-37-230-71.eu-west-3.compute.amazonaws.com | FR |
| 15.236.18.99 | ec2-15-236-18-99.eu-west-3.compute.amazonaws.com | FR |
| 35.181.50.44 | ec2-35-181-50-44.eu-west-3.compute.amazonaws.com | FR |
| 13.38.137.201 | ec2-13-38-137-201.eu-west-3.compute.amazonaws.com | FR |
| 15.237.50.142 | ec2-15-237-50-142.eu-west-3.compute.amazonaws.com | FR |
| 35.181.43.76 | ec2-35-181-43-76.eu-west-3.compute.amazonaws.com | FR |
| IP地址(7) | 伺服器名稱 | 所屬國家 |
|---|---|---|
| 213.246.45.63 | frhb60348ds.ikexpress.com | FR |
| 51.159.186.213 | 213-186-159-51.instances.scw.cloud | FR |
| 51.159.144.254 | 254-144-159-51.instances.scw.cloud | FR |
| 51.159.177.141 | 141-177-159-51.instances.scw.cloud | FR |
| 51.159.160.208 | 208-160-159-51.instances.scw.cloud | FR |
| 51.159.168.160 | 160-168-159-51.instances.scw.cloud | FR |
| 51.159.169.115 | 115-169-159-51.instances.scw.cloud | ? |
對於未知蜘蛛或者爬蟲。它的用途對網站來說可能是好的，也可能是壞的，這取決於它是什麼。所以說，這需要站長進一步分析判斷這些尚不明確的爬蟲行為，再作最終決定。 但，根據以往的經驗，未宣告行為目的及未命名的蜘蛛爬蟲，通常都有不可告人的祕密，我們理應對其行為進行控制，比如攔截。
您可以通過在網站的 robots.txt 中設定使用者代理訪問規則來遮蔽 EMDASH bot 或限制其訪問許可權。我們建議安裝 Spider Analyser 外掛，以檢查它是否真正遵循這些規則。
# robots.txt # 下列程式碼一般情況可以攔截該代理 User-agent: EMDASH bot Disallow: /
您無需手動執行此操作，可通過我們的 Wordpress 外掛 Spider Analyser 來攔截不必要的蜘蛛或者爬蟲。
