如何做搜尋引擎蜘蛛日誌分析
搜尋引擎蜘蛛日誌檔案是一種非常強大但未被站長充分利用的檔案，分析它可以獲取有關每個搜尋引擎如何爬取網站內容的相關資訊點，及檢視搜尋引擎蜘蛛在一段時間內的行為。
| IP地址(91) | 伺服器名稱 | 所屬國家 |
|---|---|---|
| 5.255.253.34 | 5-255-253-34.spider.yandex.com | RU |
| 5.255.253.42 | 5-255-253-42.spider.yandex.com | RU |
| 5.255.253.36 | 5-255-253-36.spider.yandex.com | RU |
| 5.255.253.41 | 5-255-253-41.spider.yandex.com | RU |
| 5.255.253.46 | 5-255-253-46.spider.yandex.com | RU |
| 5.255.253.39 | 5-255-253-39.spider.yandex.com | RU |
| 141.8.142.38 | 141-8-142-38.spider.yandex.com | RU |
| 5.255.253.38 | 5-255-253-38.spider.yandex.com | RU |
| 5.255.231.84 | 5-255-231-84.spider.yandex.com | RU |
| 5.45.207.65 | 5-45-207-65.spider.yandex.com | RU |
| 141.8.142.33 | 141-8-142-33.spider.yandex.com | RU |
| 141.8.142.40 | 141-8-142-40.spider.yandex.com | RU |
| 5.255.253.37 | 5-255-253-37.spider.yandex.com | RU |
| 5.255.253.32 | 5-255-253-32.spider.yandex.com | RU |
| 141.8.142.46 | 141-8-142-46.spider.yandex.com | RU |
| 141.8.142.36 | 141-8-142-36.spider.yandex.com | RU |
| 5.255.253.33 | 5-255-253-33.spider.yandex.com | RU |
| 80.85.152.185 | rx350c.pserver.space | RU |
| 141.8.142.44 | 141-8-142-44.spider.yandex.com | RU |
| 173.255.175.58 | 173-255-175-58.gdl.as54138.net | US |
| 205.185.218.16 | 205-185-218-16.lim.as54138.net | US |
| 146.70.102.166 | 146.70.102.166 | RO |
| 178.175.129.173 | 178.175.129.173 | MD |
| 69.16.147.25 | 69-16-147-25.mxp.as54203.net | US |
| 205.185.218.15 | 205-185-218-15.lim.as54138.net | US |
| 87.249.139.241 | 87.249.139.241 | CZ |
| 205.185.193.13 | 205.185.193.13 | US |
| 103.107.197.56 | 103.107.197.56 | AU |
| 195.158.249.83 | 195.158.249.83 | SI |
| 138.199.58.218 | 138.199.58.218 | BR |
| 216.131.105.159 | 159.105.131.216.unassigned.reliablehosting.com | US |
| 185.147.213.94 | 185-147-213-94.arn.as54203.net | SE |
| 176.67.84.14 | 176-67-84-14.otp.as62651.net | RO |
| 198.181.163.43 | 198.181.163.43 | US |
| 45.9.250.120 | 45.9.250.120 | AE |
| 216.131.68.27 | 27.68.131.216.x.reliablehosting.com | LU |
| 176.67.87.17 | 176-67-87-17.bts.as62651.net | SK |
| 173.255.175.52 | 173-255-175-52.gdl.as54138.net | US |
| 84.247.50.245 | 84.247.50.245 | NO |
| 141.8.142.43 | 141-8-142-43.spider.yandex.com | RU |
| 138.199.58.230 | 138.199.58.230 | BR |
| 80.246.28.17 | 80.246.28.17 | AL |
| 84.247.50.243 | 84.247.50.243 | NO |
| 173.245.209.162 | 173-245-209-162.syd.as54203.net | AU |
| 141.8.142.45 | 141-8-142-45.spider.yandex.com | RU |
| 5.255.253.45 | 5-255-253-45.spider.yandex.com | RU |
| 87.249.139.237 | 87.249.139.237 | CZ |
| 116.90.73.6 | 116.90.73.6 | AU |
| 198.181.163.31 | 198.181.163.31 | US |
| 5.255.253.44 | 5-255-253-44.spider.yandex.com | RU |
| 154.47.29.69 | unn-154-47-29-69.datapacket.com | US |
| 37.120.144.234 | 37.120.144.234 | HU |
| 173.245.209.139 | 173-245-209-139.syd.as54203.net | AU |
| 173.245.219.117 | 173-245-219-117.icn.as54203.net | KR |
| 176.67.86.109 | 176-67-86-109.waw.as54203.net | PL |
| 103.4.31.138 | 103.4.31.138 | TW |
| 216.131.109.67 | 216-131-109-67.prg.as62651.net | CZ |
| 216.151.180.126 | 216-151-180-126.jfk.as54203.net | US |
| 173.255.175.16 | 173-255-175-16.gdl.as54138.net | US |
| 216.131.105.158 | 158.105.131.216.unassigned.reliablehosting.com | US |
| 69.16.157.14 | 69-16-157-14.mxp.as54203.net | IT |
| 45.133.192.167 | 45.133.192.167 | IS |
| 216.131.88.37 | 216-131-88-37.ams.as54203.net | NL |
| 185.147.212.60 | 185-147-212-60.cdg.as54203.net | FR |
| 216.131.106.73 | 73.106.131.216.client.dyn.strong-sf59.reliablehosting.com | US |
| 5.154.174.52 | 5.154.174.52 | PT |
| 216.131.104.146 | 146.104.131.216.client.dyn.strong-sf83.reliablehosting.com | US |
| 216.131.106.171 | 171.106.131.216.shared.ex1.reliablehosting.com | US |
| 103.209.255.93 | 103-209-255-93.tlv.as62651.net | IL |
| 216.131.105.30 | 30.105.131.216.srv.co66.reliablehosting.com | BE |
| 205.185.222.144 | 205-185-222-144.mia.as54203.net | US |
| 64.145.79.175 | 64-145-79-175.jfk.as54203.net | US |
| 37.46.113.239 | lux-net-ip.as51430.net | LU |
| 146.70.52.220 | 146.70.52.220 | RU |
| 89.149.22.74 | 89.149.22.74 | CH |
| 185.225.28.8 | 185.225.28.8 | MK |
| 37.46.113.226 | lux-net-ip.as51430.net | LU |
| 194.32.122.10 | 194.32.122.10 | LT |
可能不需要。工具型別爬蟲通常為網站所有者使用此類工具對網站進行相關服務請求才會出現。當然，實際情況需站長判斷後再作決定。
您可以通過在網站的 robots.txt 中設定使用者代理訪問規則來遮蔽 Yandex.Webmaster 或限制其訪問許可權。我們建議安裝 Spider Analyser 外掛，以檢查它是否真正遵循這些規則。
# robots.txt # 下列程式碼一般情況可以攔截該代理 User-agent: Yandex.Webmaster Disallow: /
您無需手動執行此操作，可通過我們的 Wordpress 外掛 Spider Analyser 來攔截不必要的蜘蛛或者爬蟲。
