
如何做搜尋引擎蜘蛛日誌分析
搜尋引擎蜘蛛日誌檔案是一種非常強大但未被站長充分利用的檔案,分析它可以獲取有關每個搜尋引擎如何爬取網站內容的相關資訊點,及檢視搜尋引擎蜘蛛在一段時間內的行為。
IP地址(25) | 伺服器名稱 | 所屬國家 |
---|---|---|
157.245.117.25 | spider-crawler9.do.feeder.co | US |
157.245.117.2 | spider-crawler8.do.feeder.co | US |
159.89.183.157 | spider-crawler4.do.feeder.co | US |
174.138.49.150 | spider-crawler5.do.feeder.co | US |
104.131.105.190 | spider-crawler7.do.feeder.co | US |
68.183.157.237 | spider-crawler3.do.feeder.co | US |
167.71.89.133 | spider-crawler10.do.feeder.co | US |
159.89.43.145 | spider-crawler6.do.feeder.co | US |
68.183.151.227 | spider-crawler1.do.feeder.co | US |
174.138.57.235 | spider-crawler2.do.feeder.co | US |
IP地址(15) | 伺服器名稱 | 所屬國家 |
---|---|---|
2600:3c03::f03c:91ff:fe08:7a05 | crawler3.spider.feeder.co | US |
2600:3c03::f03c:91ff:fe3e:eef | crawler6.spider.feeder.co | US |
2600:3c03::f03c:91ff:fe08:7a61 | crawler4.spider.feeder.co | US |
45.79.191.117 | crawler4.spider.feeder.co | US |
2600:3c03::f03c:91ff:fe3e:ee3 | crawler5.spider.feeder.co | US |
97.107.132.213 | crawler2.spider.feeder.co | US |
2600:3c03::f03c:91ff:fe08:79c3 | crawler2.spider.feeder.co | US |
2600:3c03::f03c:91ff:fe08:7903 | crawler1.spider.feeder.co | US |
45.33.89.177 | crawler1.spider.feeder.co | US |
45.79.147.125 | crawler6.spider.feeder.co | US |
IP地址(14) | 伺服器名稱 | 所屬國家 |
---|---|---|
2600:3c03::f03c:91ff:fe08:7a05 | crawler3.spider.feeder.co | US |
2600:3c03::f03c:91ff:fe08:79c3 | crawler2.spider.feeder.co | US |
97.107.132.213 | crawler2.spider.feeder.co | US |
2600:3c03::f03c:91ff:fe08:7a61 | crawler4.spider.feeder.co | US |
45.79.147.125 | crawler6.spider.feeder.co | US |
2600:3c03::f03c:91ff:fe08:7903 | crawler1.spider.feeder.co | US |
2600:3c03::f03c:91ff:fe3e:ee3 | crawler5.spider.feeder.co | US |
2600:3c03::f03c:91ff:fe3e:eef | crawler6.spider.feeder.co | US |
45.79.139.96 | li1238-96.members.linode.com | US |
45.79.152.182 | li1251-182.members.linode.com | US |
IP地址(16) | 伺服器名稱 | 所屬國家 |
---|---|---|
161.35.246.138 | ? | NL |
164.92.218.89 | 164.92.218.89 | NL |
138.91.26.194 | 138.91.26.194 | JP |
164.90.255.66 | 164.90.255.66 | US |
157.230.195.44 | 157.230.195.44 | SG |
206.189.247.132 | 206.189.247.132 | GB |
174.138.57.235 | spider-crawler2.do.feeder.co | US |
68.183.151.227 | spider-crawler1.do.feeder.co | US |
159.89.183.157 | spider-crawler4.do.feeder.co | US |
104.131.105.190 | spider-crawler7.do.feeder.co | US |
2400:6180:0:d0::1672:b001 | 2400:6180:0:d0::1672:b001 | SG |
167.71.89.133 | spider-crawler10.do.feeder.co | US |
157.245.117.25 | spider-crawler9.do.feeder.co | US |
157.245.117.2 | spider-crawler8.do.feeder.co | US |
174.138.49.150 | spider-crawler5.do.feeder.co | US |
68.183.157.237 | spider-crawler3.do.feeder.co | US |
159.89.43.145 | spider-crawler6.do.feeder.co | US |
2600:3c03::f03c:91ff:fe08:7a05 | crawler3.spider.feeder.co | US |
2600:3c03::f03c:91ff:fe3e:eef | crawler6.spider.feeder.co | US |
2600:3c03::f03c:91ff:fe08:7a61 | crawler4.spider.feeder.co | US |
45.79.191.117 | crawler4.spider.feeder.co | US |
2600:3c03::f03c:91ff:fe3e:ee3 | crawler5.spider.feeder.co | US |
97.107.132.213 | crawler2.spider.feeder.co | US |
2600:3c03::f03c:91ff:fe08:79c3 | crawler2.spider.feeder.co | US |
2600:3c03::f03c:91ff:fe08:7903 | crawler1.spider.feeder.co | US |
45.33.89.177 | crawler1.spider.feeder.co | US |
45.79.147.125 | crawler6.spider.feeder.co | US |
45.79.139.96 | crawler5.spider.feeder.co | US |
45.79.152.182 | crawler3.spider.feeder.co | US |
2600:3c03::f03c:91ff:fe05:c4ee | spider-worker2.spider.feeder.co | US |
104.248.13.205 | spider-worker2.do.feeder.co | ? |
104.131.26.198 | spider-worker3.do.feeder.co | US |
68.235.33.187 | static-68-235-33-187.cust.tzulo.com | US |
51.158.236.129 | 129-236-158-51.instances.scw.cloud | FR |
IP地址(16) | 伺服器名稱 | 所屬國家 |
---|---|---|
45.79.152.182 | crawler3.spider.feeder.co | US |
97.107.132.213 | crawler2.spider.feeder.co | US |
45.33.89.177 | crawler1.spider.feeder.co | US |
45.79.139.96 | crawler5.spider.feeder.co | US |
45.79.191.117 | crawler4.spider.feeder.co | US |
45.79.147.125 | crawler6.spider.feeder.co | US |
2600:3c03::f03c:91ff:fe05:c4ee | spider-worker2.spider.feeder.co | US |
2600:3c03::f03c:91ff:fe3e:ee3 | crawler5.spider.feeder.co | US |
2600:3c03::f03c:91ff:fe08:79c3 | crawler2.spider.feeder.co | US |
2600:3c03::f03c:91ff:fe08:7a05 | crawler3.spider.feeder.co | US |
通常不需要。除非您不希望資訊流網站或者APP對您的網站內容進行抓取,網站也不提供Feed訂閱服務,則可以考慮攔截此型別爬蟲。
您可以通過在網站的 robots.txt 中設定使用者代理訪問規則來遮蔽 feeder.co 或限制其訪問許可權。我們建議安裝 Spider Analyser 外掛,以檢查它是否真正遵循這些規則。
# robots.txt # 下列程式碼一般情況可以攔截該代理 User-agent: feeder.co Disallow: /
您無需手動執行此操作,可通過我們的 Wordpress 外掛 Spider Analyser 來攔截不必要的蜘蛛或者爬蟲。
Feeder是一個新聞管理器,它跟蹤你選擇的任何線上來源,並將其捆綁成一個易於消化的閱讀體驗。