如何做搜尋引擎蜘蛛日誌分析
搜尋引擎蜘蛛日誌檔案是一種非常強大但未被站長充分利用的檔案，分析它可以獲取有關每個搜尋引擎如何爬取網站內容的相關資訊點，及檢視搜尋引擎蜘蛛在一段時間內的行為。
| IP地址(107) | 伺服器名稱 | 所屬國家 |
|---|---|---|
| 44.197.251.99 | ec2-44-197-251-99.compute-1.amazonaws.com | US |
| 54.205.233.143 | ec2-54-205-233-143.compute-1.amazonaws.com | US |
| 35.171.26.112 | ec2-35-171-26-112.compute-1.amazonaws.com | US |
| 52.70.217.87 | ec2-52-70-217-87.compute-1.amazonaws.com | US |
| 54.173.40.53 | ec2-54-173-40-53.compute-1.amazonaws.com | US |
| 3.219.45.152 | ec2-3-219-45-152.compute-1.amazonaws.com | US |
| 3.94.163.146 | ec2-3-94-163-146.compute-1.amazonaws.com | US |
| 44.197.203.101 | ec2-44-197-203-101.compute-1.amazonaws.com | US |
| 54.198.194.230 | ec2-54-198-194-230.compute-1.amazonaws.com | US |
| 54.147.164.255 | ec2-54-147-164-255.compute-1.amazonaws.com | US |
| 34.207.117.146 | ec2-34-207-117-146.compute-1.amazonaws.com | US |
| 44.200.15.216 | ec2-44-200-15-216.compute-1.amazonaws.com | US |
| 54.90.70.241 | ec2-54-90-70-241.compute-1.amazonaws.com | US |
| 18.209.230.189 | ec2-18-209-230-189.compute-1.amazonaws.com | ? |
| 44.202.54.110 | ec2-44-202-54-110.compute-1.amazonaws.com | US |
| 44.203.83.7 | ec2-44-203-83-7.compute-1.amazonaws.com | US |
| 34.201.50.79 | ec2-34-201-50-79.compute-1.amazonaws.com | US |
| 44.203.251.238 | ec2-44-203-251-238.compute-1.amazonaws.com | US |
| 3.234.220.90 | ec2-3-234-220-90.compute-1.amazonaws.com | US |
| 44.211.65.169 | ec2-44-211-65-169.compute-1.amazonaws.com | US |
| 34.201.169.242 | ec2-34-201-169-242.compute-1.amazonaws.com | US |
| 35.173.237.215 | ec2-35-173-237-215.compute-1.amazonaws.com | US |
| 174.129.167.13 | ec2-174-129-167-13.compute-1.amazonaws.com | US |
| 54.160.71.137 | ec2-54-160-71-137.compute-1.amazonaws.com | US |
| 100.26.254.230 | ec2-100-26-254-230.compute-1.amazonaws.com | US |
| 18.232.98.25 | ec2-18-232-98-25.compute-1.amazonaws.com | US |
| 54.234.218.132 | ec2-54-234-218-132.compute-1.amazonaws.com | US |
| 18.206.74.2 | ec2-18-206-74-2.compute-1.amazonaws.com | US |
| 34.224.3.76 | ec2-34-224-3-76.compute-1.amazonaws.com | US |
| 18.232.173.16 | ec2-18-232-173-16.compute-1.amazonaws.com | US |
| 212.47.238.235 | 235-238-47-212.instances.scw.cloud | ? |
| 34.230.61.10 | ec2-34-230-61-10.compute-1.amazonaws.com | US |
| 54.88.192.131 | ec2-54-88-192-131.compute-1.amazonaws.com | US |
| 34.207.179.153 | ec2-34-207-179-153.compute-1.amazonaws.com | US |
| 44.212.90.203 | ec2-44-212-90-203.compute-1.amazonaws.com | US |
| 54.225.47.51 | ec2-54-225-47-51.compute-1.amazonaws.com | US |
| 44.204.244.86 | ec2-44-204-244-86.compute-1.amazonaws.com | US |
| 3.87.78.75 | ec2-3-87-78-75.compute-1.amazonaws.com | US |
| 3.226.253.39 | ec2-3-226-253-39.compute-1.amazonaws.com | US |
| 44.202.244.186 | ec2-44-202-244-186.compute-1.amazonaws.com | US |
| 18.209.16.125 | ec2-18-209-16-125.compute-1.amazonaws.com | US |
| 54.160.195.164 | ec2-54-160-195-164.compute-1.amazonaws.com | US |
| 54.226.188.48 | ec2-54-226-188-48.compute-1.amazonaws.com | US |
| 18.210.17.225 | ec2-18-210-17-225.compute-1.amazonaws.com | US |
| 3.236.154.48 | ec2-3-236-154-48.compute-1.amazonaws.com | US |
| 54.86.139.116 | ec2-54-86-139-116.compute-1.amazonaws.com | ? |
| 44.211.60.77 | ec2-44-211-60-77.compute-1.amazonaws.com | US |
| 54.237.6.144 | ec2-54-237-6-144.compute-1.amazonaws.com | US |
| 44.200.208.113 | ec2-44-200-208-113.compute-1.amazonaws.com | US |
| 54.144.217.21 | ec2-54-144-217-21.compute-1.amazonaws.com | US |
| 54.166.219.193 | ec2-54-166-219-193.compute-1.amazonaws.com | US |
| 34.227.224.82 | ec2-34-227-224-82.compute-1.amazonaws.com | US |
| 100.26.247.179 | ec2-100-26-247-179.compute-1.amazonaws.com | US |
| 3.239.7.84 | ec2-3-239-7-84.compute-1.amazonaws.com | US |
| 44.201.251.126 | ec2-44-201-251-126.compute-1.amazonaws.com | US |
| 54.159.123.169 | ec2-54-159-123-169.compute-1.amazonaws.com | US |
| 34.202.235.21 | ec2-34-202-235-21.compute-1.amazonaws.com | US |
| 54.221.63.73 | ec2-54-221-63-73.compute-1.amazonaws.com | US |
| 54.226.179.19 | ec2-54-226-179-19.compute-1.amazonaws.com | US |
| 44.197.113.186 | ec2-44-197-113-186.compute-1.amazonaws.com | US |
| 3.229.226.150 | ec2-3-229-226-150.compute-1.amazonaws.com | US |
| 44.207.153.71 | ec2-44-207-153-71.compute-1.amazonaws.com | US |
| IP地址(3) | 伺服器名稱 | 所屬國家 |
|---|---|---|
| 3.85.184.181 | ec2-3-85-184-181.compute-1.amazonaws.com | US |
| 44.200.138.149 | ec2-44-200-138-149.compute-1.amazonaws.com | US |
| 3.235.251.191 | ec2-3-235-251-191.compute-1.amazonaws.com | US |
對於未知蜘蛛或者爬蟲。它的用途對網站來說可能是好的，也可能是壞的，這取決於它是什麼。所以說，這需要站長進一步分析判斷這些尚不明確的爬蟲行為，再作最終決定。 但，根據以往的經驗，未宣告行為目的及未命名的蜘蛛爬蟲，通常都有不可告人的祕密，我們理應對其行為進行控制，比如攔截。
您可以通過在網站的 robots.txt 中設定使用者代理訪問規則來遮蔽 SiteScoreBot 或限制其訪問許可權。我們建議安裝 Spider Analyser 外掛，以檢查它是否真正遵循這些規則。
# robots.txt # 下列程式碼一般情況可以攔截該代理 User-agent: SiteScoreBot Disallow: /
您無需手動執行此操作，可通過我們的 Wordpress 外掛 Spider Analyser 來攔截不必要的蜘蛛或者爬蟲。
