如何做搜尋引擎蜘蛛日誌分析
搜尋引擎蜘蛛日誌檔案是一種非常強大但未被站長充分利用的檔案，分析它可以獲取有關每個搜尋引擎如何爬取網站內容的相關資訊點，及檢視搜尋引擎蜘蛛在一段時間內的行為。
| IP地址(8) | 伺服器名稱 | 所屬國家 |
|---|---|---|
| 3.38.135.32 | ec2-3-38-135-32.ap-northeast-2.compute.amazonaws.com | KR |
| 54.180.143.100 | ec2-54-180-143-100.ap-northeast-2.compute.amazonaws.com | KR |
| 54.93.112.237 | ec2-54-93-112-237.eu-central-1.compute.amazonaws.com | DE |
| 3.39.229.197 | ec2-3-39-229-197.ap-northeast-2.compute.amazonaws.com | KR |
| 3.69.242.206 | ec2-3-69-242-206.eu-central-1.compute.amazonaws.com | DE |
| 18.118.30.81 | ec2-18-118-30-81.us-east-2.compute.amazonaws.com | US |
| 13.212.243.26 | ec2-13-212-243-26.ap-southeast-1.compute.amazonaws.com | SG |
| 3.14.133.71 | ec2-3-14-133-71.us-east-2.compute.amazonaws.com | US |
| 13.125.211.144 | ec2-13-125-211-144.ap-northeast-2.compute.amazonaws.com | KR |
| 3.120.175.84 | ec2-3-120-175-84.eu-central-1.compute.amazonaws.com | DE |
| 18.117.10.207 | ec2-18-117-10-207.us-east-2.compute.amazonaws.com | US |
| 18.189.178.77 | ec2-18-189-178-77.us-east-2.compute.amazonaws.com | US |
| 18.192.206.115 | ec2-18-192-206-115.eu-central-1.compute.amazonaws.com | DE |
| 52.24.186.221 | ec2-52-24-186-221.us-west-2.compute.amazonaws.com | US |
| 54.93.177.28 | ec2-54-93-177-28.eu-central-1.compute.amazonaws.com | DE |
| 35.87.241.217 | ec2-35-87-241-217.us-west-2.compute.amazonaws.com | US |
| 3.73.86.229 | ec2-3-73-86-229.eu-central-1.compute.amazonaws.com | DE |
| 18.193.254.30 | ec2-18-193-254-30.eu-central-1.compute.amazonaws.com | ? |
| 54.201.104.171 | ec2-54-201-104-171.us-west-2.compute.amazonaws.com | ? |
| 3.67.39.57 | ec2-3-67-39-57.eu-central-1.compute.amazonaws.com | DE |
| 18.194.242.103 | ec2-18-194-242-103.eu-central-1.compute.amazonaws.com | DE |
| 52.15.235.142 | ec2-52-15-235-142.us-east-2.compute.amazonaws.com | US |
| 52.215.190.48 | ec2-52-215-190-48.eu-west-1.compute.amazonaws.com | IE |
| 54.183.177.59 | ec2-54-183-177-59.us-west-1.compute.amazonaws.com | US |
| 18.231.141.117 | ec2-18-231-141-117.sa-east-1.compute.amazonaws.com | BR |
| 13.125.43.53 | ec2-13-125-43-53.ap-northeast-2.compute.amazonaws.com | KR |
| 3.8.233.236 | ec2-3-8-233-236.eu-west-2.compute.amazonaws.com | GB |
| 34.254.98.129 | ec2-34-254-98-129.eu-west-1.compute.amazonaws.com | IE |
| 13.229.120.26 | ec2-13-229-120-26.ap-southeast-1.compute.amazonaws.com | SG |
| 3.253.84.110 | ec2-3-253-84-110.eu-west-1.compute.amazonaws.com | IE |
| 3.38.106.29 | ec2-3-38-106-29.ap-northeast-2.compute.amazonaws.com | KR |
| 18.119.108.223 | ec2-18-119-108-223.us-east-2.compute.amazonaws.com | US |
| 18.119.166.235 | ec2-18-119-166-235.us-east-2.compute.amazonaws.com | US |
| 18.119.129.146 | ec2-18-119-129-146.us-east-2.compute.amazonaws.com | US |
| 35.86.209.2 | ec2-35-86-209-2.us-west-2.compute.amazonaws.com | US |
| 35.92.156.181 | ec2-35-92-156-181.us-west-2.compute.amazonaws.com | US |
| 52.40.185.41 | ec2-52-40-185-41.us-west-2.compute.amazonaws.com | US |
| 3.73.51.233 | ec2-3-73-51-233.eu-central-1.compute.amazonaws.com | DE |
| 3.127.27.112 | ec2-3-127-27-112.eu-central-1.compute.amazonaws.com | ? |
| 52.78.71.100 | ec2-52-78-71-100.ap-northeast-2.compute.amazonaws.com | KR |
| 18.157.80.92 | ec2-18-157-80-92.eu-central-1.compute.amazonaws.com | DE |
| 54.180.140.71 | ec2-54-180-140-71.ap-northeast-2.compute.amazonaws.com | KR |
| 18.191.5.200 | ec2-18-191-5-200.us-east-2.compute.amazonaws.com | US |
| 35.86.243.48 | ec2-35-86-243-48.us-west-2.compute.amazonaws.com | US |
| 3.144.6.124 | ec2-3-144-6-124.us-east-2.compute.amazonaws.com | US |
| 18.119.136.60 | ec2-18-119-136-60.us-east-2.compute.amazonaws.com | US |
| 35.85.145.137 | ec2-35-85-145-137.us-west-2.compute.amazonaws.com | US |
| 18.224.251.111 | ec2-18-224-251-111.us-east-2.compute.amazonaws.com | US |
| 3.137.158.241 | ec2-3-137-158-241.us-east-2.compute.amazonaws.com | US |
| 3.73.32.157 | ec2-3-73-32-157.eu-central-1.compute.amazonaws.com | ? |
| 18.224.51.243 | ec2-18-224-51-243.us-east-2.compute.amazonaws.com | US |
| 3.71.99.156 | ec2-3-71-99-156.eu-central-1.compute.amazonaws.com | DE |
| 3.141.45.168 | ec2-3-141-45-168.us-east-2.compute.amazonaws.com | US |
| 35.158.137.156 | ec2-35-158-137-156.eu-central-1.compute.amazonaws.com | DE |
| 3.136.11.206 | ec2-3-136-11-206.us-east-2.compute.amazonaws.com | US |
| 3.78.222.168 | ec2-3-78-222-168.eu-central-1.compute.amazonaws.com | DE |
| 34.218.231.231 | ec2-34-218-231-231.us-west-2.compute.amazonaws.com | US |
| 3.137.181.118 | ec2-3-137-181-118.us-east-2.compute.amazonaws.com | US |
| 3.144.9.2 | ec2-3-144-9-2.us-east-2.compute.amazonaws.com | US |
| 18.116.14.165 | ec2-18-116-14-165.us-east-2.compute.amazonaws.com | US |
| 18.192.48.66 | ec2-18-192-48-66.eu-central-1.compute.amazonaws.com | DE |
| 3.37.175.237 | ec2-3-37-175-237.ap-northeast-2.compute.amazonaws.com | KR |
| 18.195.214.235 | ec2-18-195-214-235.eu-central-1.compute.amazonaws.com | DE |
| 18.217.91.238 | ec2-18-217-91-238.us-east-2.compute.amazonaws.com | US |
| 3.76.215.10 | ec2-3-76-215-10.eu-central-1.compute.amazonaws.com | DE |
| 52.79.35.145 | ec2-52-79-35-145.ap-northeast-2.compute.amazonaws.com | KR |
| 3.141.194.81 | ec2-3-141-194-81.us-east-2.compute.amazonaws.com | US |
| 18.116.163.140 | ec2-18-116-163-140.us-east-2.compute.amazonaws.com | US |
| 3.70.170.114 | ec2-3-70-170-114.eu-central-1.compute.amazonaws.com | DE |
| 18.197.158.97 | ec2-18-197-158-97.eu-central-1.compute.amazonaws.com | DE |
| 52.28.226.221 | ec2-52-28-226-221.eu-central-1.compute.amazonaws.com | DE |
| 3.34.125.62 | ec2-3-34-125-62.ap-northeast-2.compute.amazonaws.com | KR |
| 35.91.38.89 | ec2-35-91-38-89.us-west-2.compute.amazonaws.com | US |
| 3.123.6.72 | ec2-3-123-6-72.eu-central-1.compute.amazonaws.com | DE |
| 35.81.166.216 | ec2-35-81-166-216.us-west-2.compute.amazonaws.com | US |
| IP地址(8) | 伺服器名稱 | 所屬國家 |
|---|---|---|
| 18.194.242.103 | ec2-18-194-242-103.eu-central-1.compute.amazonaws.com | DE |
| 52.15.235.142 | ec2-52-15-235-142.us-east-2.compute.amazonaws.com | US |
| 52.215.190.48 | ec2-52-215-190-48.eu-west-1.compute.amazonaws.com | IE |
| 54.183.177.59 | ec2-54-183-177-59.us-west-1.compute.amazonaws.com | US |
| 18.231.141.117 | ec2-18-231-141-117.sa-east-1.compute.amazonaws.com | BR |
| 13.125.43.53 | ec2-13-125-43-53.ap-northeast-2.compute.amazonaws.com | KR |
| 3.8.233.236 | ec2-3-8-233-236.eu-west-2.compute.amazonaws.com | GB |
| 34.254.98.129 | ec2-34-254-98-129.eu-west-1.compute.amazonaws.com | IE |
對於未知蜘蛛或者爬蟲。它的用途對網站來說可能是好的，也可能是壞的，這取決於它是什麼。所以說，這需要站長進一步分析判斷這些尚不明確的爬蟲行為，再作最終決定。 但，根據以往的經驗，未宣告行為目的及未命名的蜘蛛爬蟲，通常都有不可告人的祕密，我們理應對其行為進行控制，比如攔截。
您可以通過在網站的 robots.txt 中設定使用者代理訪問規則來遮蔽 CheckMarkNetwork 或限制其訪問許可權。我們建議安裝 Spider Analyser 外掛，以檢查它是否真正遵循這些規則。
# robots.txt # 下列程式碼一般情況可以攔截該代理 User-agent: CheckMarkNetwork Disallow: /
您無需手動執行此操作，可通過我們的 Wordpress 外掛 Spider Analyser 來攔截不必要的蜘蛛或者爬蟲。
