DotBot DotBot

蜘蛛/爬虫名称
DotBot
开发商
SEOmoz, Inc.
官网
moz.com
当前状态
活动
  • DotBot/1.2
  • DotBot/1.1
  • DotBot/1.1
  • dotbot
用户代理字符串
Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)
类型
营销
首次出现
2021-03-18 12:51:46
最后出现
2024-10-10 18:19:03
遵循robots.txt
IP地址
22
来源
IP地址 服务器名称 所属国家
216.244.66.197 ? US
216.244.66.247 ? US
216.244.66.243 ? US
216.244.66.248 ? US
216.244.66.237 216-244-66-237.reverse.wowrack.com US
216.244.66.203 ? US
216.244.66.244 ? US
216.244.66.231 ? US
216.244.66.242 216.244.66.242 US
216.244.66.241 ? US
216.244.66.202 216.244.66.202 US
216.244.66.245 ? US
216.244.66.199 216.244.66.199 US
216.244.66.239 216.244.66.239 US
216.244.66.198 216.244.66.198 US
216.244.66.233 ? US
用户代理字符串
Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)
类型
营销
首次出现
2009-05-11 05:50:00
最后出现
2016-12-27 11:15:45
遵循robots.txt
IP地址
10
来源
IP地址 服务器名称 所属国家
35.153.133.186 ec2-35-153-133-186.compute-1.amazonaws.com US
54.196.89.197 ec2-54-196-89-197.compute-1.amazonaws.com US
54.196.168.34 ec2-54-196-168-34.compute-1.amazonaws.com US
54.224.84.45 ec2-54-224-84-45.compute-1.amazonaws.com US
54.80.131.215 ec2-54-80-131-215.compute-1.amazonaws.com US
18.206.188.9 ec2-18-206-188-9.compute-1.amazonaws.com ?
44.192.97.244 ec2-44-192-97-244.compute-1.amazonaws.com US
3.227.3.232 ec2-3-227-3-232.compute-1.amazonaws.com US
44.202.184.87 ec2-44-202-184-87.compute-1.amazonaws.com US
3.237.106.28 ec2-3-237-106-28.compute-1.amazonaws.com US
44.205.250.15 ec2-44-205-250-15.compute-1.amazonaws.com US
3.236.125.181 ec2-3-236-125-181.compute-1.amazonaws.com US
44.197.204.182 ec2-44-197-204-182.compute-1.amazonaws.com US
3.238.136.10 ec2-3-238-136-10.compute-1.amazonaws.com US
34.235.151.198 ec2-34-235-151-198.compute-1.amazonaws.com US
3.239.111.229 ec2-3-239-111-229.compute-1.amazonaws.com US
3.235.252.243 ec2-3-235-252-243.compute-1.amazonaws.com US
18.207.135.73 ec2-18-207-135-73.compute-1.amazonaws.com US
3.239.84.110 ec2-3-239-84-110.compute-1.amazonaws.com US
3.229.124.160 ec2-3-229-124-160.compute-1.amazonaws.com US
208.115.111.243 crawl2.dotnetdotcom.org US
208.115.111.248 crawl7.dotnetdotcom.org US
208.115.111.245 crawl4.dotnetdotcom.org US
208.115.111.246 crawl5.dotnetdotcom.org US
208.115.111.247 crawl6.dotnetdotcom.org US
208.115.111.249 crawl8.dotnetdotcom.org US
208.115.111.250 crawl9.dotnetdotcom.org US
208.115.111.251 crawl10.dotnetdotcom.org US
208.115.111.242 crawl1.dotnetdotcom.org US
208.115.111.244 crawl3.dotnetdotcom.org US
44.197.231.14 ec2-44-197-231-14.compute-1.amazonaws.com US
3.229.134.72 ec2-3-229-134-72.compute-1.amazonaws.com US
3.237.11.62 ec2-3-237-11-62.compute-1.amazonaws.com US
44.192.80.95 ec2-44-192-80-95.compute-1.amazonaws.com US
34.239.164.153 ec2-34-239-164-153.compute-1.amazonaws.com US
18.207.243.221 ec2-18-207-243-221.compute-1.amazonaws.com US
3.234.212.255 ec2-3-234-212-255.compute-1.amazonaws.com US
3.237.36.121 ec2-3-237-36-121.compute-1.amazonaws.com US
34.204.199.192 ec2-34-204-199-192.compute-1.amazonaws.com ?
44.210.121.213 ec2-44-210-121-213.compute-1.amazonaws.com US
44.200.169.65 ec2-44-200-169-65.compute-1.amazonaws.com US
44.213.64.21 ec2-44-213-64-21.compute-1.amazonaws.com US
44.210.134.11 ec2-44-210-134-11.compute-1.amazonaws.com US
44.211.34.138 ec2-44-211-34-138.compute-1.amazonaws.com US
44.210.242.117 ec2-44-210-242-117.compute-1.amazonaws.com US
3.236.27.36 ec2-3-236-27-36.compute-1.amazonaws.com US
54.162.134.154 ec2-54-162-134-154.compute-1.amazonaws.com US
3.215.186.233 ec2-3-215-186-233.compute-1.amazonaws.com US
34.238.190.234 ec2-34-238-190-234.compute-1.amazonaws.com US
3.231.212.0 ec2-3-231-212-0.compute-1.amazonaws.com US
44.200.192.79 ec2-44-200-192-79.compute-1.amazonaws.com US
54.227.97.128 ec2-54-227-97-128.compute-1.amazonaws.com US
54.204.114.191 ec2-54-204-114-191.compute-1.amazonaws.com US
3.227.217.255 ec2-3-227-217-255.compute-1.amazonaws.com US
54.196.246.162 ec2-54-196-246-162.compute-1.amazonaws.com US
18.212.244.63 ec2-18-212-244-63.compute-1.amazonaws.com US
18.212.79.37 ec2-18-212-79-37.compute-1.amazonaws.com US
3.208.10.124 ec2-3-208-10-124.compute-1.amazonaws.com US
54.224.206.204 ec2-54-224-206-204.compute-1.amazonaws.com US
34.229.216.200 ec2-34-229-216-200.compute-1.amazonaws.com US
18.212.99.239 ec2-18-212-99-239.compute-1.amazonaws.com US
18.209.7.69 ec2-18-209-7-69.compute-1.amazonaws.com US
54.234.157.63 ec2-54-234-157-63.compute-1.amazonaws.com US
52.23.170.189 ec2-52-23-170-189.compute-1.amazonaws.com US
54.234.165.70 ec2-54-234-165-70.compute-1.amazonaws.com ?
54.144.219.20 ec2-54-144-219-20.compute-1.amazonaws.com US
54.197.30.103 ec2-54-197-30-103.compute-1.amazonaws.com US
44.210.235.38 ec2-44-210-235-38.compute-1.amazonaws.com ?
44.213.70.16 ec2-44-213-70-16.compute-1.amazonaws.com US
34.201.120.80 ec2-34-201-120-80.compute-1.amazonaws.com US
3.81.136.3 ec2-3-81-136-3.compute-1.amazonaws.com US
34.235.87.203 ec2-34-235-87-203.compute-1.amazonaws.com US
35.173.139.239 ec2-35-173-139-239.compute-1.amazonaws.com US
52.23.215.219 ec2-52-23-215-219.compute-1.amazonaws.com US
18.207.154.18 ec2-18-207-154-18.compute-1.amazonaws.com US
34.226.205.11 ec2-34-226-205-11.compute-1.amazonaws.com US
35.168.113.2 ec2-35-168-113-2.compute-1.amazonaws.com US
3.236.229.54 ec2-3-236-229-54.compute-1.amazonaws.com US
54.204.254.4 ec2-54-204-254-4.compute-1.amazonaws.com US
18.232.80.217 ec2-18-232-80-217.compute-1.amazonaws.com US
54.81.171.117 ec2-54-81-171-117.compute-1.amazonaws.com US
54.224.66.250 ec2-54-224-66-250.compute-1.amazonaws.com US
3.236.239.150 ec2-3-236-239-150.compute-1.amazonaws.com US
3.238.111.188 ec2-3-238-111-188.compute-1.amazonaws.com US
44.223.37.137 ec2-44-223-37-137.compute-1.amazonaws.com US
3.236.147.179 ec2-3-236-147-179.compute-1.amazonaws.com US
3.215.22.58 ec2-3-215-22-58.compute-1.amazonaws.com US
3.214.224.47 ec2-3-214-224-47.compute-1.amazonaws.com US
18.207.134.188 ec2-18-207-134-188.compute-1.amazonaws.com US
34.236.33.47 ec2-34-236-33-47.compute-1.amazonaws.com US
3.239.82.234 ec2-3-239-82-234.compute-1.amazonaws.com US
3.238.120.99 ec2-3-238-120-99.compute-1.amazonaws.com US
34.235.124.204 ec2-34-235-124-204.compute-1.amazonaws.com US
44.221.61.15 ec2-44-221-61-15.compute-1.amazonaws.com US
44.192.50.21 ec2-44-192-50-21.compute-1.amazonaws.com US
18.215.182.53 ec2-18-215-182-53.compute-1.amazonaws.com US
52.90.232.88 ec2-52-90-232-88.compute-1.amazonaws.com US
18.212.202.116 ec2-18-212-202-116.compute-1.amazonaws.com US
3.85.176.168 ec2-3-85-176-168.compute-1.amazonaws.com US
3.92.166.120 ec2-3-92-166-120.compute-1.amazonaws.com US
54.87.219.183 ec2-54-87-219-183.compute-1.amazonaws.com US
44.200.187.12 ec2-44-200-187-12.compute-1.amazonaws.com US
用户代理字符串
Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)
类型
营销
首次出现
2009-05-11 05:50:00
最后出现
2016-12-27 11:15:45
遵循robots.txt
IP地址
10
来源
IP地址 服务器名称 所属国家
208.115.111.243 crawl2.dotnetdotcom.org US
208.115.111.248 crawl7.dotnetdotcom.org US
208.115.111.247 crawl6.dotnetdotcom.org US
208.115.111.249 crawl8.dotnetdotcom.org US
208.115.111.250 crawl9.dotnetdotcom.org US
208.115.111.251 crawl10.dotnetdotcom.org US
208.115.111.242 crawl1.dotnetdotcom.org US
208.115.111.244 crawl3.dotnetdotcom.org US
208.115.111.245 crawl4.dotnetdotcom.org US
208.115.111.246 crawl5.dotnetdotcom.org US
用户代理字符串
dotbot
类型
营销
首次出现
2013-11-09 11:26:07
最后出现
2014-05-17 20:35:05
遵循robots.txt
IP地址
6
来源
IP地址 服务器名称 所属国家
208.184.81.30 ? US
72.44.63.103 ec2-72-44-63-103.compute-1.amazonaws.com US
107.20.88.61 ec2-107-20-88-61.compute-1.amazonaws.com US
184.72.90.45 ec2-184-72-90-45.compute-1.amazonaws.com US
184.73.99.2 ec2-184-73-99-2.compute-1.amazonaws.com US
50.19.146.243 ec2-50-19-146-243.compute-1.amazonaws.com US

在本页面中,您将了解更多关于我们的爬虫Dotbot的信息,它被用来抓取Moz链接索引,为Moz API、链接浏览器和您的活动中的链接部分提供动力。

Moz链接索引抓取器

Dotbot是Moz的网络爬虫,它为Moz链接指数收集网络数据。我们通过Dotbot收集的这些数据可以在您的Moz Pro广告系列的链接部分、链接浏览器和Moz链接API中找到。Dotbot与Rogerbot不同,Rogerbot是我们为Moz Pro广告系列提供的网站审计爬行器。

为什么Moz会抓取网络?

我们的一些工具,如Link Explorer,需要我们抓取网站。当这种情况发生时,用户代理Dotbot被用来识别我们的爬行器。最好记住,你需要一个Moz专业账户来访问收集到的大部分信息。我们的免费网络营销社区成员的访问权限有限。要看我们收集的数据类型的一个例子,在链接浏览器的搜索框中输入一个URL。

如何阻止Dotbot抓取你的网站

如果您不希望Dotbot抓取您的网站,我们总是尊重标准的Robots排除协议(又称robots.txt)。如果您想阻止Dotbot,您只需将我们的user-agent字符串添加到您的robots.txt文件。

禁止Dotbot进入您网站的某些区域

User-agent: dotbot 
Disallow: /admin/ 
Disallow: /scripts/ Disallow: /images/

完全禁止Dotbot进入您网站

User-agent: dotbot
Disallow: /

减缓Dotbot的速度

User-agent: dotbot
Crawl-delay: 10

Dotbot何时能检测网站robots.txt文件的变化?

Dotbot只在新的索引抓取过程中第一次遇到该网站时查看你的robots.txt文件。这意味着,如果Dotbot看到它被允许进入该网站一次,对该权限的任何改变都不会被查看,直到Dotbot下次定位到你的网站的链接。

常见问题

FAQs
查看更多 >