w推测“域名解析过程中，Google crawlers中首先是Googlebo中的Google Web search上阵”。

 +-----+----------------+---------------------+-------------------------+------------------+

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 119.147.32.253 | -- :: | Unidentified User Agent |                  |

 |   | 183.57.53.197  | -- :: | Mozilla 5.0             | iOS              |

 |   | 123.56.233.103 | -- :: | Unidentified User Agent |                  |

 |   | 112.90.142.207 | -- :: | Firefox 3.0             | Windows XP       |

 |   | 183.232.120.37 | -- :: | Firefox 3.0             | Windows XP       |

 |   | 117.136.40.218 | -- :: | ZTE                     | Android          |

 |   | 117.136.40.218 | -- :: | ZTE                     | Android          |

 |   | 117.136.40.218 | -- :: | ZTE                     | Android          |

 |   | 117.136.40.218 | -- :: | ZTE                     | Android          |

 |   | 117.136.40.218 | -- :: | Safari 534.30           | Android          |

 |   | 117.136.40.218 | -- :: | Safari 534.30           | Android          |

 |   | 117.136.40.218 | -- :: | Chrome 37.0.0.0         | Android          |

 |   | 117.136.40.218 | -- :: | Chrome 37.0.0.0         | Android          |

 |   | 117.136.40.218 | -- :: | Chrome 37.0.0.0         | Android          |

 |   | 117.136.40.218 | -- :: | Chrome 37.0.0.0         | Android          |

 |   | 117.136.40.218 | -- :: | Chrome 55.0.2883.87     | Windows         |

 |   | 177.193.53.212 | -- :: | Googlebot               | Unknown Platform |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 139.162.108.53 | -- :: | Chrome 50.0.2661.102    | Windows        |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 61.142.176.19  | -- :: | Firefox 3.6.           | Windows         |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 23.251.63.45   | -- :: | Unidentified User Agent |                  |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 61.142.176.20  | -- :: | Unidentified User Agent | Unknown Platform |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 23.251.63.45   | -- :: | Unidentified User Agent |                  |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |   | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 23.251.63.45   | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 23.251.63.45   | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 125.39.207.33  | -- :: | Unidentified User Agent | Unknown Platform |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 183.60.48.110  | -- :: | Unidentified User Agent | Unknown Platform |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 101.226.51.229 | -- :: | Chrome 45.0.2454.101    | Windows XP       |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 |  | 111.251.93.170 | -- :: | Unidentified User Agent |                  |

 +-----+----------------+---------------------+-------------------------+------------------+

https://support.google.com/webmasters/answer/1061943?hl=en

Google crawlers

See which robots Google uses to crawl the web

"Crawler" is a generic term for any program (such as a robot or spider) used to automatically discover and scan websites by following links from one webpage to another. Google's main crawler is called Googlebot. This table lists information about the common Google crawlers you may see in your referrer logs, and how they should be specified in robots.txt, the robots meta tags, and the X-Robots-Tag HTTP directives.

Crawler User agent token Full user agent string (as seen in website log files)
Googlebot (Google Web search) Googlebot Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
or
(rarely used): Googlebot/2.1 (+http://www.google.com/bot.html)
Googlebot News Googlebot-News
(Googlebot)		 Googlebot-News
Googlebot Images Googlebot-Image
(Googlebot)		 Googlebot-Image/1.0
Googlebot Video Googlebot-Video
(Googlebot)		 Googlebot-Video/1.0
Google Smartphone Googlebot

​Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Google Mobile AdSense Mediapartners-Google

or

Mediapartners
(Googlebot)

 [various mobile device types] (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
Google AdSense Mediapartners-Google
Mediapartners
(Googlebot)		 Mediapartners-Google
Google AdsBot landing page quality check AdsBot-Google AdsBot-Google (+http://www.google.com/adsbot.html)

Google app crawler

(Used to fetch resources for mobile apps, obeys AdsBot-Google robots rules.)

 AdsBot-Google-Mobile-Apps AdsBot-Google-Mobile-Apps

robots.txt

Where several user-agents are recognized in the robots.txt file, Google will follow the most specific. If you want all of Google to be able to crawl your pages, you don't need a robots.txt file at all. If you want to block or allow all of Google's crawlers from accessing some of your content, you can do this by specifying Googlebot as the user-agent. For example, if you want all your pages to appear in Google search, and if you want AdSense ads to appear on your pages, you don't need a robots.txt file. Similarly, if you want to block some pages from Google altogether, blocking the user-agent Googlebot will also block all Google's other user-agents.

But if you want more fine-grained control, you can get more specific. For example, you might want all your pages to appear in Google Search, but you don't want images in your personal directory to be crawled. In this case, use robots.txt to disallow the user-agent Googlebot-image from crawling the files in your /personal directory (while allowing Googlebot to crawl all files), like this:

User-agent: Googlebot

Disallow:

User-agent: Googlebot-Image

Disallow: /personal

To take another example, say that you want ads on all your pages, but you don't want those pages to appear in Google Search. Here, you'd block Googlebot, but allow Mediapartners-Google, like this:

User-agent: Googlebot

Disallow: /

User-agent: Mediapartners-Google

Disallow:

robots meta tag

Some pages use multiple robots meta tags to specify directives for different crawlers, like this:

<meta name="robots" content="nofollow"><meta name="googlebot" content="noindex">

In this case, Google will use the sum of the negative directives, and Googlebot will follow both the noindex and nofollow directives. More detailed information about controlling how Google crawls and indexes your site.

Googlebot (Google Web search)的更多相关文章

  1. Google Web Designer – 创建引人入胜的 HTML5 网站

    Google Web Designer 可以帮助你创建引人入胜,互动的基于 HTML5 的设计和动画,可以在任何设备上运行.如果你喜欢自己动手,设计背后的所有的代码都是可以手工编辑的. 虽然可视化工具 ...

  2. Angular JS | Closure | Google Web Toolkit | Dart | Polymer 概要汇集

    AngularJS | Closure | Google Web Toolkit | Dart | Polymer GWT https://code.google.com/p/google-web-t ...

  3. Google Web Toolkit (GWT)怎么制作多个用户界面

    Google Web Toolkit即GWT是目前基于AJAX技术开发的一个比较成功的框架包,但是其附带例程中只有单页面的实例,那么应该怎么样制作多个页面呢? 其实很简单,GWT的一个模块,就是一个页 ...

  4. GWT(Google Web Tookit) Eclipse Plugin的zip下载地址（同时提供GWT Designer下载地址）

    按照Eclipse Help->Install new software->....(这里是官方安装文档:http://code.google.com/intl/zh-CN/eclipse ...

  5. Mac效率：配置Alfred web search

    // 这是一篇导入进来的旧博客,可能有时效性问题. 想用搜索引擎搜东西,或者查字典时,一般的workflow是:打开浏览器-->打开搜索引擎/字典网站-->输入搜索关键字-->回车. ...

  6. Google Web Designer打开白屏问题的解决方案

    Google Web Designer是谷歌出品的一个可视化的  HTML5  网页和广告的设计开发工具  Google Web Designer . 官网地址:https://www.google. ...

  7. google web design html5制作工具

    Google 推出 Web Designer,帮助你做 HTML 5 设计的免费本地应用,支持 Windows 和 OS X 2013年10月1日        感谢读者 SamRaper 的提醒. ...

  8. 通过Google Custom Search API 进行站内搜索

    今天突然想把博客的搜索改为google的站内搜索,印象中google adsense中好像提高这个站内搜索的代码,但苦逼的是google adsense帐号一直审核不通过,所以只能通过google c ...

  9. AdMob设计工具google web designer

    一.google web designer工具中文文档: https://support.google.com/webdesigner?hl=zh-Hans#topic=3227692 我用的版本:应 ...

随机推荐

  1. 李洪强iOS开发之Block和协议

    李洪强iOS开发之Block和协议 OC语言BLOCK和协议 一.BOLCK (一)简介 BLOCK是什么?苹果推荐的类型,效率高,在运行中保存代码.用来封装和保存代码,有点像函数,BLOCK可以在任 ...

  2. C++常函数

    常函数即在类的成员函数参数列表后放置const的函数,常函数的作用是限制函数体对成员变量的修改,此外,常函数也不能调用非 常函数. #include <iostream> using na ...

  3. poj2082单调栈

    本来实在做后缀数组的题目的,不巧,碰到了pku3415这题,需要用到单调栈来维护,但是之前又没有学习过单调栈这方面的知识,于是水了几题....... 题意:给你一些连续的小矩形,高宽不定,求最大的矩形 ...

  4. jsp error-page没有生效

    1.首页检查web.xml中的配置,确保路径是正确的 <error-page> <error-code>404</error-code> <location& ...

  5. 一款基于jQuery和HTML5全屏焦点图

    今天爱编程小编给大家分享一款非常绚丽的jQuery焦点图插件,同时这款焦点图也利用了HTML5和CSS3的相关特性,使图片切换效果更加丰富多彩.另外,这款jQuery焦点图插件的特点是全屏的效果,因此 ...

  6. JavaScript匿名类整理学习笔记

    以下为总结在开源的JavaScript框架中能看到很多这样语法结构(function(){})()比如我最近看的jQuery,及chediter.刚开始的时候我看到这样的结果有点奇怪,它是怎么执行的, ...

  7. HBase学习系列

    转自:http://www.aboutyun.com/thread-8391-1-1.html 问题导读: 1.hbase是什么? 2.hbase原理是什么? 3.hbase使用中会遇到什么问题? 4 ...

  8. 【BZOJ】2555: SubString（后缀自动机）

    http://www.lydsy.com/JudgeOnline/problem.php?id=2555 学到了如何快速维护right值orz (不过这仍然是暴力维护,可以卡到O(n) 首先我们在加一 ...

  9. 【BZOJ】1674: [Usaco2005]Part Acquisition（spfa）

    http://www.lydsy.com/JudgeOnline/problem.php?id=1674 想法很简单...将每一种看做一个点,如果i可以换成j,那么连边到j.. 费用都为1.. 然后拥 ...

  10. Java去除所有非中文字符串

    "fdsfjasd阿斯顿飞机阿斯蒂芬,,,,,,,,....".replaceAll("[^\u4E00-\u9FA5]", "");