ROBOTS.TXT屏蔽笔记、代码、示例大全

自己网站的ROBOTS.TXT屏蔽的记录，以及一些代码和示例：

屏蔽后台目录，为了安全，做双层管理后台目录/a/xxxx/，蜘蛛屏蔽/a/，既不透露后台路径，也屏蔽蜘蛛爬后台目录

缓存，阻止蜘蛛爬静态缓存文件

下载，阻止蜘蛛爬下载目录，若无用，删除下载目录

编辑器，阻止蜘蛛爬编辑器，也防止编辑器目录被发现产生安全隐患

邮件，阻止蜘蛛爬静态邮件模板

其他页面，无收录价值页面屏蔽

图片，阻止蜘蛛爬除JPG/jpg类文件之外的任何类型图片

核心文件目录，阻止蜘蛛直接爬include及其子目录（函数/类库/模型/模板等）

媒体目录，阻止爬播放类型媒体目录，若无用，删除该目录

附加参数页面，阻止蜘蛛爬带参数的页面

RAR ZIP GZ文件类型

无效蜘蛛、恶意蜘蛛屏蔽

指定sitemap.xml位置

目录屏蔽：

User-agent: *

Disallow: /a/

Disallow: /cache/

Disallow: /download/

Disallow: /editors/

Disallow: /email/

Disallow: /extras/

Disallow: /images/

Disallow: /includes/

Disallow: /media/

Disallow: /pub/

Disallow: /nddbc.html

Disallow: /page_not_found.php

Disallow: /login.html

Disallow: /privacy.html

Disallow: /conditions.html

Disallow: /contact_us.html

Disallow: /gv_faq.html

Disallow: /discount_coupon.html

Disallow: /unsubscribe.html

Disallow: /shopping_cart.html

Disallow: /ask_a_question.html

Disallow: /popup_image_additional.html

Disallow: /product_reviews_write.html

Disallow: /tell_a_friend.html

Disallow: /pages-popup_image.html

Disallow: /popup_image_additional.html

Disallow: /login.html

阻止蜘蛛爬非jpg图片（限制产品图片格式为jpg）

User-agent: Googlebot

Allow: .jpg$

Disallow: .jpeg$

Disallow: .gif$

Disallow: .png$

Disallow: .bmp$

阻止蜘蛛爬压缩文件

User-agent: *

Disallow: .zip$

Disallow: .rar$

Disallow: .gz$

Disallow: .tar $

制定sitemap地址

Sitemap: http://www.xxx.jp/sitemap.xml

其他无效蜘蛛、恶意蜘蛛屏蔽：

User-Agent: almaden

Disallow: /

User-Agent: ASPSeek

Disallow: /

User-Agent: Axmo

Disallow: /

User-Agent: BaiduSpider

Disallow: /

User-Agent: booch

Disallow: /

User-Agent: DTS Agent

Disallow: /

User-Agent: Downloader

Disallow: /

User-Agent: EmailCollector

Disallow: /

User-Agent: EmailSiphon

Disallow: /

User-Agent: EmailWolf

Disallow: /

User-Agent: Expired Domain Sleuth

Disallow: /

User-Agent: Franklin Locator

Disallow: /

User-Agent: Gaisbot

Disallow: /

User-Agent: grub

Disallow: /

User-Agent: HughCrawler

Disallow: /

User-Agent: iaea.org

Disallow: /

User-Agent: lcabotAccept

Disallow: /

User-Agent: IconSurf

Disallow: /

User-Agent: Iltrovatore-Setaccio

Disallow: /

User-Agent: Indy Library

Disallow: /

User-Agent: IUPUI

Disallow: /

User-Agent: Kittiecentral

Disallow: /

User-Agent: iaea.org

Disallow: /

User-Agent: larbin

Disallow: /

User-Agent: lwp-trivial

Disallow: /

User-Agent: MetaTagRobot

Disallow: /

User-Agent: Missigua Locator

Disallow: /

User-Agent: NetResearchServer

Disallow: /

User-Agent: NextGenSearch

Disallow: /

User-Agent: NPbot

Disallow: /

User-Agent: Nutch

Disallow: /

User-Agent: ObjectsSearch

Disallow: /

User-Agent: Oracle Ultra Search

Disallow: /

User-Agent: PEERbot

Disallow: /

User-Agent: PictureOfInternet

Disallow: /

User-Agent: PlantyNet

Disallow: /

User-Agent: QuepasaCreep

Disallow: /

User-Agent: ScSpider

Disallow: /

User-Agent: SOFT411

Disallow: /

User-Agent: spider.acont.de

Disallow: /

User-Agent: Sqworm

Disallow: /

User-Agent: SSM Agent

Disallow: /

User-Agent: TAMU

Disallow: /

User-Agent: TheUsefulbot

Disallow: /

User-Agent: TurnitinBot

Disallow: /

User-Agent: Tutorial Crawler

Disallow: /

User-Agent: TutorGig

Disallow: /

User-Agent: WebCopier

Disallow: /

User-Agent: WebZIP

Disallow: /

User-Agent: ZipppBot

Disallow: /

User-Agent: Xenu

Disallow: /

User-Agent: Wotbox

Disallow: /

User-Agent: Wget

Disallow: /

User-Agent: NaverBot

Disallow: /

User-Agent: mozDex

Disallow: /

User-Agent: Sosospider

Disallow: /

User-Agent: Baidupider

Disallow: /