### Rules for all crawlers ### # Disallow indexing guest books, since there may be arbitrary content there, # and also because they are penalized by Google (PR=0), dragging down other pages. User-agent: * Disallow: /guestbook.jsp Disallow: /guestbookps.jsp Disallow: /choice.jsp ### Banned crawlers ### User-agent: abot Disallow: / User-agent: aipbot Disallow: / User-agent: appie Disallow: / User-agent: AtlocalBot Disallow: / User-agent: BecomeBot Disallow: / User-agent: BigBrother/1.7a Disallow: / User-agent: cfetch Disallow: / User-agent: ConveraCrawler Disallow: / User-agent: e-SocietyRobot Disallow: / User-agent: FunWebProd Disallow: / User-agent: Gaisbot Disallow: / User-agent: Java Disallow: / User-agent: ichiro Disallow: / User-agent: ia_archiver Disallow: / User-agent: IRLbot Disallow: / User-agent: LJM Systems Disallow: / User-agent: MJ12bot Disallow: / User-agent: NP/0.1 Disallow: / User-agent: POE-Component-Client-HTTP Disallow: / User-agent: psbot Disallow: / User-agent: Ocelli Disallow: / User-agent: ShopWiki/1.0 Disallow: / User-agent: Teoma Disallow: / User-agent: updated Disallow: / User-agent: WebCopier Disallow: / User-agent: ZyBorg Disallow: /