User-agent: * Disallow: /admin/ Disallow: /cgi-admin/ Disallow: /buy/ Disallow: /mobile/ User-agent: dotbot Disallow: / User-agent: Googlebot Disallow: /*.gz$ Disallow: /*.wmv$ Disallow: /*? Disallow: /admin/ Disallow: /cgi-admin/ Disallow: /buy/ Disallow: /mobile/ sitemap: https://www.booklender.com/sitemap_index.xml User-agent: JikeSpider Disallow: / User-agent: Yandex Disallow: / User-agent: MJ12bot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: PetalBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: barkrowler Disallow: /wp-admin/ User-agent: BLEXBot Disallow: /