User-agent: dotbot Crawl-delay: 10 User-agent: * Disallow: /search Allow: / Sitemap: /sitemap.xml