# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file #User-agent: Slurp User-agent: * Crawl-delay: 10 Disallow: /posts Disallow: /questions Disallow: /profiles Disallow: /facebook