# www.robotstxt.org/ # Allow crawling of all content except of some directories User-agent: * Disallow: /bower_components/ Disallow: /downloads/ Disallow: /js/ Disallow: /styles/