User-agent: Scrapy
Allow: /

User-Agent: Googlebot
Allow: .js
Allow: .css
Allow: .jpg