# All robots will spider the domain User-agent: * Disallow: # Disallow directory admin User-agent: * Disallow: admin # Disallow directory files User-agent: * Disallow: files