Robots Config
Optional
You can change the contents of the robots.txt file used by crawlers using the robots_txt option. The default is shown below.
robots_txt: >-
User-agent: *
Optional
You can change the contents of the robots.txt file used by crawlers using the robots_txt option. The default is shown below.
robots_txt: >-
User-agent: *