Robots Config
Optional
You can change the contents of the robots.txt
file used by crawlers using the robots_txt
option. The default is shown below.
robots_txt: >-
User-agent: *
Optional
You can change the contents of the robots.txt
file used by crawlers using the robots_txt
option. The default is shown below.
robots_txt: >-
User-agent: *