Languages Features Creators Calendar CSV Resources Blog About Pricing Add Language
GitHub icon

Robots.txt

Robots.txt - Config format

< >

Robots.txt is a config format created in 1994 by Martijn Koster.

#1915on PLDB 29Years Old 0Books
0Papers

A robots.txt file tells search engine crawlers which URLs the crawler can access on your site.


Example from the web:
User-agent: googlebot # all Google services Disallow: /private/ # disallow this directory User-agent: googlebot-news # only the news service Disallow: / # disallow everything User-agent: * # any robot Disallow: /something/ # disallow this directory

Language features

Feature Supported Token Example
Comments
# A comment
Line Comments #
# A comment
fishlang.html · robots.txt.html · whirl.html

View source

- Build the next great programming language · Search · v2023 · Day 205 · Docs · Acknowledgements · Traffic · Traffic Today · Mirrors · GitHub · feedback@pldb.com