What is robots.txt?

Robots.txt is a text file located in the root of a website and is used to instruct search engine robots (also known as crawlers or bots) on how to navigate and index the content of the website.