A bot is a computer program that interacts with websites and applications automatically. Robots is a file on a website that tells search engine crawlers, which portions of the site should not be accessed by a bot program. These bots "crawl" web pages and index the content so that it can be found in search engine results. A robots.txt file assists web crawlers in organizing their activities so that they do not overload the webserver or index pages which are not required to show in search results.
Robots.txt is a plaintext file that contains web crawler-specific instructions and language. It is typically followed by all search engines, even though it is not formally standard.
The "language" of these files is believed to be the Robots txt syntax. Any plain text editor may be used to create a new robots text file. A robots.txt file will typically contain the following five terms:
User-agent - The unique bot on each website to which you are sending crawling instructions (usually a search engine).
Disallow - This command instructs a user-agent not to crawl a certain URL. For each URL, just one "Disallow:" line is permitted.
Allow - (only applies to the Google bot or Googlebot): This directive tells the Google bot that it can enter a page or sub-folder even if its main page or sub-folder is not permitted.
Crawl-delay - Specifies the number of seconds a bot should wait before loading and crawling the page's content. (Googlebot does not understand this directive; however, the crawl frequency may be set in the Google Search Console.)
Sitemap - It is used to specify the location of the XML sitemap that is related to this URL.