A robots.txt file is a text file that notifies search engine crawlers not to explore certain areas of your website OR
Robots.txt is a text file that allows a website to provide instructions to web crawling bots.
It is located at the root of your website. It’s also known as the Robots Exclusion Protocol because it stops search engines from indexing some irrelevant and/or specific content.
When a search engine bot is about to crawl a URL on your website, it will look for your robots.txt file first. Web crawlers, often known as web robots, are used by search engines like Google to store and categorize websites. Before reading any other file from the website, most bots are configured to look for a robots.txt file on the server. It performs this to discover if the owner of a website has any particular instructions for crawling and indexing their site.
When a search engine crawler arrives at a site, the first thing it does is search for a robots.txt file. If there isn’t one, it’ll continue crawling the rest of the site as usual. If the crawler finds the file, it will search it for any commands before moving on.
In a robots.txt file, there are four frequent commands:
Disallow: It prevents search engine crawlers from examining and indexing specified site files. This can help you avoid duplicate content, staging regions, and other secret files from showing up in search results.
Allow: It allows access to subfolders while disallowing access to parent folders.
Crawl-delay: It tells crawlers how long they should wait before loading a file.
Sitemap: Any sitemaps linked with your website are referenced by the term.
To make their commands obvious, robots.txt files are always formatted in the same way:
User agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpEach subdomain of a website must have its own robots.txt file. It’s vital to know that a robots.txt file isn’t respected by all bots. Some malicious bots will examine the robots.txt file to figure out which files and folders to attack first. Furthermore, even if a robots.txt file advises bots to ignore specific pages on a website, such pages may still appear in search results if they are connected to other crawling pages.