Generate a robots.txt file for free with this easy-to-use tool. Make sure search engine crawlers are crawling & indexing your site.
A file called robots.txt can be placed in the root folder of your website to help search engines index it more accurately. Website crawlers, or robots, are used by search engines like Google to review all of the material on your website. You may not want certain areas of your website crawled and included in user search results, such as the admin page. These pages can be expressly ignored by adding them to the file. The Robots Exclusion Protocol is used in Robots.txt files. With the inputs of pages to be omitted, this website will easily build the file for you.