[SEO] How to Validate your Robots.txt File
Robots.txt files are used by web owners and webmasters to signify to spider-bots and search engines not to crawl or index certain files and directories. The Robots.txt file is a command file that most creditable search-bots will obey and check for on entry to your website URL.
This robots.txt validation will analyze the syntax of your robots.txt file to see if its format is valid. If there are any errors in your robots.txt file they will be marked in red to point out which sections of your robots.txt file need attention.
How to check your robots.txt file?
To check your robots.txt file, simply put the full URL. Example: http://www.yoursite.com/robots.txt
What will the robots.txt validator look for?
The robots.txt validation will examine your file and look for syntax errors, mistyped words, improper use of the asterisk * and the disallow line. It will mark any errors found in red to indicate what sections need correction.
