Creates an effective robots.txt file that helps Google and other search engines crawl and index your site according to your instruction.
Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.