Robots.txt Generator

Default - All Robots are:	Allowed Refused
Crawl-Delay:	Default - No Delay 5 Seconds 10 Seconds 20 Seconds 60 seconds 120 Seconds
Sitemap: (leave blank if you don't have)
Search Robots:	Google	Same as Default Allowed Refused
	Google Image	Same as Default Allowed Refused
	Google Mobile	Same as Default Allowed Refused
	MSN Search	Same as Default Allowed Refused
	Yahoo	Same as Default Allowed Refused
	Yahoo MM	Same as Default Allowed Refused
	Yahoo Blogs	Same as Default Allowed Refused
	Ask/Teoma	Same as Default Allowed Refused
	GigaBlast	Same as Default Allowed Refused
	DMOZ Checker	Same as Default Allowed Refused
	Nutch	Same as Default Allowed Refused
	Alexa/Wayback	Same as Default Allowed Refused
	Baidu	Same as Default Allowed Refused
	Naver	Same as Default Allowed Refused
	MSN PicSearch	Same as Default Allowed Refused
Restricted Directories:	The path is relative to root and must contain a trailing slash "/"

Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.

About Robots.txt Generator

Robert.  Text generator tool

 

 

You can easily create a new or edit an existing robots.txt file for your site with a robots.txt generator. To upload an existing file and pre-populate the, type or paste the root domain URL in the top text box and click Upload. Use the robots.txt generator tool to create directives with either Allow or Disallow directives (Allow is default, click to change) for User Agents (use * for all or click to select just one) for specified content on your site. Click Add directive to add the new directive to the list. To edit an existing directive, click Remove directive, and then create a new one. 

Create custom user agent directives 

In our robots.txt generator Google and several other search engines can be specified within your criteria. To specify alternative directives for one crawler, click the User Agent list box (showing * by default) to select the bot. When you click Add directive, the custom section is added to the list with all of the generic directives included with the new custom directive. To change a generic Disallow directive into an Allow directive for the custom user agent, create a new Allow directive for the specific user agent for the content. The matching Disallow directive is removed for the custom user agent. 

To learn more about robots.txt directives, see The Ultimate Guide to Blocking Your Content in Search. 

You can also add a link to your XML-based Sitemap file. Type or paste the full URL for the XML Sitemap file in the XML Sitemap text box. Click Update to add this command to the robots.txt file list. 

When done, click Export to save your new robots.txt file. Use FTP to upload the file to the domain root of your site. With this uploaded file from our robots.txt generator Google or other specified sites will know which pages or directories of your site should not show up in user searches.