New robots.txt tool
The Google Sitemaps team has just introduced a new robots.txt tool into their Sitemaps. Do check in for some robots.txt tutorial and even a robots.txt validator, to make your robot file work the best way. Google Sitemaps new tool for robots.txt will also be a great help for all this.