To remove your own content (pages, sites, images, and more) from Google's indexing. Use the URL removal tool in Google's Webmaster Tools. To prevent search engines from indexing or crawling content on your site, use one of the following:
A robots.txt file - It's simply a text file that will restrict access to your site. Search Engines won't crawl or index the page. Note: If Google follows a link from another web then it will index the page. You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file (not even an empty one).
The simplest robots.txt file uses two rules:
User-agent: the robot the following rule applies to
A noindex meta tag. Put this tag in the header of the page. Google will drop it from there search results. This method allows you to control access on a page by page basis.
