Effective use’s of robots.txt

A "robots.txt" file tells search engines whether they can access and therefore crawl parts of our site. This file, which must be named "robots.txt", is placed in the root directory of our site, http://www.abc.com/robots.txt .
The address of our robots.txt file is-
User-agent :*
Disallow : /images/
Disallow : /search/

All compliant search engine bots (denoted by the wildcard * symbol) shouldn't access and crawl the content under /images/ or any URL whose path begins with /search. We may not want certain pages of our site crawled because they might not be useful to users if found in a search engine's search results. If we do want to prevent search engines from crawling our pages, Google Webmaster Tools has a friendly robots.txt generator to help us create this file.
Note that if our site uses subdomains and we wish to have certain pages not crawled on a particular subdomain, we'll have to create a separate robots.txt file for that subdomain.
There are a handful of other ways to prevent content appearing in search results, such as adding "NOINDEX" to your robots meta tag.

Good practices for robots.txt
• Use more secure methods for sensitive content - we shouldn't feel comfortable using robots.txt to block sensitive
or confidential material.
The reason is that search engines could still reference the URLs we block (showing just the URL, no title or snippet). Also, non-compliant or rogue search engines that don't acknowledge the Robots Exclusion Standard could disobey the instructions of your robots.txt.

Avoid:
• allowing search result-like pages to be crawled (users dislike leaving one search result page and landing on another search result page that doesn't add significant value for them)
• allowing a large number of auto-generated pages with the same or only slightly different content to be crawled: "Should these 100,000 near-duplicate pages really be in a search engine's index?"
• allowing URLs created as a result of proxy services to be crawled.

SEO delhi , Reseller SEO