The robots.txt gives instructions to search engine crawler about a site. You would use it to hide files or directories that you don't wish the search engine crawlers to find.
What does a robots.txt file look like?
The robots.txt file is a text file and will look something like this:
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /admin/
By telling a search engine spider what not to crawl you will save valuable bandwidth. When a search engine crawler arrives at your website it will read the robots.txt file to see which pages to index and which to ignore.
Place the robots.txt file in the root directory.
For more on creating a robots.txt visit robotstxt.org.
Common crawler traps
Search engine crawlers can have difficulty in crawling pages with:
- Input forms
- Session ID's in the URL
- Frames
- Login pages
- Pages restricted by cookies
How does this help SEO?
A well written robots.txt file helps improve your websites search engine ranking by providing important information to the search engine crawler.
By controlling what the search engine crawlers view you can also remove duplicate content.
With a proper robots.txt file you can help the search engines to give you a higher page ranking.
Search Engine Optimisation
seo, robots.txt