Robots.txt

27

A file which sits in the root of a site and tells search engines which files not to crawl. Some search engines will still list your URLs as URL only listings even if you block them using a robots.txt file.

Do not put files on a public server if you do not want search engines to index them!

See also:

SHARE
Previous articleRelative Link
Next articleSocial News

1 COMMENT

What say you?