Spider

Search engine crawlers which search or “spider” the web for pages to include in the index.

Many non-traditional search companies have different spiders which perform other applications. For example, TurnItInBot searches for plagiarism. Spiders should obey the robots.txt protocol.

SHARE
Previous articleSERP
Next articleTaxonomy