Robots, spiders and your website

Phil Bradley, Independant Internet Consultant, 5 Walton Gardens, Feltham, Middlesex.     philb@philb.com

Ariadne 2001
(ReLIS:doi:doiari:y:2001:i:27:p:9)

Abstract:

This issue I thought I’d take a look at a subject which is of absolute importance to those of us who use search engines, but is something we know virtually nothing about, and that is how do web pages end up in search engine directories? If you’re short on time, the quick summary is that search engines of the free text variety (rather than the Index/Directory type) employ specialised utilities which visit a site, copy the information they find back to base, and then include this information the next time that they update their index for users. OK, that’s it. You can move onto the next article now.

Note: Phil Bradley looks at the effect these have on the vital statistics of your web site
Month: 23-March-2001
Year: 2001

 (go top)