What is a search engine spider?

The search engine spider is also known as a bot or crawler. It is a program that follows web links and crawls web pages on behalf of search engines. Spiders process and index the content of a web page based on the search engine algorithm and store the pages in the search engine's database. Googlebot is the name of Google’s spider program.

Having many crawlable (or available for the spider) web pages is the first step to a successful search engine marketing program.