A spider is a piece of software that browses the Internet in order to find and catalog webpages for search engines. Spiders are used by all of the big search engines, including Google, to create and maintain their databases. These programs navigate the Web continuously, clicking on links one after the other.

A crawler might find 20 connections on the main page of a website, for instance. Each connection will be followed by the crawler, which will then add every website it discovers to the search engine's database. Of course, the newly discovered sites might also contain connections, which the crawler keeps following. Internal links are those that connect to sections on the same website, while exterior links go to other websites (external links). The crawler will visit new websites as a result of the exterior connections, collecting even more pages.

Spiders frequently visit previously scanned websites because of the interconnected structure of website connections. This enables search algorithms to keep account of the number of outside sites that connect to a given website. Typically, a website will appear better in search engine results the more inbound connections it has. Spiders help search engine databases remain current by not only locating new sites and monitoring connections, but also by monitoring changes to individual pages.

For those who dislike arachnids, the names spiders go by other than spiders are robots and crawlers. In sentences like "That search engine eventually spidered my website last week," the term "spider" can also be a verb.