Crawling

Crawling
« Back to Glossary Index

Crawling is the process of discovery during which search engines send out a team of bots (also known as crawlers or spiders) intending to find new and updated content.

What this content is can vary; for example, this could be a webpage, an image, a video, a PDF, etc. — but either way, content is discovered by internal links. Without these links, bots cannot locate and crawl the content. In essence, crawling is a data discovery process.

Once content has been crawled and discovered, it is added to the queue to be indexed. However, crawling is not a guarantee of indexing; if there are quality issues or other types of blockers, the content will not be indexed.

What is Crawl Budget?

Another term you may encounter is crawl budget. The crawl budget refers to the number of pages Googlebot can crawl and index on a website within a given timeframe.

Most websites don’t need to worry about crawl budget, as Google is excellent at finding and indexing pages.

Still, if, for any reason, the number of pages exceeds your website’s crawl budget, then you are likely to run into issues with pages on your site that aren’t indexed. If this happens, it may be an idea to look at your crawl errors.

Synonyms:
crawl, crawled
« Back to Glossary Index
Facebook
Twitter
LinkedIn

Subscribe To Tech SEO Tips Newsletter

The latest news from the SEO industry, plus tips and discussions on improving your tech SEO performance.