How do search engines work?

A search engine is a system that makes an enormous collection of web pages searchable and picks the most useful results for every query. Understanding that process explains why SEO works. Broadly, a search engine goes through three steps: crawling, indexing and ranking.

Step 1: crawling

Search engines use programs called crawlers or spiders to explore the web. Google's crawler is called Googlebot. It starts from known pages, follows the links on them and discovers new and updated pages. An XML sitemap helps, as does a good internal linking structure. A robots.txt file lets you indicate which parts of your site don't need to be crawled.

Worth knowing: a search engine has limited time and resources per website, known as crawl budget. Sites with many unnecessary or duplicate pages, or with content that only appears after a lot of JavaScript has run, make the crawler's job harder.

Step 2: indexing

After fetching a page, the search engine tries to understand it. What is the text about, which images does it contain, how does it relate to other pages? The system also checks whether a page duplicates another; if so, it usually picks one version as the canonical. Whatever is worth keeping goes into the index: a huge database from which search results are later assembled. Not every crawled page gets indexed.

Step 3: ranking

When someone searches, the engine looks up relevant pages in the index and puts them in order. Hundreds of signals play a role, including:

• how well the content matches the query and the intent behind it;
• the quality and trustworthiness of the page and the website;
• links from other websites;
• usability, such as loading speed and mobile display;
• the searcher's context, such as language and location.

Search engines and people

However clever search engines become, everything ultimately revolves around the person behind the search. Search engines are getting better at reading intent: does someone want to learn something, compare options or buy? The page that serves that intent best has the best chance. Optimising ‘for Google’ and optimising for your visitor are increasingly the same thing.

How do AI search engines work?

AI search engines add a step. They retrieve relevant pages, often via a regular search index, and let a language model turn them into a summarised answer, sometimes with source links. The foundation stays the same: pages that are easy to find and easy to read have the best chance of appearing in an AI answer. Read more in What is GEO?