Dissection of Google indexing services

Question

There are more than a few questions that hop into mind when someone thinks about Google's indexing services. Jeff Atwood wrote about them at The Elephant in the Room: Google Monoculture and Trouble In the House of Google.

I have two questions:

• How does google index dynamic websites?

  This site has dynamic pages, QUESTIONS, TAGS, USERS, BADGES, UNANSWERED, ASK QUESTION. The content of these pages is dynamically generated, therefore we access the dynamic content and not the physical files on the server. But how does Google shows every question of the site or other dynamic websites?

• What does Google index and keep on its servers? Does it copy the complete page into its server or just the title, meta tags and body?

Answer 1

How does google index dynamic websites?

"Crawler" is a generic term for any program (such as a robot or spider) used to automatically discover and scan websites by following links from one webpage to another. Google's main crawler is called Googlebot. Check out this list of Google Crawlers.

Today, most crawlers cannot index the entire dynamic page, but they will index most of the content on all the pages, and that is exactly the end result we are seeking.

After a page is crawled, the next step is to index its content. The indexed page is stored in a giant database, from where it can later be retrieved.

Essentially, the process of indexing is identifying the words and expressions that best describe the page and assigning particular keywords to the page. For a human it will not be possible to process such amounts of information but generally search engines deal just fine with this task.

Sometimes they might not get the meaning of a page right but if you help them by optimizing it, it will be easier for them to classify your pages correctly and for you to get higher rankings.

When a search request comes, the search engine processes it – i.e. it compares the search string in the search request with the indexed pages in the database. Since it is likely that more than one page (practically it is millions of pages) contains the search string, the search engine starts calculating the relevancy of each of the pages in its index with the search string.

For a list of ranking factors you can see Search Engine Ranking Factors and the googlewebmastercentral blog. A good article regarding the indexing issue is "How to optimize a dynamic web site" that will clear you doubts regarding the indexing of dynamic websites.

What does Google index and keep on its servers?

From the Wikipedia article on Search Engine Index:

Search engine indexing collects, parses, and stores data to facilitate fast and accurate information retrieval. Index design incorporates interdisciplinary concepts from linguistics, cognitive psychology, mathematics, informatics, physics, and computer science. An alternate name for the process in the context of search engines designed to find web pages on the Internet is Web indexing.

Answer 2

How does google index dynamic websites?

Google, and every other search engine, use internet bots called web crawlers, to index websites. From the Wikipedia article:

A Web crawler is a computer program that browses the World Wide Web in a methodical, automated manner or in an orderly fashion. Other terms for Web crawlers are ants, automatic indexers, bots, Web spiders, Web robots, or—especially in the FOAF community—Web scutters.

Essentially, and without going into much detail, the web crawler visits this and every other website periodically and "sees" exactly the same as you see when you visit a site. It makes no difference if it's a static or a dynamic website, the crawler "sees" the rendered output as you see it in your browser.

The crawler visits high traffic sites many times per day and stores the indexing information each time, giving the impression that it has access to live data. There are a lot of open source web crawlers you can examine to get a more technical grasp of the concepts involved. Google's own web crawler is called Googlebot and there's an official faq page for it.

What does Google index and keep on its servers?

That kind of information is not fully publicly available. We can assume that they store and analyse everything that would make sense search-wise. There are a lot of talks and presentations on Google internals at various conferences but no definitive source on exactly what kind of information they store.

As to how the information is stored and analyzed, there is a good description of the Google index on the Wikipedia article on the Google platform:

Like most search engines, Google indexes documents by building a data structure known as inverted index. Such an index allows obtaining a list of documents by a query word. The index is very large due to the number of documents stored in the servers.

The index is partitioned by document IDs into many pieces called shards. Each shard is replicated onto multiple servers. Initially, the index was being served from hard disk drives, like it's done in traditional information retrieval (IR) systems.

And they don't just index website content, but also several types of binary files, anything that can be found on the web really.