The number of web pages combined with the fluidity of these pages make it impossible to index the web through the traditional methods of human-mediated cataloging. The first attempts to create automatic web indices were based on tabulations of all words contained with every page. Scholars can then locate pages by searching for those that contain a specific combination of words -- most commonly sorted by those with the highest frequency of occurence of the target words.

This method has many flaws:

• It is very possible to write at length on a topic and happen not to use the generally-accepted key words for that area.
• Several authors have learned to "cheat", by deliberately over-utilizing the expected key words in order to assure that their pages will appear at the top of most automated searches.
• This approach has no ability to evaluate the accuracy or quality of the information within the page.

Are other search strategies possible?

The IBM Clever project is developing a new search criterion based on web page links instead of page contents. They particularly value "authorities" -- the pages that contain the largest number of inward links from other pages.

This approach views each author's decision to create a hyperlink as an implicit endorsement of the other site's content. Thus, the pages with the most inward links are the pages considered the most authoritative sites devoted to that topic.

For more information, see:

1. Scientific American, 6/99, (html)
2. Clever Project home page at IBM Almaden Research Center

Another startup company, Why.com, enlists surfers to rate the quality of each page returned by the searches. It then uses this information to refine future searches.

For more information see:

1. Summary description from Technology Review, May, 2000.