I just read the first half of John Battelles’s The Search. The book covers the history of search engines, culminating with our current Google experience.
One of the interesting things Battle discussed was Google Zeitgeist (which has since been replaced with Hottrends). This tool ranks popular search terms everyday. This tool goes way beyond the simple tracking of search terms, it tracks the collective thoughts of the Internet. It shows what is on people’s minds and what information they want to learn about. It shows that Paris Hilton has peaked; it shows that people are worried about terrorist attacks; it shows people want the hoverboards in Back to the Future 2. Ok, so I made those up, but conceptually it is true. Hottrends has sumed up the conscious and feelings of the Internet population each day since its inception. This is an incredible marketing tool. Movie producers should go on this site everyday to see what topic would create the next blockbuster. Reporters should check it each day to write articles on topics people want to read. Historians should view it as the exact feelings and state of the culture that day. This tool is amazing.
The rest of the first 150 pages gives a history of searching on the internet. In the late 1990s, companies though portals were the thing of the future. Searches were not given much thought and they worked “well-enough.” Yahoo and AOL were riding high on their success. Searches still needed a lot of refinement.
Up until that point in time, searches were typically used in offline databases. Defining the parameters for these searches was rather easy compared to an internet search. In the offline search, the database was typically used for a well defined purpose and all possible combinations of searches could be programmed into the database. There are only so many ways a person can ask for the number of books still in stock in a database. The database won’t be used for anything elseOn the Internet, this is very different. There are no defined parameters for an Internet search. A person can ask for anything. The problem was figuring out the proper algorithm to bring up the most relevant results.
Remembering search engines of the past (I used hotbot) bringing back poor results. Typing in what you were after didn’t typically provide useful results. Site owners used metatags to aid search engines in finding their sites, but they had to use the same terms you did to bring up their page in a search engine. Thinking of alternate search terms became an art. Shortly afterward, spammers got wise and started putting in random words to bring up their site in search engines.
To help curtail this problem Google developed the pagerank system. Sites with more links pointing at them came back higher in the search results. Legitimate sites should be well linked. This system, used in conjunction with the web crawlers that searched and indexed site text, provided amazingly accurate search results.