What's the latest with Search Engines
these days?

INFORMATION OVERLOAD

Having trouble finding what you're looking for? Getting upset because you can't? Don't despair ... you're not alone.

Contrary to what some folks would have you believe, the Internet is growing. According to research released in August by Lawrence Roberts, one of the pioneers of the Internet, cyberspace data traffic has been doubling every six months. This in turn is fueling huge growth in the volume of material on the Web, which makes it hard for search engines and other outfits that try to index the information to keep up.

The result: Web Rage, where users quickly get frustrated looking for what they want. In January, Internet consulting firm Roper Starch Worldwide concluded that users get angry after only 12 minutes of searching.

Some of this anger is understandable. It is a huge digital world out there and it is pretty daunting. Google, reckoned by many to be the biggest, baddest search engine around, claims to index more than 1.6 billion Web pages. But some folks who have been around longer than they are willing to admit can recall how hard it was to find anything pre-Internet, or before the user-friendly Web interface came along and made arcane search programs such as the Gopher obsolete. Yahoo, Lycos, InfoSeek and others battled early on for consumers' eyeballs. Then, in 1998, Google arrived and proved that there was room for a better mousetrap.

But times are now tougher than ever in the Internet world. Advertising and sponsorship dollars, the traditional revenue sources for search sites, are scarce. To succeed, new search technology must operate cheaply, while still outshining competitors.

OK, there are a lot of pages out there, but if I run a search on Google for a cake recipe (Battenburg, for example) I get 117 hits in 0.21 seconds. The first on the list was right on the money.

So, the first piece of advice: keep some perspective. Type in a sensible range of words for what you are looking for and don't expect search engines to be rocket scientists. Google is still probably the best for general use, but there are others.

THE LATEST SEARCH TOOLS
Now a new generation of search tools is here. The newcomers hope to profit by making it easier for you to track down information scattered across the Internet. To accomplish that, they go beyond scanning the Web and ranking the results. Increasingly, these recent arrivals suggest, the focus will be on how to best present the results.

The arrival of these new contenders represents a bet on the ever-expanding need to deal with information overload. The search-engine industry was one of the first segments of the Internet to be saturated by competitors.

AllTheWeb (www.alltheweb.com), for example, returns 95 matches for the Battenburg recipe almost as soon as I had finished typing the request. Another recent enhancement to web searching is the free Google Toolbar (toolbar.google.com), which settles in the menu part of Internet Explorer--it doesn't work with other browsers--lets you enter searches without having to jump to the Google home page.

And if you are looking for more than brute-force searches, there are some interesting alternatives: Oingo (www.oingo.com), from Applied Semantics, uses technology that finds cultural links between words. The results are presented in a grid that shows category matches as well as direct text matches. Vivisimo (vivisimo.com) does much the same thing, as does Wisenut (www.wisenut.com). All three are well worth checking out.

Type the word "speakers" into the search box at Vivisimo and you'll see how approaches to Web searches are evolving. On the right side of the screen is a set of search results, presented in a familiar format. Users see a list of Web pages, with links to those pages and brief snippets showing which portions of the page contain the search terms.

On the left side, though, is something different: a list of categories, including "Speakers Bureau" and "Loudspeakers." Those categories represent shortcuts to very different search results for the word "speaker." Clicking on one would help a user looking to line up a speaker for an event. The other would assist someone interested in stereo equipment.

Categories have been used before to help Web users find things. Yahoo itself started out as a list of Web links arranged by topic. But Yahoo and other directory-style listings have depended on humans to evaluate pages and assign them to fixed, hierarchical categories, such as "Home: Entertainment: Consumer Electronics: Audio: Speakers." Because the approach requires people to do the sorting, categorized directories usually cover only a portion of the sites indexed by search engines

. Vivisimo and other new search engines create categories on the fly. Computers analyze and sort the results--no humans necessary. Raul Valdes-Perez, Vivisimo's president, thinks the approach will revolutionize searching. When search results are presented as one long list, he says, the information you're looking for may well be so far down, that you never see it. "By following our folder labels, you get to results that would otherwise be invisible," he says. It's an interesting point. Google's great achievement has been to rank search results so intelligently that people consistently find what they're looking for at the top of the list. But even the best ranking system can't decide what you want all the time. By grouping related results, the new generation of search tools provides a belt-and-suspenders approach.

Technically speaking, Vivisimo isn't even a search engine. Instead, the site grabs results from other search engines, then uses its technology to organize and display the results. The company--with a grand total of three full-time employees--hopes to sell its technology as an add-on to other search systems.

Wisenut, another newcomer, also uses automatically generated categories to sort results. It tosses in a nice twist, too, a feature called Sneak-a-Peek. Users can preview individual sites without leaving Wisenut. The idea is to end the back and forth that occurs when users have to leave a search page to check individual results that then prove irrelevant.

Another new entry, Teoma, makes a distinction between what it refers to as communities and experts. Located at www.teoma.com, the site organizes results into subject categories--the communities--and displays those topics. But it also includes a section called Experts' Links. These are pages that Teoma's system has identified as key lists of links on individual topics, often produced by individuals or organizations with expertise in a field. Teoma's system uses the Experts' Links to help define subject groups, but it also offers direct access to the expert pages. (In mid-September, Teoma was acquired by Ask Jeeves.)

One other new search site worthy of mention is Daypop. This site, at www.daypop.com, isn't meant to compete with Yahoo, Google or other general-purpose search tools. Instead, Daypop scans and indexes some 5,800 news sites and web-logs, those personal online journals that usually contain commentary and links to other sites. A new Advanced Search feature allows you to limit your search by language, country or time frame. Also, the "Daypop Top 40" lists the links that are currently popular with webloggers from around the world. Daypop's creator is Dan Chan, an avid weblogger who got the idea for what he calls a "current-events search engine" while trying to follow news about the 2000 presidential campaign. For news junkies, especially in the aftermath of the Sept. 11 attacks, Daypop is a must- bookmark site.

In summary, unless you are a serious, full-time researcher, install the Google toolbar, use it for basic searches, and be done with it. For more elaborate searches, download the free version of Copernic 2001 Basic (www.copernic.com), a program that trawls 80 major search engines including AltaVista, Excite, HotBot, Infoseek, Lycos, WebCrawler and Yahoo into seven different categories and filters the results for duplicates and relevance.

DISTURBING NEW DEVELOPMENT
-CPC SEARCHING
Clearly someone believes search engines still will make money, or there wouldn't be so many new ones coming online. Indeed, here is something that may further confuse users: the arrival of cost-per-click searching (abbreviated to CPC). According to a report from Wired News, some of these new search engines are generating cash by accepting paid listings to get sites mentioned above the others when a surfer enters a search phrase. So, entering "Battenburg cake recipe" on one of these engines could, in theory, throw up matches that would lead me to commercial sites selling cake ingredients or tours of Battenburg, the town. Those commercial sites would then pay a few cents to the search engine depending on where they are placed in the listing. A nice little money-maker for the search engine, but not necessarily what I'm looking for if I want to bake a cake.

Another problem with these CPC search engines is that it isn't always clear who they are. Ah-ha (www.ah-ha.com) is one, as is SearchBoss (www.searchboss.com); both will jiggle the search results according to which commercial sites are willing to pay up for prime space. We can understand that search engines are, like every other dot-com venture, governed by money, but their argument that such sponsorship helps consumers by leading them to the most commercially viable sites, seems a bit dishonest. More importantly, it blurs the line between a search and a sales pitch. It's kind of like asking a ticket scalper outside a sports stadium where you can buy legitimate tickets.....you don't exactly trust his answer.

Hopefully these search engines won't take over the planet. Instead, they could make their money in more straightforward ways, such as licensing their technology for corporate networks or by hooking up with hardware manufacturers: Google, for example, recently announced a deal with Logitech, to embed access to Google in Logitech mouse buttons and keyboards. Maybe not a bad idea to help newbies.

© Algorithmica Japonica Copyright Notice: Copyright of material rests with the individual author. Articles may be reprinted by other user groups if the author and original publication are credited. Any other reproduction or use of material herein is prohibited without prior written permission from TPC. The mention of names of products without indication of Trademark or Registered Trademark status in no way implies that these products are not so protected by law.