St. Petersburg Times Online
 Devil Rays Forums

printer version

In search of . . .

Crawlers, spiders, search engines. Which one's right for you? Experts agree on only one thing: Internet searches leave a lot to be desired.

By DAVE GUSSOW Times Technology Editor

© St. Petersburg Times, published October 4, 1999

Ask a simple question, and what do you get? With an Internet search engine, it could be anything, and not necessarily what you're looking for.

"A lot of people don't really know how search engines work," said Steve Lawrence, a researcher at the NEC Research Institute in Princeton, N.J. "If you don't know how they work, you might have an impression that you're searching the entire Web. And that's not true."

Which search engine was used? How precisely was the question worded? Did the user do enough to narrow the focus of the search so he doesn't have to wade through millions of irrelevant hits?

It's not only users who have trouble understanding search engines. It's also search engines that do a lousy job of giving users basic information, such as how they work, what they cover and how best to use the service, experts say.

Lawrence was an author of a study released in the summer that showed Internet search engines, even the major ones, scan only a fraction of the estimated 800-million Web pages.

The findings may have surprised many people, but not experts. They say search engines have come a long way in a few years but still have room for improvement.

"No two search engines are exactly alike," said Danny Sullivan, editor of the Search Engine Watch newsletter ( in London. "They not only have different databases, but they actually pull information out of that data source differently."

For casual users, though, it leaves a not-so-simple question: How do they effectively search the Internet?

The first thing to know about the hundreds of search engines available is that not all search engines are search engines in the strict definition of the term: a program that searches the Internet for documents with requested keywords. And it's not easy for users to find out how a particular engine works:

* Some are directories, including the widely-known Yahoo (, with listings compiled by people, not software. Squads of Web catalogers constantly prepare and update long directories of links on thousands of subjects.

* Some use software known as crawlers or spiders to roam the Web and create their indexes. AltaVista ( and Excite ( are examples. It's not a foolproof system, though, partly because some sneaky Web designers load a page with keywords, invisible to the casual visitor, that are intended to give their sites higher rankings in results.

For example, typing in Pamela Anderson Lee on AltaVista brings back almost 52,000 matches. At No. 20 was a site offering to pay people to surf the Web, with no obvious reference to the former Baywatch star whose racy photos are in demand by some male Web users.

* Some, such as Lycos (, combine results from a human-compiled directory with automated database searches.

* Some are metasearch sites that rev up other search engines to look for a cross section of results. MetaCrawler ( is an example.

* Some newer ones, such as Google ( and Direct Hit (, skew their search results to sites that are most popular with users.

New search sites, including some with searches using photos instead of words, pop up all the time. Which is best? It depends on what you're looking for.

Sullivan gave a general recommendation: "If you don't know where to begin, directories are good places to start."

Experts say to try several searches to make sure you're getting the best information available. "Users don't try very many different search engines," said NEC Research Institute's Lawrence. "They stick with one or two."

People seem to gravitate to the engine du jour that others are talking about or that won some media coverage. Newer engines such as LookSmart, Ask Jeeves and GoTo ranked among the most popular search sites over the summer, according to Media Metrix, a firm that measures Web usage.

In addition to differences in how they retrieve information, search engines also differ in how users are supposed to conduct searches. For most, typing in a word or phrase is all that is needed. Some, such as Ask Jeeves (, allow users to type in questions in plain English sentences. Of course, that doesn't guarantee the searcher will get the answer he's seeking.

Users can help themselves with some simple steps, experts say: Word each request precisely and use singular terms instead of plural ones. It may seem obvious, but some engines offer users a choice of languages for the search. If users don't click, say, English, the search will include sites in foreign languages.

On some sites, putting a plus sign (+) before a word means that it has to be in the answer. Using quotation marks around a phrase will help ensure only sites that include that phrase are found. For example, typing in pepperoni pizza on AltaVista brought back more than 81,000 hits. But using "pepperoni pizza" reduced that to 2,888.

On many sites, a search can be narrowed by wording the request as a Boolean expression -- using words such as "and," "or" and "not" to specify what does or does not need to be in the results.

"Nobody wants to do all that stuff," said Sullivan of Search Engine Watch. "People are afraid of Boolean. It's frightening to them."

And users don't always take the simplest route. Sullivan said one search engine did an experiment: It had a screen with a word search field where users could type in requests as well as a hyperlink that would take them directly to a company's stock price. Users were asked: How would you search for this company's stock price? Most people typed in the company's name in the word search field rather than simply clicking on the link.

One method Sullivan uses to gauge a site is by just typing in a company's name. If it finds a link to the company, it gets a perfect score. Not many do.

"I'm still surprised that search engines haven't gotten better at making that happen," Sullivan said.

Not all engines offer users a choice of Boolean expressions (named after 19th century mathematician George Boole) or other methods to narrow searches. And many searchers who choose not to take those extra steps can end up with an avalanche of hits. But even millions of choices shouldn't intimidate, the experts say, because "relevance rankings" generally place the results most likely to be helpful near the top.

"Overall, if you take the first 10 or 20, you'll find some pretty good ones in there," said Randolph Hock, author of Extreme Searcher's Guide to Web Search Engines: A Handbook for the Serious Searcher ( "How they're ranked in the top 10 or 20 or 30 has a lot of variability. Quite honestly, I'm not sure what's in the mind of some of the programs."

What some programs have in mind is touting their advertisers.

Search engines started out as free services to look for information. Now, they also are looking for profits.

Many, including the pioneer Yahoo, evolved into Web portals that include searches, free e-mail, communities for people with similar interests to gather, shopping, news and more. Those free services for users, though, cost money to maintain.

So advertising dots the virtual landscape. Most of it is easy to spot, but some may not be readily apparent.

For example,'s advertisers bid for placement on search results, with the site noting in parentheses how much the advertiser paid for the placement. Typing in new cars, for example, gives results that are topped by a site for a free insurance quote, with the cost to the advertiser listed as 72 cents for each "hit" by a searcher.

This fee-based placement is an economic necessity, chief executive Jeffrey Brewer says, so sites can offer free searches.

"The portals don't have an economic model for search," Brewer said. "Search is used as a loss-leader at portals."

Some critics say the drive for profitability has come at a cost.

"They have been failing in their core mission of helping people find things," said Jakob Nielsen, co-founder of the Nielsen Norman Group, a Mountain View, Calif., company that studies the usability of technology.

Companies deny those charges, saying they haven't neglected search services in their pursuit of profits. In response to the NEC study, for example, Excite@Home and Lycos said they were going to expand their coverage to include more Web pages.

Including more pages in their search territory, though, isn't necessarily the right answer.

"They need to improve their relevancy as much as possible to get people to better sites for simple queries," Sullivan said.

Exactly, says Sergey Brin, president of Google Inc., one of the hottest new search engines. Brin said Google was born out of his frustration with the mishmash of results from searches, so he and Larry Page started work on it about four years ago at the Stanford University Computer Science Department.

Even before they took down the "beta" label from the site on Sept. 21, they had won accolades for their engine, including an excellent rating from PC Magazine for relevancy of its results. Now, Google's ready to make its name in the search engine field, but not as a portal, said Brin, president of the Mountain View, Calif., company.

"I think we want to be a search destination site and features related to search," Brin said. The site will be supported by advertising, some of it tied to searches, but the ads will be clearly labeled.

The quality of the results are important, Brin said, because users "pick the tools that work the best." And, in the search engine market, loyalty can be fleeting.

"There is no allegiance out there," said John Corcoran, an analyst with Stephens Inc. in Boston. "Consumers are saying, "If I can get the information with only three clicks on Yahoo, fantastic. But if I can get it with two clicks on a newer search engine, I'll use that one.' "

Competition will make search engines better, as will improved technology, the experts say.

Some see a shakeout, with a few big ones surviving and a lot of specialized sites thriving. For example, one search engine might specialize in legal issues for lawyers; another in health-care issues for doctors; and yet another in entertainment.

Google's Brin says engines need to be more than robotic answer machines.

"A perfect search engine would understand the query and would understand all the information out there," Brin said. "The perfect search engine is a long way off. There's a lot of work to be done."

Even with the flaws, the experts say, search engines are doing a good job and will get better.

"On the average, they're quite good considering the problems they are faced with," said Hock, author of the Web searcher's handbook. "And that is there is more material out there that is available than there's ever been before, and most of that material does not come to them with any type of organization or any clues for indexing."

Remember, he said, what it was like going to a library 30 or 40 years ago, with only the information on those shelves readily available.

"Our expectations have increased so dramatically with the coming of the Web," Hock said, "that we sometimes don't look back."

-- Information from Times wires was used in this report.

Back to Tech Times

Back to Top
© St. Petersburg Times. All rights reserved.