Using a search engine to "research" information on the web: A primer

This page was built to support a workshop to teach college students smart searching. The questions and answers presented here are general. The world of search engine technology changes so fast, always check websites like searchenginewatch.com for up to date information on the search engine industry/discipline. ALWAYS verify the time stamp and source of any web page that purports to provide you with information you are seeking. (including this one!)

How does a search engine work?
How are pages ranked?
What tricks are there for refining a search?
Why would I ever NOT use Google?

What is Google Scholar?

How does a Search engine work? Well, that depends.

There are many types of search "engines." For information on what companies own which engines, take a look at the chart at Searchenginewatch called "Who Powers Whom? Search Providers Chart."

Directories/Indexes or "engines" for profit. Many new search engines are actually ad driven. In other words, companies or vendors buy their spots on these engines; the more money spent, the higher the initial ranking. Most of these engines are known as "Pay-per-Click."
Crawlers or Bot driven engines. This is what we hope we are using when we type a query into a search box. Crawler engines use "bots" that follow link after link on every web page adding each url to the mothership database. When you type in a word or phrase, your query finds a match within this database of cached information, and then that information is returned to the browser in the results page.
Specialized engines. Library databases, private collections and other online databases are special engines. In order to search these treasure troves of information, you have to go to a particular place and look. This is where (some say) more than 90% of information lives on the web--sometimes called the "dark web" or the "invisible web." The Vassar Library has a great collection of these .

When you get a page of results after a query, how are pages ranked and returned to the searcher?

Once again, the answer is, "It depends." For the purposes of this discussion, we will talk about how google works--in a very small nutshell. Google uses their own proprietary PageRank™ technology, which gauges a web page's popularity based on on a number of factors including:

how many other pages link to it
how popular/authoritative are the pages that link to it
how relevant the page is based on textual content

Be aware that all engines also return advertising links as well as authentic links--including Google. Google is good because it clearly labels the two making a clear distinction between what some call "church and state."

Here is a link just for fun: Google's PigeonRank

How can I refine my search? What are those special characters I can use again?

For the record, every search engine is different and has different assumptions. Some engines will understand spaces between words as AND and others as OR (not as common). Google has a great search tips page and an exhaustive list of their supported modifiers.

**boolean and special operators**
" "	exact phrase
+	Required word in results
-	Omit this word from results
~	Find synonyms for this word
OR	Either this OR that
...	number range $20...25

Why would I ever NOT use Google or the web for my research?

Two reasons:

Search engines are powerful and very cool, but they only return a SMALL SUBSET of the information available. Some folks say that roughly 90% of the web is inaccessible to search engines. Dark web; Invisible web.
Even if you could access 100% of what was on the web, most of it is garbage--or at least, not academically inclined. A huge percentage of the knowledge of mankind is still available only in paper format or in specialized search engines, databases and library indexes. Libraries are great places for accessing ALL of this information--including everything on the internet. Libraries also have full-time dedicated staff who are there to help you in your quest. How cool is that?

What is Google Scholar. Should I use that?

Google Scholar is an attempt to enable web users to search scholarly content in the way that has long been possible in an academic library. The tool finds links to pertinent articles, books and technical papers and even provides the citations from other works in the databases. Essentially, Google Scholar is using its PageRank™ technology to search a number of scholarly publisher's data stores--but Google does not list the specifics. The "About Google Scholar" page does not list the participating publishers. Scholar is still in beta and it shows, although it can be quite useful for those who are searching only in the sciences. The scientific disciplines seem to be better represented than the humanities. Scholar is an ambitious attempt at what libraries call "federated search," the ability to meta-search multiple databases simultaneously using the same searching rules and technologies.

This page was last updated on 8/8/06.

Computing