searengi.htm: How to search: the sublime art, explained by fravia+

S E A R C H	fravia's search lab Fravia's Nofrill Web design (1998)		partly updated November 1998	How to search: the sublime art There are 350 million sites out there, doubling every four months... add to this the 'second' internet (the new 'university connection' net) and the wide and huge usenet, and you already have a plethora of universes to explore. And there are also all the old dark web-corridors, made of forgotten archies (and veronikas! :-), 'obsolete' fidonets and much more... Therefore: where, where, where is the info you need? Study! If you don't master the sublime art of searching you'll never find that what you seek! You'll be able to learn here the first elements of this art. Once you'll have understood them, you'll be able to go even further (pretty quickly) if you learn perl (THE language for bots building) and if you will study and research a LOT. Meanwhile, for a start, you will be able to peruse this 'heavy' search engines page (which loads quickly anyway, since it does not carry silly backgrounds, useless frames or slow loading futile publicity). There is also a "light" version, which you may want to copy onto your hard disk as a quick search starting point. Since search engines do play a relevant role when searching (duh), I have also added a search engines' vagaries page, that you may find interesting. Of course, you should learn how to search the web using inter alia my own lessons!

Special 'Mover' for this search engines ('heavy') page
Useful tips	evaluate results	search engines forms (and explanations)
check which search strings others use		Go to the professional search page
inference robots and scripts		Go to the how to search page
search fravia's site		Go to the search engines' vagaries page

some useful tips on how to search the web!

1) Use always lowercase search strings: fravia will fetch "fravia" and "Fravia"; frAvia will fetch only "frAvia" occurrences.
2) keep in mind that searching on the web is THE CONTRARY of a search in a normal library: in a library, the more generic your search, the more easy it is to find what you are seeking, on the web the more specific your search, the more easy it is to find what you are seeking. So it is relatively easy to re-find a page that you happen to have read, even if it has changed site, as long as you remember (well) a part of it. Try it: just cut and paste into Altavista these words (from my main page) "contains many teachings, and will help you gain knowledge that you will not find elsewhere. Please wander slowly inside: sip a good" (Yes, all of the preceding 'blue' words: don't forget the " and remember that there's no limit at what you can paste inside a search engine form)... you'll find -amazingly enough- my very page!
3) It helps to do lots of searches from lots of different search engines, each of the big engines is different, and each has its own presentation format

For instance, Yahoo does things in categories, Alta Vista uses phrases best, and Lycos is good for huge keyword searches. Webcrawler gives at times surprisingly good results. Hotbot has a HUGE database, and is more than good for just quickie searches, anyway as it seems: Altavista and Hotbot are 'going down', while Infoseek is 'catching up'... Hotbot is easyly configurated to avoid 'stale' links... well, yes, there are quite a lot of "must know" differences between all the main search engines.
I still prefer Altavista though... my advice?

Use always (at least) TWO search engines (Infoseek and Altavista IMO) and a directory!
According to myself (and that's not a bad source of info :-) combined searches using multiple engines increase the likelihood of finding the desired information by 4-5 times. A very good idea is also to use one of the existing inference robots and scripts...
3) Check the traffic in your area before starting a web session at http://www.internettrafficreport.com! Slow connections? Wait for a better moment!
4) DO NOT GET DISTRACTED! Web surfing is terribly distracting! One search session out of three crashes before reaching the desired target because you got lost on those 'interesting side-trips' that you found on your way (translated, this means that you moronically just forgot what you originally wanted and began following godzilla knows which paths). Be coeherent (and be 'hard', man: real seekers are quite concentrated fellows) write down your target and go for it (and NOTHING else until you find it!)
Devote a minimum of 30 minutes and a maximum of 60 to your search session, that's the best time range IMO, less isn't enough and more bores. Remember also that real queries are made in phases: phase one is 'broad', after a while (say a week, you must first digest and understand what you found) you go for a more concentrated approach, and then, last phase, you really find what you wanted (after having perused the newsgroups and searched the archies).
5) Search USENET! Remember that there are THREE search strategies and that usenet searching is the second and very powerful one!
- ONE: you search yourself (searching);
- TWO: you use people that have already searched (combing), for instance on usenet, and use the links and info that they have gathered during years-long queries (no need to re-invent the wheel every friday)
- THREE: you look what kind of people (and from where) come and sniff at the info you provide on your own sites (klebing)... but that's advanced stuff and you'll learn it in due time...
6) I would recommend that you turn off auto image loading before submitting a request to ANY search engine when you'll use the forms below, this will speed up your searches CONSIDERABLY. (Come to think of it, it's always better to turn off mage loading IN GENERAL, unless you really need to find/see images). Btw, the control about image loading options is another of the many fields where the new slick Opera browser demonstrates its absolute superiority over both browsersaurii...

If you want to learn more about search engines, go to the search engines' vagaries page

search engines explained!

Please note that there are a couple of "all in one" forms at the bottom of this nice page

AltaVista is USEFUL: it's the largest Web index, with millions of Web pages AND articles from usenet newgroups (and these may be REALLY useful). Altavista is still the search engine of choice for all "old hands" of the web, yet Infoseek is now (1998) slowly catching up!
AltaVista can also be used to get all the page linked to your page. I can get with the following link the pages linked to a no more existing page of mine (useful in order to find stale links): try it. Another useful method is to use the *HTML TAGS*, if you for instance search "image:bettie", AltaVista will find quite a lot of images about Bettie Page.

Search Method: Keyword
Data Base System: Full text, largest and more inclusive indices
Operators: If used without suitable operators Altavista produce enormous noise and little signal

For the more paranoids (or the more careful) among you, here is a link to the

anonymized Altavista search form
(Courtesy of fravia... do not leave your tracks around!)

Yahoo (it's an attempt to catalogue the entire Web, search on a topic) Very good for beginners, well organized.

Search Method: Subject and Keyword
Data Base System: Limited coverage: indexed by human operators
Operators: In keyword searches selects only sites that contain ALL search words. If no exact match is found switches automatically to AltaVista

Exite (a concept-based search engine, tries to figure out what you mean). It's awfully slow, because they want to "impose" you to read the stupid pubs before getting on with your search. Use it only as last resource or if you are a beginner (it's very easy).

Describe what you want to find using Excite...
I want to search		Do not use query syntax in the search forms below
My search results MUST contain
My search results MUST NOT contain
My search results CAN contain

Display my results by document with	and	results per page.
Display the top 40 results grouped by web site.

Search Method: Subject and Keyword
Data Base System: Full text, circa 50 millions docs, titles are not searched (or so it seems)
Operators: Automatic word-root search (without the need to use a joker like '*') and sorting by site

WWW Worm: It is extremely flexible, allowing regular expression searches on URL, subject and content. In spite of its flexibility, it's not difficult to use, and there exist a straightforward tutorial.

Lykos Big and slow, Lykos has a large number of binary files in its database, has greater depth than most search engines, because it also indexes FTP archives and Gopher menus. Besides, you can search with Lykos per email. There are some other interesting Lykos resources as well, including a list of the frequency of over six million words used on the Web. Lycos is one of those facilities that's almost too good, presenting you with more information than you really need, but it's a great resource if you use it carefully to narrow down your search.

Find

Lycos catalog Point reviews a2z directory

lycos_ftp normal
lycos_ftp advanced

NORTHERN LIGHT, one of the most recent search engines.

(Click redball to perform your search)

Select:	All Sources -- Search the World Wide Web & Special Collection
	World Wide Web -- Search the entire World Wide Web
	Special Collection -- 1 million articles not on other search engines

Webcrawler fast and cool, returns surprisingly relevant results. This excellent search engine indexes the contents of Web documents, so you can find pages that contain a particular word or phrase.

InfoSeek Net Search Good output, their attempt to get "commercial" failed against the sound Web altruistic spirit (:-), it's now a free service. This claims to be the largest set of searchable indexes to WWW pages and to USENET newsgroups. 1998: INFOSEEK is getting better and better! (They value QUALITY very high!)

Search Method:Subject and Keyword
Data Base System: Full text, over 50 million pages
Operators: detailed instructions under help

HotBot (ex Inktomi, claims to have the largest index and the best scalability in terms of keeping up with the Web's exponential growth, claims to be able to re-index the entire Web every week)
Well known for its awful colors, this was one of the BEST search engines available, unfortunately slowed down by useless graphics frills.

Search Method: Keyword
Data Base System: Full text, over 60 millions documents
Operators: Simple and advanced... detailed instructions under help

MAGELLAN

Internet search wizard Compuserve's forms' selection, this one you are using is better

ftp search 3000 FTP sites around the world, quite slow and busy at times. ("This server is located in Trondheim, Norway")

Archie request form (but you better use Archie trough email, see

here how to do it).

Here a useful list of Archie services (gateways) in the World Wide Web. The latest version is always at http://www.nexor.com/public/archie

All search engines of the planet... the coveted list by Andrei Nedashkovsky (some of the links do not work, though)

Planet search

And, finally, a 'portal' to the

Japanese search engines... (:-)

Inference robots and scripts

I have decided to use this funny name for those scripts that allow you to query (almost) simultaneously more than one search engine. I have two forms here for you: inference find and dogpile. Both are mighty interesting for the casual or the 'hurried' searcher, yet I believe inference find to be a VERY USEFUL TOOL even for advanced seekers: It will not only query AltaVista, Excite, Infoseek, Yahoo! and Webcrawler (quite a good and correctly limited choice per se), but it will present to you SURPRISING RELEVANT ANSWERS in a special formatted *.htm file that you can IMMEDIATELY DOWNLOAD AND USE!. Here you are:

Inference find search

Here is how you perform an e-mail search on dogpile... Infact, this is the above form "cracked".
Dogpile searches ALL main search engines and compacts the results for you. Often e-mail searches are cleverer than live searches on the web (for the same reasons that make "dead listing" more effective than live winice approach when reverse engineering, btw :-)

email an Agora server (be ashamed if you do not know what an Agora robot is) with following TEXT (if you want to search for "numega" AND "softice" for instance :-)

send http://207.126.101.190/?q=%22numega+softice%22\&to=forty\&sort=key

And, last but not least, these are my all in one forms (courtesy of fravia)

(Altavista, Excite, Lycos, Netsearch, Webcrawler, Yahoo, ftpsearch, filez.com, interpix image search)

(AV, EX, IS, LY, YA)

Search the internet for

evaluate the results of your searches!

Do not forget that a most difficult art is to learn how to evaluate the results of your searches!

check which search strings others are using!

Really useful search engines allow you to check which strings others are using as queries. We have already seen (inside the "klebing" search technique section) how important "alien" search strings are, for each one, in order to ameliorate your own search strategies... yet it's still pretty funny to check what people look for... (and pretty sad at times :-( see how frequently people misspell their queries, and how incredibly often strings like 'Pamela Anderson' or analoguous idiotical "slave lemmings" subjects get search requests.

http://webcrawler.com/WebCrawler/Fun/SearchTicker.html

http://search1.metacrawler.com/perl/metaspy

http://voyeur.mckinley.com/cgi-bin/voyeur.cgi

use a "professional" search page on your harddisk!

You should copy the "professional" ("light") version of the search engines' forms on this same page on your hard disk and choose it as "bookmarked" or "hot" (or as "favourite" if you are a Micro$oft's slave). Use it as your "main search engines" starting page. You'll have quite a lot of advantages:

1) You'll have all main search engines at your fingertips
2) The search starting page won't be fetched from a busy server and will therefore load immediately
3) The search starting page won't have any images (and therefore don't suffer ANY delay whatsoever)
4) The search starting page will spare you HOURS of connection wasted time!
5) The search starting page will be edited by YOU with your own additions and comments! (send me any interesting addition :-)

You dig it? So Shift and click on

this link.

NB!: The "light" searchpage above is A MUST
for (beginner) wizard searchers!
(and you can download it for free, say "thank!" to fravia+ :-)

how to search ~