How to search the web
(Updated: July 1999)

Fravia's lessons
based on some original private emailings from +ORC
Robin Hood's stratagems
Surreal5's search tips
Cassandra's search engines
Other essays and tools

[go to the
heavy search engines page]     [go to the light search engine page]

An introduction to web-searching
(Websearching, the sublime art)

I see it coming... in a few years (actually already now, even if most zombies establishments have not yet realized it) one of the most important jobs will be, of course, websearcher.
We'll have many specialized branches: web-searchers, web-stalkers, web-seekers and so on. Zen and 'feeling' as well as a very broad 'global' knwoledge will be required.
It's a good anthidot to the hyperspecialisation that has nearly brought the whole silly commercial oriented society we are compelled to live in into a well deserved dead end: only large-minded, capable searchers will be able to keep the 'larger' over-perspective, and will be able to find ANYTHING they need (for free, of course), from Vivaldi's Concerto n.7 in F for four violins and cello (it's on the Web) through the second edition of the Police Criminelle, Technique et Tactique (it's on the Web) to A Western Australian survival kit for writing English (it's on the Web).
For the first time in the history of humanity it DOES NOT MATTER ANYMORE (for knowledge purposes) if you are located in a big rich city with huge libraries and universities or if you happen to live in the middle of nowhere or in a very poor country!
EVERYTHING is on the Web for free! I mean: any book, any newspaper, any university paper and any image, moreover (soon) any sound, any music!
This means that (albeit amidst mountains of useless garbage) ALL KNOWLEDGE is on the Web, for free, for you to discover and enjoy!
If you don't believe it, just learn how to search!

You'll gain here knowledge that is very 'handy', as you will see, and that you will not easily find elsewhere, even if I have noticed that many "universities zombies" have already begonnen to use (not to spread, to use, a subtle yet significant difference :-) the techniques that we have invented and published on this site, giving very seldom -how characteristic of university "habitat" and mentality!- any sign of recognition of our work :-(

Anyway, I'm sure that the importance of this section will be more and more clear with the development of the Web (or at least of the 'sound' and neither commercialized nor brainwashed part of it :-)
This is a 'living' workshop, of course, that will florish gathering more and more additions from all the great wizard searchers among my readers (and beyond, since "real" wizard searchers will obviously find this section all by themselves :-)
Hope to hear from you, and to receive contributions, from many searchers. Remember: the whole thing works only if we build on the shoulders of others and if others build on our ones... if you just leech, you loose and we loose! (that's incidentally the reason I have started with my last lesson to produce 'light' public versions and 'heavy' versions for contributors :-(

Robin Hood... Surreal5... Cassandra... you lazy scoundrels... where are your promised next lessons and essays?

Fravia's own lessons
[Available lessons:]

redlesson_5 ~ General use of agora, http:// retrieving ~ July 1996 ~ complete
redlesson_6 ~ Ftping files, agora queries and emailing altavista ~ December 1996 ~ complete
redlesson_7 ~ W3gate, search spiders, error messages and evaluation of results ~ March 1997 ~ complete
redlesson_8 ~ Advanced searching techniques (combing and klebing) ~ November 1997 ~ complete
redlesson_9 ~ Searching effectively ~ Site monitoring ~ January 1998 ~ complete
redlesson_10 ~ Let the bots search for you ~ and build your own search-bots :-) ~ June 1998 ~ 'light'

Robin Hood's lessons
Well, from the dark webwoods of Sherwood, all of a sudden, Robin Hood has sent me three 'promising' how to search lessons that you should by all means read (and head). I'm very happy that good searchers start to contribute to this section, and hope that many other searchers and stalkers will add their own findings. In fact effectively web searching means to master techniques that are far from obvious and that anyone seriously intentioned in reversing should master. So I'm very happy to host here these "how to search" redlessons and stratagems by Robin hood.
In fact, believe it or not, on the Web ther are some 'rangers' like in Tollkien's books, people that are neither proper hackers nor crackers, but that know how to fetch what they want, if needs be. Robin Hood seems to be one of them and I'm glad that he decided to start helping us.

Surreal5's lessons
Well, Surreal5's lessons (and his introduction) that you'll find red "here", are IMO more important for the approaches and the techniques they deal with than for their specific subjects (basically how to find warez). Nevertheless, since warez ARE on the web (much too much, IMO) why shouldn't you learn how to find them?
This is what Surreal5 wrote:
I've tried to organise this (provisorial) essay so, that both beginners and
advanced users can get something out of each lesson, and can almost
directly use it. Even if you are already an expert on how to find the
example types i used, you may get something out of it, a different
perspective, something you hadn't thought of. I like Robin Hoods tutorial,
he has a broader experience than i do, but i feel my lessons give faster
results, with less effort spent on learning boolean ,and other tool
specifics. Consider this a crash course if you will. Consider it a
preliminary to Robin Hoods tutorial. Consider it an alternative view from a
lazy guy who wants results fast ;)
Methodolgically ('Nethodologically'? :-) Surreal5's documents are very important essays. I have tried myself some of his tips... for instance, if you need to navigate and find goodies inside huge 'messy' sites (like mine :-) you just add/update the relevant starting urls to a couple of quick (and reliable) search engines and let their robots do all the work for you! Alternatively, I may add, I found the service of those little search robots like the freefind or wired-worker's bots quite useful for searching purpose on sites that don't even need to know you have indicized them completely :-)

Cassandra's search engines
No good 'how to search' section would make any sense without some good 'homemade' search engines... of course you can always learn how to use better the main ones, for instance on my own redsearch redengines pages, yet there is another interesting solution by Cassandra, here it is:

Here you have Cassandra's redstalker
and here you have Cassandra's redfetcher

They are still in a experimental phase, so bear with us and let's hope that cassandra will send some updates

This is what Cassandra wrote to me:
I have divided my work in two parts : the Fetcher and the Stalker. 
The Fetcher provides an easy-to-use access to various search engines:
  Altavista, Yahoo, HotBot, Lycos, Infoseek, webcrawler, Dogpile, FTP
search, ASK SINA (an ftpsearch-like with a database containing records of
germany sites, mostly), Northern Light Search (recommanded by US army,
although it's not supposed to mean anything), Goto (formerly WWW Worm).

The stalker provides gateways to finger and whois, along with the 'dejanews
search filter' without all the ugly grafix. It's not really developped, for
stalking matter is related to the country your prey lives in.

btw, Fetcher uses frames (yes, frames!), but in a clever way : you'r never
jailed in a small portion of your browser window. If used sparingly, frames
can be useful.

Of couse, each one might grow with engines or stalking services. But i'll
develop it only if you or someone else you could give it to is interested
and find it worth growing.
Well, I hope that this 'search engines' form approach will be developed too. I'm sure my readers will immediately understand the practicality and the convenience of having some single, well tuned, search engine forms, without having to use the awful commercial entrances, delayed by all the crap advertisement they put on (as if they would not gather enough money and power just LOOKING at what people search :-(

Other essays and tools
Essays, tools and papers about search-related matters

new_0101.htm: How to search: The phf exploit
by +Thor
27 April 1998

boyd1.htm: Fravia's copy of G.E.Boyd's E-Mail Servers Listing
by G.E.Boyd
June 1998, updated July 1999
unvaluable list, if you know or learn how to use these beasts. Boyd is a (the :-) famous master accmailer.
Added in July 1999:
The importance of Accmail, by fravia+
kmart_s1.htm: More searching tips (Advanced searching)
by Kmart
23 December 1998
Advanced tips for advanced searchers, if you use these search techniques you can find yourself getting quite a bit of class A, quality info
search engines - heavy ~ search engines - light ~
redhomepage redlinks redsearch engines red+ORC redstudents' essays redacademy database
redtools redjavascripts wars redcocktails redanonimity academy redantismut CGI-scripts
redbots wars redcounter measures redmail_fravia+
redIs reverse engineering legal?

red(c) Fravia 1995, 1996, 1997, 1998, 1999. All rights reserved