$Id$

Web Crawler                                     Kirthika Parameswaran
-----------

The Web Crawler follows the  HTTP_1.1 protocol.

This Crawler crawls in  either FIFO or LIFO order over the URLs
now stored in a ACE_Unbounded_Queue. The Command Processor pattern is
used in this example.

Also the auto-purging feature where connections are removed from the cache
when the process runs out of file descriptors, is added to this example.

[Use the -l option to set the handle limit].

Run:
---


> make

> main -r -u www.cs.wustl.edu/~kirthika/test.html -o LIFO

or

> main -r -u www.cs.wustl.edu/~kirthika/test.html -o FIFO
