Subject: Re: [SWIPL] Web-whacking feasibility

From: "J.J.Green" <[email-masked]>
Date: Fri, 3 Aug 2001 15:59:29 +0100 (BST)

Hi Paul

> I need to build a "web whacker" to take a snapshot of a (finite) dynamic
> web site I'm working on.  Are most of the parts available to build this
> in SWI-Prolog?  I need to make HTTP GET calls, parse the returned HTML,
> extract all the linked-to URLs, identify those within the same site,
> avoid going around cycles, save the pages as HTML files with all local
> links replaced by 'file:' URLs.

Have you tried gnu wget? This does what you want, but
is coded in C -- might be worth a look for 
algorithms etc.

good luck

-j
--
J. J. Green,  Dept. Applied Mathematics, F40, Hicks Bld.,  
Hounsfield Rd.,  University of Sheffield,  Sheffield, UK. 
+44 (0114) 222 3742,           [email-masked], 
http://www.arbs.demon.co.uk  PGP http://www.keyserver.net