From j.j.green@sheffield.ac.uk Fri Aug  3 16:59:27 2001
Received: from mailhub2.shef.ac.uk (mailhub2.shef.ac.uk [143.167.2.154])
	by swi.psy.uva.nl (8.11.2/8.11.2) with ESMTP id f73ExQb01277
	for <prolog@swi.psy.uva.nl>; Fri, 3 Aug 2001 16:59:26 +0200 (MET DST)
Received: from gold.shef.ac.uk ([143.167.2.5])
	by mailhub2.shef.ac.uk with esmtp (Exim 3.22 #10)
	id 15SgQD-0004iC-00; Fri, 03 Aug 2001 15:59:29 +0100
Received: from ap1jjg (helo=localhost)
	by gold.shef.ac.uk with local-esmtp (Exim 3.14 #1)
	id 15SgQD-0003Ns-00; Fri, 03 Aug 2001 15:59:29 +0100
Date: Fri, 3 Aug 2001 15:59:29 +0100 (BST)
From: "J.J.Green" <J.J.Green@sheffield.ac.uk>
X-Sender: ap1jjg@gold
To: Paul Singleton <p.singleton@keele.ac.uk>
cc: SWI Prolog <prolog@swi.psy.uva.nl>
Subject: Re: [SWIPL] Web-whacking feasibility
In-Reply-To: <3B692992.3AEB8464@keele.ac.uk>
Message-ID: <Pine.GSO.4.10.10108031556510.21259-100000@gold>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII

Hi Paul

> I need to build a "web whacker" to take a snapshot of a (finite) dynamic
> web site I'm working on.  Are most of the parts available to build this
> in SWI-Prolog?  I need to make HTTP GET calls, parse the returned HTML,
> extract all the linked-to URLs, identify those within the same site,
> avoid going around cycles, save the pages as HTML files with all local
> links replaced by 'file:' URLs.

Have you tried gnu wget? This does what you want, but
is coded in C -- might be worth a look for 
algorithms etc.

good luck

-j
--
J. J. Green,  Dept. Applied Mathematics, F40, Hicks Bld.,  
Hounsfield Rd.,  University of Sheffield,  Sheffield, UK. 
+44 (0114) 222 3742,           j.j.green@sheffield.ac.uk, 
http://www.arbs.demon.co.uk  PGP http://www.keyserver.net


