From p.singleton@keele.ac.uk Thu Aug  2 14:40:09 2001
Received: from mail1.svr.pol.co.uk (mail1.svr.pol.co.uk [195.92.193.18])
	by swi.psy.uva.nl (8.11.2/8.11.2) with ESMTP id f72Ce9b08622
	for <prolog@swi.psy.uva.nl>; Thu, 2 Aug 2001 14:40:09 +0200 (MET DST)
Received: from modem-203.potassium.dialup.pol.co.uk ([62.136.18.203] helo=keele.ac.uk)
	by mail1.svr.pol.co.uk with esmtp (Exim 3.13 #0)
	id 15SHlm-0004am-00
	for prolog@swi.psy.uva.nl; Thu, 02 Aug 2001 13:40:07 +0100
Message-ID: <3B692992.3AEB8464@keele.ac.uk>
Date: Thu, 02 Aug 2001 11:21:06 +0100
From: Paul Singleton <p.singleton@keele.ac.uk>
Organization: SmartArts Computing Consultancy
X-Mailer: Mozilla 4.74 [en] (WinNT; U)
X-Accept-Language: en
MIME-Version: 1.0
To: SWI Prolog <prolog@swi.psy.uva.nl>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Subject: [SWIPL] Web-whacking feasibility

I need to build a "web whacker" to take a snapshot of a (finite) dynamic
web site I'm working on.  Are most of the parts available to build this
in SWI-Prolog?  I need to make HTTP GET calls, parse the returned HTML,
extract all the linked-to URLs, identify those within the same site,
avoid going around cycles, save the pages as HTML files with all local
links replaced by 'file:' URLs.

(I also seek reassurance that I haven't seriously underestimated the
complexity of the problem :-)

Paul Singleton


