From jan@swi.psy.uva.nl Mon Mar  4 10:51:34 2002
Received: from there (jan@gollem.swi.psy.uva.nl [145.18.152.30])
	by swi.psy.uva.nl (8.11.6/8.11.2) with SMTP id g249pYu29379;
	Mon, 4 Mar 2002 10:51:34 +0100 (MET)
Message-Id: <200203040951.g249pYu29379@swi.psy.uva.nl>
Content-Type: text/plain;
  charset="iso-8859-1"
From: Jan Wielemaker <jan@swi.psy.uva.nl>
Organization: SWI, University of Amsterdam
To: Bijan Parsia <bparsia@email.unc.edu>, prolog@swi.psy.uva.nl
Subject: Re: [SWIPL] Fetching files over HTTP
Date: Mon, 4 Mar 2002 10:46:14 +0100
X-Mailer: KMail [version 1.3.2]
References: <Pine.A41.4.21L1.0203011508360.36862-100000@login3.isis.unc.edu>
In-Reply-To: <Pine.A41.4.21L1.0203011508360.36862-100000@login3.isis.unc.edu>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit

On Friday 01 March 2002 21:11, Bijan Parsia wrote:
> Hello,
>
> I feel as if I must have asked this before :)
>
> What's the most straightforward way of fetching files over HTTP (well, in
> particular; other protocols like FTP would be welcome, too) in
> SWI-Prolog? I notice a fair bit of HTTP server code, but I'm having a bit
> of trouble finding client code.
>
> Assuming that there isn't anything bundled, any recommendations for
> libraries for this? I vaugely recall trying out Pillow (or some such) but
> it had real problems (as I recall) with modern domain names (virtual hosts
> in particular, I think).

There isn't a simple answer as you have seen.  There are issues such
as portability, range of protocols, completeness and robustness of
protocols, performance, handling errors, etc.

Basically, the options are:

	* Use a nice complete pure Prolog implementation.  Unless you
	  find it somewhere on the web, you'll have to make that yourself.
	
	* Use a partial Prolog implementation.  See below.

	* Use a nice complete library and connect it through the foreign
	  interface.  Options are the W3C networking library or possibly
	  Java components through JPL.
	
	* Use a commandline tool.  On Unix there is a wealth of these things
	  (lynx, wget, GET, ...) and good built-in support for reading from
  	  a pipe.  Most of these programs have been ported to Windows too,
    	  but a bit less ready available.  There is also no nice support
	  to read from the output of a process.

For partial Prolog implementations, there is (as you found), the XPCE
class http_client.  You can use this to fetch data and store it in some
XPCE object, such as a file, string or text_buffer.  This choice depends
largely on what to do next, and how much data you expect.  Here is an
example to store the result on a Prolog atom:

	url_to_atom(URL, Data) :-
		new(Client, http_client(URL)),
		new(S, string),
		send(Client, fetch_data, S),
		get(S, value, Data0),
		free(Client),
		free(S),
		Data = Data0.

Using pce_open(S, read, InStream) you can also make the data accessible
as a Prolog stream.

An alternative is to use http_client.pl from 
http://gollem.swi.psy.uva.nl/twiki/pl/bin/view/Library/PlHttp, which
provides support for GET and POST based directly on the library(socket).

Or, you can break the url into parts using library(url) and simply do
the job yourself using library(socket).  If you use the non-version
numbered HTTP protocol, this is really easy: open the port on the
host and send GET <URI> following by a newline.  Now read upto the
end of the input.  Unfortunately this doesn't handle virtual hosts
that may be present on the server.

	Cheers --- Jan






