IAFA-WG Publishing Information on the Internet with Anonymous FTP (IAFA DOC II) DRAFT 92.06.10 Over the past several years, Anonymous FTP has become the primary method of publishing information in the Internet environment. Anonymous FTP is an application-level service that makes use of the File Transfer Protocol [1], one of the principal protocols of the TCP/IP suite. A well organised and well maintained Anonymous FTP archive (AFA) can provide a cheap and simple way to distribute the programs, documents and datasets that are produced for general availability on the network today. Those groups wishing to set up an Anonymous FTP Archive should refer to RFC [* IAFA Doc I *] , "A Guide to Anonymous FTP Site Administration" which provides details on why one would want to set up such an archive and what steps are required to have a secure, well maintained system. This document recommends a set of information that your site may wish to construct and make available on the Anonymous FTP Archive to the user community. Automatic archive indexing tools have been created that gather and index this information, thus making it easier for users to find and access it. It also may be used by the general user community for extracting information about the archive itself or about material contained on the archive. Although not required, providing such information will make your archive a more useful resource. In Part I of this document we make recommendations that are intended to provide a standardized means for sharing information about the contents of a specific archive site. These include information concerning such things as services provided by the institution, document abstracts, software descriptions as well as administrative contacts, local Timezone and other site-specific details. Part II contains a set of encoding procedures for the information outlined in Part I. These procedures allow the you, the AFA administrator, to take into account site-specific issues such as whether your particular operating system offers the capability of creating and using subdirectories, any limitations on filename length or the inability to use specific characters in filenames. Part I: Useful Configuration and Contents Information In this section we define a recommended set of information that you could make available as the administrator of an archive site. In doing so, you would extend the functionality of your archive, as well as the functionality of the indexing and resource discovery tools that pick up and redistribute such information. It is expected that this information will be made available through the anonymous FTP archive mechanism. The actual encoding method used will be host and OS specific. Encoding methods for some of the most popular computing environments are presented in Part II. Note that this document does not mandate or require that any particular class of information be offered. However it is recommended that those sites wishing to offer the information described here adhere to the formats recommended in Part II. Collectively the files described in this document will be referred to as "indexing files". Site-Specific Configuration Information --------------------------------------- Information about your archive site itself can often be valuable to users of your system in order for them to utilize the resource in an efficient manner. Desctiption Information ----------------------- - You should make available a brief description of the kind of information stored in your anonymous FTP archive. If your site specializes in a particular type of information you should mention this. Examples might include software for a specific machine type, on-line copies of particular types of literature or research papers or information covering a particular branch of science or engineering. - You should also make available a set of suitable keywords appropriate to the above description to aid in the creation of suitable indexes. Configuration Information ------------------------- Site configuration information will help users better understand your wishes on how and when to access your site. This would include such information as: Contact: - The name of the site - The name of the organization or group owning the site - The name of the person responsible for administering the site - The type of the organization - The postal address - The telephone number - Email address of person or persons responsible for site administration. - The location of the site by city, state, country - The geographical (latitude/longitude) location - The timezone of site - A brief description of the kind of information stored at this anonymous FTP archive. If your site is intended to specialize in a particular type of information (examples might include software for a specific machine type, on-line copies of a particular type of literature or research papers and information in a particular branch of science or engineering) you should indicate this. - A summary of the access policies of this site. This should include such information as preferred times of usage, conventions or restrictions for uploading files to this site etc. Logical Configuration --------------------- Note that one physical archive site may possibly contain multiple "logical" archives. For example, a single archive host may be shared among multiple departments, each responsible for the administration their own part of the anonymous FTP directory subtree. Some information (such as a host's location) will remain constant for the site as a whole. We therefore recommend that you to list Logical Archive specific and site-specific information separately. Logical Archive configuration: - A summary of the access policies of this logical archive - A summary of the type of information that this site may specialize in. - The name of the organization or group owning logical archive - The name of the person responsible for administering logcial archive - The type of the organization - The postal address of primary contact - The telephone number of primary contact - Email address of primary contact Site-Specific Content Information --------------------------------- The preceeding collections of information make available access and utilization policies for a site. You could also wish to make available a selection of information about the actual contents of your archive or the services available from your organization or institution. The following categories have been identified. Services: - The archive can offer an overall description of each the various Internet services offered by your organization's systems, along with corresponding contact information. This description would then indicate whether the the parent organization offers such services as: - on-line library catalogues - Interactive online information services such as WAIS [3], gopher [4], Prospero [5], WWW [6] or archie [7] - specialized information servers such as those providing weather, geographic information, newswire feeds etc. - Other information services The following information can be made available: - Name of service - Host providing service - A brief description of the service - The required access protocol (telnet, FTP, Prospero etc) - Keywords Document abstracts: - You might wish to make available a brief description of documents contained in the archive. This might correspond to the actual abstract for each technical report or other document served from this archive site or simply be a brief description created to help potential readers. All such documents offered should have a corresponding abstract made available for it. - Filename or directory name of the document - Title of the document - Filename of the document - Name(s) of author(s) - Last revision date - Scope (technical report, conference paper etc) - The summary or abstract of the document being referenced - Appropriate keywords - Format that the document is stored in (ASCII text, PostScript, DVI etc.) - Publication status (draft, published etc.) - Document size [* length ? *] Datasets: - You should offer adescription of any datasets stored on the archive. These could include star catalogs, DNA sequences, census statistics etc. - Name of the dataset - Title - Version number or string of the dataset - Date last revised for dataset - Source of data - Name of individual or group responsible for compilation - Size of dataset - Format of data (special record format name etc.) - Programs used to manipulate the data in the set Software Packages: - Outlines for each of the program packages offered at this site. - Name of file containing the package - Title of package - Version of the package - Description of the function of package - Name of author - Package maintainer - Package origin (original site, copied) - Special considerations or restrictions on the package's use (GNU copyleft, hardware restrictions, etc). - The copying policy (Public Domain, Freely Redistributable) - Keywords appropriate for the package Mailing Lists: - Publicly available mailing lists are maintained by this organization or institution. It is assumed that only lists admitting general subscriptions will be listed. - Name of mailing list - Description of list (function) - Email address of list - Email address of administrative contact - Archives of list (if any) - Keywords appropriate for describing the function of the list [* Some work needs to be done on this next one *] Complete File listing: - A listing of all archive entries at this archive in format appropriate for that environment. Such listings, if properly maintained, reduce network traffic while simplifying the task of archive indexing services. [* The following is a new catagory added at the suggestion of Markus Stumpf *] Usenet Newsgroup Archives - Information concerning Usenet newgroups that are archive at your site and relevant contact details. Part II: Information Encoding for Specific Environments ------------------------------------------------------- In this section we offer a recommended encoding format for each of the standard items of information suggested in Part I. In many cases these recommendations should be applicable to all environments. Where this is not true standardized encodings are offered for specific operating systems. We offer such a standardized format so that if such information _is_ to be offered, it is formatted in such a way that it can be utilized by automated indexing and retrieval tools. The encoding methods proposed were developed to be extensible, so that additional information can be offered in a similar format, if the site administrator so wishes. Developing such recommendations offers several challenges. It is hoped that the encoding conventions should be applicable to as wide a variety of operating systems, file structures and encoding schemes as possible. In addition, the globalization of the Internet requires attention to constraints such as the language in use at an archive site. In addition, the encoding methods proposed must be easy to implement and for the moment, use existing methods of access and retrieval. We currently assume that the site language is English. It is assumed that additional formats for other languages will be developed over time. Naming conventions ------------------ For the greatest flexibility it is assumed that unless otherwise stated, each file containing the indexing information may reside anywhere in the anonymous FTP subtree and in addition, any number of these files may exist. The intention here is that they may reside in the same location as the information they are indexing. You, as the administrator are free to place these files wherever you think appropriate. Note that some files may carry information from their place in the directory structure and therefore they may not just be randomly placed in the file structure. For example, all other indexing files could be placed in one directory. A file of the given name will exist for each category listed in Part I. For the sake of consistency across operating systems and for the ability to distinguish them from non-configuration files of the same name, the filenames will be in all uppercase letters. Because of restrictions on some systems, filenames will be kept to a maximum of 14 characters. In order for tools to easily identify an indexing file from the other data files at the archive site, all indexing filenames should begin with the five character string "IAFA-". Files that may contain multiple instances of a given category (Mailing Lists for example) will logically be divided into records and each record containing multiple fields. Unless otherwise specified, files may contain one or multiple records as defined below and multiple records are separated by one or more blank lines. The start of each field is marked by a special fieldname on a new line in the rightmost column followed by a colon (:). Field data may be separated from fieldname by whitespace. Any field may continue on the next line by whitespace (blank, tab) in the first column. An Encoding for UNIX Systems --------------------------- Indexing files should be made world readable. It is assumed that size and last modification times can be obtained through the existing FTP mechanism. The advantages to this system are that this information need only be constructed once with infrequent periodic updates as changes occur. Many of these files may never change during the lifetime of the host as an anonymous FTP site. They require no special programs or protocols to construct: a text editor is all that is needed. Site Description ---------------- Filename: IAFA-DESCRIPT This file contains 1 record with the following fields. IMPORTANT: There should only be one instance of this file in each archive. Description: This file contains text describing any areas of specialization for this site. For example, if the site contains information related to the filed of molecular biology a paragraph or two with the keywords "molecular biology" and some further description would be in order. It should also mention if this site contains "logical" archives. Keywords: Appropriate keywords describing contents of this AFA Example for IAFA-DESCRIPT file. Description: This site contains data relating to DNA sequencing particularly Yeast chromosome 1. Datasets are available. There is also a selection of programs available for manipulating this information. Keywords: DNA, sequencing, yeast, genome, chromosome Physical Site Information ------------------------- This file contains one (1) record with the following fields. IMPORTANT: There should only be one instance of this file in each archive. Filename: IAFA-SITEINFO Fields for this file. Name: Primary DNS name Cname: Preferred DNS-registered canonical name for the site Postal-Address: The postal address of the site Telephone: The telephone number of the site. Should be in international format and including the country code Organization: Name of institution/organization/individual to which the site belongs Organization-Type: Type of organization (univeristy, commercial, research) Electronic-Address: Email address in RFC 822 format for the AFA administrator (See Note <1>) Contact: Name of person or group responsible for AFA administration Location: City, State and Country of the site Latitude-Longitude: Latitude and longitude of site (See Note <2>) Timezone: Timezone as hours and minutes from UTC (See Note <3>) Written-by: Name of person writing this file (See Note <4>) Frequency: Preferred frequency of retrieval of all AFA extended configuration information by automated retrieval tools (See Note <5>) Access-times: Period of preferred times of access to anonymous FTP users in UTC. (See Note <6>) Policy: Information such as conventions or restrictions for uploading files to this site etc. Notes for this file. <1> Email addresses must be in RFC 822 format. Names may be included in the Email address. For example: "John Doe" or jd@ftp.bar.org (John X. Doe) are valid Email addresses. <2> Latitude and longitude are specified in that order as DD MM SS C / DD MM SS C Where DD is in degrees MM is in minutes SS is in seconds C is the direction designator which is For latitude "+" is east of the Greenwich meridian "-" is west of the Greenwich meridian For longitude "+" is north of the equator "-" is south of the equator The double quotes (") are not part of the designator, but are used here to delimit the symbols. <3> Timezone is specified in hours and minutes from UTC (GMT). Specified as DHHMM Where D is one of "+" is east of the Greenwich meridian "-" is west of the Greenwich meridian HH is hours from UTC MM is minutes from UTC <4> Email address and date of document composition may be included in this field in free format. <5> The period is measured in days. This value should be chosen to reflect the turnover of information at the archive. <6> Times in UTC between which access to this site is preferred. This takes the format HHMM / HHMM Where HH is in hours MM is in minutes The first HHMM is starting time, the second the ending time. An example of a IAFA-CONTACT file: Name: foo.bar.org Cname: ftp.bar.org Organization: Beyond All Recognition Foundation Contact: John Doe Electronic-Address: FTP@bar.org Telephone: +1 215 555 1212 Postal-Address: PO Box. 6977, Marinetown, PA 17602 Location: Lampeter, PA, USA Latitude-Longitude: 37 24 43 - / 121 58 54 + Timezone: -0400 (Eastern Standard Time) Written-by: jd@bar.org (John X. Doe); Mon Feb 10 22:43:31 PST 1992 Frequency: 10 Access-Times: 0200 / 1300 Policy: Non-proprietary data may be uploaded to this site in the "incoming" directory. Please contact site administrators if you do so. Proprietary material found in this directory will be removed. This site is not to be used as a temporary storage area. Logical Archive Info -------------------- Filename: IAFA-LARCHIVE IMPORTANT: The placement of this file in the file structure is significant: It implies that the directory in which this file exists and all subdirectories are part of the logical archive. Any number of these files may exist in the archive. Policy: Information such as conventions or restrictions for uploading files to this site etc. Description: Contains text describing any field or area of specialization that the site subscribes to. Keywords: Appropriate keywords describing contents of this logical AFA Organization: Name of institution/organization/individual to which the site belongs Organization-Type: Type of organization (univeristy, commercial, research) Contact: Name of person or group responsible for AFA administration Postal-Address: The postal address of the site Telephone: The telephone number of the site. Should be in international format and including the country code Electronic-Address: Email address in RFC 822 format for the AFA administrator (See Note <1>) Notes for this file. <1> Email addresses must be in RFC 822 format. Names may be included in the Email address. For example: "John Doe" or jd@ftp.bar.org (John X. Doe) are valid Email addresses. For the following categories the assumption shouldn't be made that the information applies to the anonymous FTP host itself. Rather, the group or organization may publish general information: the specific information will be contained inside the file describing the category. Services information -------------------- Filename: IAFA-SERVICES This file contains records with the following fields. Each record is started and delimited by the "Service-Name" field. Any number of these files may exist in the archive. Service-Name: Name of the service (See Note <1>) Description: Short description of the service provided Access-protocol: Method required to access service (See Note <2>) Keywords: Keywords appropriate for the service Notes on this file. <1> This can be a generic name such as "NNTP" or "WAIS" or something more specific such as "Geographic Name Server" <2> A description of how the service is to be accessed. This may be as simple as "Email" or "telnet to port 201" or more complex such as "Prospero protocol on port 5678" Example of IAFA-SERVICES file. Service-Name: Census Bureau information server Hostname: census.foo.com (127.0.0.2) Description: This server provides information from the latest USA Census Bureau statistics (1990). Access-protocol: telnet protocol to port 3000. A server-specific query language is used. Type "help" for more information. Keywords: census, population, 1990, statistics Document Abstracts information ------------------------------ Filename: IAFA-ABSTRACTS This file contains records with the following fields. Each record is started and the previous record is delimited by the "Document-Name" field. Any number of these files may exist in the archive. Any number of these files may exist in the archive. Document-Name: Filename containing the document Title: Title of the document Authors: Name of authors (See Note <1>) Revision-Date: Last date that document was revised Category: Type of document. (See Note <2>) Abstract: Summary of the document Format: Format or formats in which the document is available (See Note <3>) Citation: The official bibliographic entry for the document Publication-Status: Current status of document (draft, published etc) Keywords: Keywords relevant to the document Size: Length of document in pages Notes for this file. <1> The names of the individuals or group appearing on the document as authors. Names can be separated by a semicolon. RFC 822 Email address for each author should be included where appropriate. addresses may also be included where appropriate. For example Authors: Alan Emtage ; Peter Deutsch 805 Sherbrooke W., Rm 222, Montreal, Quebec CANADA H3A 2K6 <2> The intention of this field is to define the category of the document. It can be "Technical Report", or perhaps the name and date of the conference at which the paper was presented. It may also be something like "General guide" or "User manual" <3> Documents are often available in several formats. Examples include "PostScript", "ASCII text", "DVI" etc. Example of IAFA-ABSTRACTS file. Document-Name: yeast-homeobox Title: The function of homeoboxes in Yeast Chromosome 1 Authors: John Doe jdoe@yeast.foobar.com; Jane Buck jane@fungus.newu.edu Revision-Date: 25 November 1991 Category: Yeastcon, January 1992, San Francisco Abstract: Homeoboxes have been shown to have a significant impact on the expressions of genes in Chromosome 1 of bakers yeast. This paper surveys this impact. Format: PostScript, ASCII (without graphs) Citation: J. Doe, J. Buck, The function of homeoboxes in Yeast Chromosome 1, Conf. proc. Yeastcon, January 1992, San Francisco, pp. 33-50 Keywords: yeast, chromosome, DNA, sequencing Publication-Status: Published Size: 18 pages Mailing List Information ------------------------ Filename: IAFA-MAILLISTS This file contains records with the following fields. New records are marked and delimited by the "Mailinglist-Name" field. Any number of these files may exist in the archive. Mailinglist-Name: The name of the list Address: The address (in RFC 822 format) that mail intended for this list should be sent to. Administration: The address (in RFC 822 format) of the administrative contact for the list. Additions and deletions to the list as well as questions about the list should be directed to this address. Description: A description of the purpose of the list. Any special conditions for the list should be included. Keywords: Keywords useful for anyone looking for the list Archive: Location and access method for any archive for this mailing list. Example of the IAFA-MAILLISTS file. Mailinglist-Name: Internet Engineering Task Force (IETF) Internet Anonymous FTP Archive working group (IAFA-WG) mailing list Address: iafa@cc.mcgill.ca Administration: iafa-request@cc.mcgill.ca Description: Discussion list for the IAFA Working Group concerning the administration of anonymous FTP archive sites. Keywords: IETF, IAFA, anonymous, FTP, archive, Internet Archive: The archive for this mailing list is available on archive.cc.mcgill.ca via anonymous FTP in the file pub/mailing-lists/iafa Software Packages Information ----------------------------- Filename: IAFA-PACKAGES This file contains records with the following fields. The record is started and delimited from other records by the "Package-Name" field. Any number of these files may exist in the archive. Package-Name: Name of the file or directory containing the package Title: Title of the package Version: This field can be used if a version number or string is associated with the package Description: Description of the function of the programs in the package Author: Name and Email address of authors if available Maintained-by: This field should be included when the current maintainer of the package is known. Contact information should be included Maintained-at: Host name of the "home" of the package if known. This is the site at which the most uptodate version of the package would be expected to be found Platforms: Any requirements or restrictions that the package may have in terms of hardware or software (OS) platforms. The programming language the package is written in should be included. Copying-Policy: The status of the package for copying purposes. (See Note <2>) Keywords: Keywords appropriate for users trying to locate the package Notes on this file. <1> The most common entries for this field would be "Public Domain" or "Freely Redistributable" or "Voluntary Payment" (shareware). However since a record may exist for software packages not resident on the AFA, this field may be entered as "Proprietary" or some other form of restricted access Example record for the IAFA-PACKAGES file. Package-Name: xarchie.tar.Z Title: xarchie Version: 1.3 Description: This program provides and X11 interface to the archie database. It allows the user to locate and retrieve files found on anonymous FTP archive sites around the world. Author: George Ferguson (ferguson@cs.rochester.edu) Maintained-by: George Ferguson (ferguson@cs.rochester.edu) Maintained-at: FTP.cs.rochester.edu Platforms: X11 based program written in C. Known to run on Sun 3's and Sun 4's under SunOS 3.X and 4.X Copying-policy: Freely Redistributable. Copyright held by author. Keywords: archie, X11, anonymous FTP archive, software location Datasets Information -------------------- Filename: IAFA-DATASETS This file contains records with the following fields. The record is started and delimited by the "Dataset-Name" field. Any number of these files may exist in the archive. Dataset-Name: Name of the file or directory containing the dataset Title: Title of the dataset. Version: Version number can be used if one is associated with the dataset Revision-Date: The date of last revision of the dataset Source: The group or organization providing the source for the dataset. Email or postal addresses should be included where possible Compiled-by: The group or organization responsible for compiling the dataset into the format for which this description applies. Email or postal addresses should be included where possible Size: Size of the dataset. (See Note <1>) Format: The format in which the dataset is distributed. (See Note <2>) Software: A list of any programs used to manipulate the dataset. Contact names and addresses (Email,postal) should be included where possible. Notes on this file. <1> This information should be in well-known units such as octets (bytes). Alternatively the number of records in the dataset and the record size may be given. <2> The format may be well-known or specific to a specific dataset. Additional information on programs used with this dataset should be provided in the "Software" field. [* Example of the DATASETS file. *] Listing Information ------------------- Filename: IAFA-LISTINGS Only one instance of this file should exist in the archive. This file differs from the others in that it has no defined fields. For UNIX sites this file should contain a long recursive listing (ls -lR) from the directory in which the anonymous FTP client would find itself on initial login. The file may be compressed by the UNIX compress(1) format. Any anonymous FTP retrieve of this file should be, by default in binary mode to accommodate the case of the file being compressed. This file should be automatically generated on a frequent basis, depending on how often the files at your site change. Compressing the file lessens the load on the network since less traffic has to flow and it lessens the load on the anonymous FTP archive host since the administrators can determine when it is most convenient for the program to run. USENET News Information ----------------------- Filename: NEWSGROUPS This file contains records with the following fields. New records are marked and delimited by the "Newsgroup-Name" field. Newsgroup-Name: The name of the newsgroup Mailinglist-Gate: The address (in RFC 822 format) of the mailing list this newsgroup is gatewayed to (if any) Description: A description of the purpose of the newsgroup as found in List_of_Active_Newsgroups regularly posted in news.announce.newusers. Keywords: Keywords useful for anyone looking for the newsgroup Archive: Location and access method for the archive for this newsgroup. Example of the NEWSGROUPS file. Newsgroup-Name: comp.windows.x Mailinglist-Gate: xpert@expo.lcs.mit.edu Description: Discussion about the X Window System Keywords: X, X11, X Window System, Xlib, Xt, Xaw Archive: Selected articles are archived in /pub/news/comp.windows.x/volumeYY/NN-MMM, where YY is the year (e.g. 91), NN is the number of the month (e.g. 03) and MMM is the name of the month (e.g. Mar). This gives something like volume91/03-Mar. ---------------------------------------------------------------------- Bibliography [* To be fixed up *] ------------ [1] RFC 959 Postel, J.B.; Reynolds, J.K. File Transfer Protocol. 1985 October [2] RFC1296 Lottor, M. Internet Growth (1981-1991). 1992 January;