NARA has released the bulk contents of it’s Archival Research Catalog on data.gov, in ARC XML format.

Unfortunately, when I say bulk contents, I mean bulk contents; the current distribution contains nearly 20GB of raw data; the Organizational authority file adds another 1.7GB.

Since XML was deliberately designed for redundancy, XML encoded data sets are usually rather compressible. These data are no exception; bzip2 on the ARC data set achieves compression rations  between 40:1 and 50:1.   The 7-zip format does even better, but is not considered archival.

The real credit for this work belongs to the good people at NARA and their contributing agencies (and viewers like you). Don’t forget to visit the original pages on data.gov to give comments and feedback on this data, and don’t forget to show off all the cool UIs you build on top of this.

Mark Matienzo, (now a Yalie as of  Monday),  Richard Urban,  GSLIS, University of Illinois,and Simon Spero, SILS, University of North Carolina have prepared a compressed distribution of these data sets, allowing much more efficient access to the data.

This distribution is available from iBiblio at this location; http://www.ibiblio.org/fred2.0/NARA/

One interesting datum that emerged during the compression process is that the compressed size of Part1 is nearly twice as large as most of the other Parts, despite having the same raw size.  There are several possible explanations for this:

  • H1: The records in Part1 are more complete than in other files.
  • H2: Part1 describes record series from less related agencies than other files.
  • H3: Part1 contains records with less compressible data (image types, numeric data, etc).
  • H4: Part1 contains more, smaller, record series than other files.

If  only had a time machine, I could do a poster and go to the iConference.  Alas, all I have is the scarf and the Jelly babies, and I’ve eaten the Jelly babies.

This release is dedicated to the memory of Prof. Brian Vickery (1918-2009), whose passing was announced on Thursday the 28th.

Higgledy-Piggledy
Brian C. Vickery
Ranganathoricly
Classified Life

Polyhierarchically
Never too snarkaly
Faceted artfully
Helped refind science

—–

See the Readme file for a bit more details.

THESE DATA SETS ARE NOT AUTHORITATIVE.

This replica is being provided to allow compressed versions of the data set
to be access to the NARA Archival Research Catalog data set. Once compressed
versions of the data sets are available, it may go away without due process.

For more information see the ARC home page at http://www.archives.gov/research/arc/

The original data can be found on the data.gov web site at the locations given
below.  The data.gov site will carry updated copies, and can be used to rate
and submit comments. 

This directory contains compressed versions of the Archival Research Catalog
(ARC) data set made available by NARA at
http://www.data.gov/details/1357 .

When complete, the dataset will contain ten files, named
ArchivalDescriptionsPart1.tar.bz2
...
ArchivalDescriptionsPart10.tar.bz2

This directory also contains a compressed version of the Organization Authority File made available by NARA at  http://www.data.gov/details/1358 

This file  identifies the names and administrative histories of Federal
and non-Federal organizations used as access points in the archival
descriptions.

This data-set is stored in a single file, ARCWebExport15Dec2009_Org.tar.bz2 .

DATE OF PREPARATION.

The data sets were retrieved on or about the 28th January 2010.
The source reports a last updated date of 15-Dec-2009.

TECHNICAL NOTES

This replica is being provided to allow compressed versions of the data set
to be accessed.  The uncompressed data is between 40-50 times larger than the
compressed versions. Be careful when working with this dataset on shared file
systems.  

The tar files contain a very large number of small files.  Be careful when
working with this dataset on hierarchical file systems.

CREDIT
The datasets were collected and prepared by the NARA and the contributing
agencies, to whom all real credit belongs.

This  replica was prepared by Simon Spero, SILS, University of
North Carolina, Mark Matienzo, University Libraries, Yale University,
and Richard Urban, GSLIS, University of Illinois. Affiliation is listed for
identification purposes only. 

This replica is hosted by iBiblio, and is transmitted using Z39.48 compliant
acid free electrons.

Leave a Reply