Unfortunately, when I say bulk contents, I mean bulk contents; the current distribution contains nearly 20GB of raw data; the Organizational authority file adds another 1.7GB.
Since XML was deliberately designed for redundancy, XML encoded data sets are usually rather compressible. These data are no exception; bzip2 on the ARC data set achieves compression rations between 40:1 and 50:1. The 7-zip format does even better, but is not considered archival.
The real credit for this work belongs to the good people at NARA and their contributing agencies (and viewers like you). Don’t forget to visit the original pages on data.gov to give comments and feedback on this data, and don’t forget to show off all the cool UIs you build on top of this.
Mark Matienzo, (now a Yalie as of Monday), Richard Urban, GSLIS, University of Illinois,and Simon Spero, SILS, University of North Carolina have prepared a compressed distribution of these data sets, allowing much more efficient access to the data.
This distribution is available from iBiblio at this location; http://www.ibiblio.org/fred2.0/NARA/
One interesting datum that emerged during the compression process is that the compressed size of Part1 is nearly twice as large as most of the other Parts, despite having the same raw size. There are several possible explanations for this:
- H1: The records in Part1 are more complete than in other files.
- H2: Part1 describes record series from less related agencies than other files.
- H3: Part1 contains records with less compressible data (image types, numeric data, etc).
- H4: Part1 contains more, smaller, record series than other files.
If only had a time machine, I could do a poster and go to the iConference. Alas, all I have is the scarf and the Jelly babies, and I’ve eaten the Jelly babies.
This release is dedicated to the memory of Prof. Brian Vickery (1918-2009), whose passing was announced on Thursday the 28th.
Brian C. Vickery
Never too snarkaly
Helped refind science
See the Readme file for a bit more details.
THESE DATA SETS ARE NOT AUTHORITATIVE. This replica is being provided to allow compressed versions of the data set to be access to the NARA Archival Research Catalog data set. Once compressed versions of the data sets are available, it may go away without due process. For more information see the ARC home page at http://www.archives.gov/research/arc/ The original data can be found on the data.gov web site at the locations given below. The data.gov site will carry updated copies, and can be used to rate and submit comments. This directory contains compressed versions of the Archival Research Catalog (ARC) data set made available by NARA at http://www.data.gov/details/1357 . When complete, the dataset will contain ten files, named ArchivalDescriptionsPart1.tar.bz2 ... ArchivalDescriptionsPart10.tar.bz2 This directory also contains a compressed version of the Organization Authority File made available by NARA at http://www.data.gov/details/1358 This file identifies the names and administrative histories of Federal and non-Federal organizations used as access points in the archival descriptions. This data-set is stored in a single file, ARCWebExport15Dec2009_Org.tar.bz2 . DATE OF PREPARATION. The data sets were retrieved on or about the 28th January 2010. The source reports a last updated date of 15-Dec-2009. TECHNICAL NOTES This replica is being provided to allow compressed versions of the data set to be accessed. The uncompressed data is between 40-50 times larger than the compressed versions. Be careful when working with this dataset on shared file systems. The tar files contain a very large number of small files. Be careful when working with this dataset on hierarchical file systems. CREDIT The datasets were collected and prepared by the NARA and the contributing agencies, to whom all real credit belongs. This replica was prepared by Simon Spero, SILS, University of North Carolina, Mark Matienzo, University Libraries, Yale University, and Richard Urban, GSLIS, University of Illinois. Affiliation is listed for identification purposes only. This replica is hosted by iBiblio, and is transmitted using Z39.48 compliant acid free electrons.