NARA has released the bulk contents of it’s Archival Research Catalog on data.gov, in ARC XML format.
Unfortunately, when I say bulk contents, I mean bulk contents; the current distribution contains nearly 20GB of raw data; the Organizational authority file adds another 1.7GB.
Since XML was deliberately designed for redundancy, XML encoded data sets are usually rather compressible. These data are no exception; bzip2 on the ARC data set achieves compression rations between 40:1 and 50:1. The 7-zip format does even better, but is not considered archival.
The real credit for this work belongs to the good people at NARA and their contributing agencies (and viewers like you). Don’t forget to visit the original pages on data.gov to give comments and feedback on this data, and don’t forget to show off all the cool UIs you build on top of this.
Mark Matienzo, (now a Yalie as of Monday), Richard Urban, GSLIS, University of Illinois,and Simon Spero, SILS, University of North Carolina have prepared a compressed distribution of these data sets, allowing much more efficient access to the data.
This distribution is available from iBiblio at this location; http://www.ibiblio.org/fred2.0/NARA/
One interesting datum that emerged during the compression process is that the compressed size of Part1 is nearly twice as large as most of the other Parts, despite having the same raw size. There are several possible explanations for this:
- H1: The records in Part1 are more complete than in other files.
- H2: Part1 describes record series from less related agencies than other files.
- H3: Part1 contains records with less compressible data (image types, numeric data, etc).
- H4: Part1 contains more, smaller, record series than other files.
If only had a time machine, I could do a poster and go to the iConference. Alas, all I have is the scarf and the Jelly babies, and I’ve eaten the Jelly babies.
This release is dedicated to the memory of Prof. Brian Vickery (1918-2009), whose passing was announced on Thursday the 28th.
Brian C. Vickery
Never too snarkaly
Helped refind science
See the Readme file for a bit more details.
Read the rest of this entry »