Posts Tagged “ARC”

Damned Yankees


<display-name>New York World's Fair (1964-1965)</display-name>

Good ol’ Boys


<title>Cleveland-Harrison (1885-1893): South Carolina – Worlds Fair</title>

Comments No Comments »

Whilst checking up on some of the character set fixes, I’ve noticed that there seem to be some inconsistencies in the use of right quotes, apostrophes, and accute accents, which may lead to problems down the road when doing searches, or performing name authority work.

Here are some examples that may make things clearer.

Use of a right single quote as an accute accent.

In  allRecords/ArchivalDescriptions/68/FileUnit_744305.xml we find the following title:

<title>Navarro, Jose’ Antonio</title>

There are several ways to represent é in UNICODE; composed, decompsed, or as an e followed by a non-combining accent  (U+00B4); none of these approaches will match this title string.

One correct form is:

<title>Navarro, José Antonio</title>

(see e.g.

Some accute accents are encoded correctly

For example, in allRecords/ArchivalDescriptions/1/Item_197118.xml we find:

<display-name>Québec (Québec province, Canada, North and Central America) inhabited place</display-name>

Vive Jay Sherman! Vive Québec

Comments No Comments »

I’m happy to announce the availability of the new  Tyler Hansbrough Special Edition of the NARA Archival Research Catalog. This special edition has fixes for the  character set encoding problems noted below. These problems were being caused by problems in the  UTF-8 encoding of   various single and double quotes, and en- and em- dashes, resulting in unparsable XML documents.

Downloading the modified data sets

The  Tyler Hansbrough special edition can be downloaded from the Fred 2.0 repository on iBiblio.

Affected characters

The following characters were affected – the counts given are for the number of times each character occurred, and not the number of documents each character occurred in.

[table id=1 /]

Affected documents

533 documents were affected.

A detailed list of which documents were affected, and which characters were replaced is available here.
Read the rest of this entry »

Comments No Comments »

There are a number of problems in the ARC data files that are caused by truncation of 16 bit unicode punctuation to 8-bits. There may be other undetected truncation errors where the truncated value is a legitimate UTF character, but this seems unlikely.

I have written code to work around this problem; it’s a fairly simple fix.

There are also about a dozen records that are ill-formed- the contain tags generated by an Oracle xml sql engine.

Scary thought for the day:
Imagine if Oracle made an ILS

Comments No Comments »

NARA has released the bulk contents of it’s Archival Research Catalog on, in ARC XML format.

Unfortunately, when I say bulk contents, I mean bulk contents; the current distribution contains nearly 20GB of raw data; the Organizational authority file adds another 1.7GB.

Since XML was deliberately designed for redundancy, XML encoded data sets are usually rather compressible. These data are no exception; bzip2 on the ARC data set achieves compression rations  between 40:1 and 50:1.   The 7-zip format does even better, but is not considered archival.

The real credit for this work belongs to the good people at NARA and their contributing agencies (and viewers like you). Don’t forget to visit the original pages on to give comments and feedback on this data, and don’t forget to show off all the cool UIs you build on top of this.

Mark Matienzo, (now a Yalie as of  Monday),  Richard Urban,  GSLIS, University of Illinois,and Simon Spero, SILS, University of North Carolina have prepared a compressed distribution of these data sets, allowing much more efficient access to the data.

This distribution is available from iBiblio at this location;

One interesting datum that emerged during the compression process is that the compressed size of Part1 is nearly twice as large as most of the other Parts, despite having the same raw size.  There are several possible explanations for this:

  • H1: The records in Part1 are more complete than in other files.
  • H2: Part1 describes record series from less related agencies than other files.
  • H3: Part1 contains records with less compressible data (image types, numeric data, etc).
  • H4: Part1 contains more, smaller, record series than other files.

If  only had a time machine, I could do a poster and go to the iConference.  Alas, all I have is the scarf and the Jelly babies, and I’ve eaten the Jelly babies.

This release is dedicated to the memory of Prof. Brian Vickery (1918-2009), whose passing was announced on Thursday the 28th.

Brian C. Vickery
Classified Life

Never too snarkaly
Faceted artfully
Helped refind science


See the Readme file for a bit more details.

Read the rest of this entry »

Comments No Comments »