I’m happy to announce the availability of the new  Tyler Hansbrough Special Edition of the NARA Archival Research Catalog. This special edition has fixes for the  character set encoding problems noted below. These problems were being caused by problems in the  UTF-8 encoding of   various single and double quotes, and en- and em- dashes, resulting in unparsable XML documents.

Downloading the modified data sets

The  Tyler Hansbrough special edition can be downloaded from the Fred 2.0 repository on iBiblio.

Affected characters

The following characters were affected – the counts given are for the number of times each character occurred, and not the number of documents each character occurred in.

[table id=1 /]

Affected documents

533 documents were affected.

A detailed list of which documents were affected, and which characters were replaced is available here.

Technical details

Here’s what was going on:

Before

<title>651.343 – Adult Education, 1939 – Workers Education, 1939</title>

If we look at the actual bytes in the file, we see that there’s a strange character after the “s” in “Workers”.

0000060    72  6b  65  72  73  19  20  45  64  75  63  61  74  69  6f  6e
           r   k   e   r   s  031      E   d   u   c   a   t   i   o   n

The hexadecimal value of this byte is 19; in UTF-8 and ASCII this is a control character used to mark the end of a paper tape. This is not a valid XML character (XML Recommendation § 2.2), which means that XML parsers won’t accept records with these characters in them.

After

<title>651.343 – Adult Education, 1939 – Workers’ Education, 1939</title>

Now we can see the saxon genitive.

0000200    6b  65  72  73  e2  80  99  20  45  64  75  63  61  74  69  6f
           k   e   r   s   ’   **  **      E   d   u   c   a   t   i   o

The correct character code for Right Single Quotation Mark – – in UNICODE is U+2019. The UTF-8 representation of this character is three bytes long – e2 80 99.

For more information on the joy of quotes, see the Wikipedia entry on Quotation mark glyphs.

«The Wonderful Thing About Standards Is There Are So Many To Choose From» – Adm. Hopper.


Leave a Reply