I’m happy to announce the availability of the new Tyler Hansbrough Special Edition of the NARA Archival Research Catalog. This special edition has fixes for the character set encoding problems noted below. These problems were being caused by problems in the UTF-8 encoding of various single and double quotes, and en- and em- dashes, resulting in unparsable XML documents.
Downloading the modified data sets
The following characters were affected – the counts given are for the number of times each character occurred, and not the number of documents each character occurred in.
|Unicode Name||Glyph||Bad Code||Fixed||Count|
|LEFT SINGLE QUOTATION MARK||‘||U+0018||U+2018||4|
|RIGHT SINGLE QUOTATION MARK||’||U+0019||U+2019||712|
|LEFT DOUBLE QUOTATION MARK||“||U+001C||U+201C||280|
|RIGHT DOUBLE QUOTATION MARK||”||U+001D||U+201D||316|
533 documents were affected.
A detailed list of which documents were affected, and which characters were replaced is available here.
Here’s what was going on:
<title>651.343 – Adult Education, 1939 – Workers Education, 1939</title>
If we look at the actual bytes in the file, we see that there’s a strange character after the “s” in “Workers”.
0000060 72 6b 65 72 73 19 20 45 64 75 63 61 74 69 6f 6e r k e r s 031 E d u c a t i o n
The hexadecimal value of this byte is 19; in UTF-8 and ASCII this is a control character used to mark the end of a paper tape. This is not a valid XML character (XML Recommendation § 2.2), which means that XML parsers won’t accept records with these characters in them.
<title>651.343 – Adult Education, 1939 – Workers’ Education, 1939</title>
Now we can see the saxon genitive.
0000200 6b 65 72 73 e2 80 99 20 45 64 75 63 61 74 69 6f k e r s ’ ** ** E d u c a t i o
The correct character code for Right Single Quotation Mark – ’ – in UNICODE is U+2019. The UTF-8 representation of this character is three bytes long – e2 80 99.
For more information on the joy of quotes, see the Wikipedia entry on Quotation mark glyphs.
«The Wonderful Thing About Standards Is There Are So Many To Choose From» – Adm. Hopper.