Archive for the “Metadata” Category

Jeff Beall reposts some of the discussion on the  MODS mailing list on the recently published draft of MADS/RDF.  Beall wonders if the proposal, and the initial comments indicate that the entire semantic web effort is doomed. This may be an over-reaction.

I spent some time working on a formal analysis of the semantics of  pre-combined[1]/subdivided subject headings, and how they could be added in a downwards compatible way to SKOS. I had hoped to discuss the topic at the DC VOCAMP held at The Library  immediately after ISWC 2009, but there was no-one at the camp who was interested in library vocabularies, I let the matter drop[2].

Since support for pre-combined headings are one of the  major design goals of  MADS/RDF, I have been looking quite closely at the initial  draft.  I am not yet ready to publish detailed results, but I wanted to take this chance to make a few quick notes on the initial draft.

Minor Issue

There is a quick change that could make it easier for others to evaluate the proposed ontology.  Currently, the URI/IRI of the ontology is “″.  At the moment does not resolve as a URL.  Also, the name is slightly different from current usual practice.  Ontology URIs are typically of the form …/<Year>/<Month>/<Name>; the MADS IRI is in the form …/<Name>/<Year>/<Month>. This encourages some common tools, such as Protege, to abbreviate the ontology name as “11″.

Major issues

  1. The SKOS mappings are imported as part of the reference ontology; however due to problems in naming, this may not show up correctly.  It might be better to include the axioms in the mapping file directly in the main ontology.
  2. The SKOS mappings are incomplete; it is possible that concentrating  on narrowing the differences between SKOS and Standard(NISO Z39.19)/Library KOS semantics would give an end result that should be more useful for use by The Library.
  3. The current draft was created without support from an ontologist or owl expert. There are several problems in the modeling that result from this lack of resourcing.  For example, there are several places where a constraint or restriction is given in a comment that could readily have been represented in OWL, and many properties are not fully or properly specified.  There is a big difference between the ontology design and schema design, and without support resources, it is easy to fall in to some common anti-patterns.
  4. The current draft was created without ready access to Subject Matter Experts (SMEs) in the Cataloging COI.  As a result, the ontology may not adequately represent the problem domain, and may make it harder to extend so as  to support LC rules and policy on subject heading formation. When modeling any complex area, regular interaction with relevant SMEs is essential throughout the process, both for knowledge elicitation and model validation.  Early, regular review of the ongoing work  by MODS/MADS Editorial Committee may be helpful.
  5. The current draft does not define the semantics of pre-combined headings; there are different possible interpretations,  but none are discussed, and some inferences appear to be common across all readings.
  6. Given known  data errors in the syndetic structure of LCSH (see e.g. LCSH is to Thesaurus as Doorbell is to Mammal), it may better to devote resources to correcting these issues before deepening the semantics of exposed Linked Data.  This would make the LC vocabularies more reuseable in the web of data, as recommended in the Deanna Marcum’s  ”Response to: On the record: Report of the Library of Congress Working Group On The Future of Bibliographic Control“.

[1] Following Soergel, I prefer this term to “Pre-coordinated”

[2] I also worked on relating the to the implied ontologies inferable from Knowledge Organization Systems that used sub-types of BT relationship (BTG, etc) that are part of the international standards, and which were originally included in SKOS, but which were moved to an extension , then finally dropped and relegated to examples in the published recommendation.

Comments No Comments »

I’m happy to announce the availability of the new  Tyler Hansbrough Special Edition of the NARA Archival Research Catalog. This special edition has fixes for the  character set encoding problems noted below. These problems were being caused by problems in the  UTF-8 encoding of   various single and double quotes, and en- and em- dashes, resulting in unparsable XML documents.

Downloading the modified data sets

The  Tyler Hansbrough special edition can be downloaded from the Fred 2.0 repository on iBiblio.

Affected characters

The following characters were affected – the counts given are for the number of times each character occurred, and not the number of documents each character occurred in.

[table id=1 /]

Affected documents

533 documents were affected.

A detailed list of which documents were affected, and which characters were replaced is available here.
Read the rest of this entry »

Comments No Comments »

There are a number of problems in the ARC data files that are caused by truncation of 16 bit unicode punctuation to 8-bits. There may be other undetected truncation errors where the truncated value is a legitimate UTF character, but this seems unlikely.

I have written code to work around this problem; it’s a fairly simple fix.

There are also about a dozen records that are ill-formed- the contain tags generated by an Oracle xml sql engine.

Scary thought for the day:
Imagine if Oracle made an ILS

Comments No Comments »

NARA has released the bulk contents of it’s Archival Research Catalog on, in ARC XML format.

Unfortunately, when I say bulk contents, I mean bulk contents; the current distribution contains nearly 20GB of raw data; the Organizational authority file adds another 1.7GB.

Since XML was deliberately designed for redundancy, XML encoded data sets are usually rather compressible. These data are no exception; bzip2 on the ARC data set achieves compression rations  between 40:1 and 50:1.   The 7-zip format does even better, but is not considered archival.

The real credit for this work belongs to the good people at NARA and their contributing agencies (and viewers like you). Don’t forget to visit the original pages on to give comments and feedback on this data, and don’t forget to show off all the cool UIs you build on top of this.

Mark Matienzo, (now a Yalie as of  Monday),  Richard Urban,  GSLIS, University of Illinois,and Simon Spero, SILS, University of North Carolina have prepared a compressed distribution of these data sets, allowing much more efficient access to the data.

This distribution is available from iBiblio at this location;

One interesting datum that emerged during the compression process is that the compressed size of Part1 is nearly twice as large as most of the other Parts, despite having the same raw size.  There are several possible explanations for this:

  • H1: The records in Part1 are more complete than in other files.
  • H2: Part1 describes record series from less related agencies than other files.
  • H3: Part1 contains records with less compressible data (image types, numeric data, etc).
  • H4: Part1 contains more, smaller, record series than other files.

If  only had a time machine, I could do a poster and go to the iConference.  Alas, all I have is the scarf and the Jelly babies, and I’ve eaten the Jelly babies.

This release is dedicated to the memory of Prof. Brian Vickery (1918-2009), whose passing was announced on Thursday the 28th.

Brian C. Vickery
Classified Life

Never too snarkaly
Faceted artfully
Helped refind science


See the Readme file for a bit more details.

Read the rest of this entry »

Comments No Comments »