marc4j 2.4 released!
It’s been a long time coming, but there is now a new release of marc4j. You can download it here: http://marc4j.tigris.org/ Most of the work on this new release was by people primarily working on SolrMarc, the indexing system for Blacklight and VuFind.
If you’re not deep into these projects, you might not know that the SolrMarc developers have adopted marc4j, which was a bit of an orphaned project in recent years since its founder Bas Peters hasn’t had time to work on it. Marc4j was always, in my mind, the java library for processing and manipulating MaRC records. However, it was sort of famously picky and unable to handle malformed MaRC records, which quite frankly is what most of us have to deal with most of the time. In order to make the marc4j library robust enough to deal with Blacklight, and then SolrMarc and VuFind, Bob Haschart, who I am proud to say works mainly on the Blacklight project at UVA, made some really clever changes. I’ll include the change notes, but in summary, badly encoded characters are now much more easily dealt with. All those darn cataloging modules that don’t allow a cataloger to type in unicode (After all, no library would ever have a book not in English, right?) have led to the unfortunate practice of typing unicode characters into Microsoft Word or something similar and then copying and pasting the text into the relevant MaRC field. This leads to a MaRC record that not only lies about its encoding, but the entire record sometimes isn’t even in the same encoding throughout! These records used to fail silently, or worse, start a cascade of errors that prevented any subsequent records from processing cleanly either, but now Marc4j does some clever detective work to try to determine what character encoding a string is in, convert it to real unicode where possible, and at the very least log the error and fail gracefully, allowing the processing of subsequent records to continue.
Bob has now mostly taken over development of marc4j, which I hope will mean it will stay in active development for a good long time. All you folks out there with marc records that won’t parse and you don’t know why, give marc4j a try, and if you still have a stumper of a record write to the mailing list and let’s try to lick this bad marc record problem together.
Here are the marc4j 2.4 release notes:
Changes to MARC4J 2.4
MARC4J 2.4 is a minor release providing some bug fixes and some new functionality.
- Added MarcPermissiveStreamReader which is more capable of reading records that contain structural or encoding errors, and is capable of translating the records to UTF-8 as they are read.
- Added ErrorHandler which is used for tracking and reporting structural or encoding errors encountered by the MarcPermissiveStreamReader.
- Added MarcDirStreamReader which iterates over all of the MARC record files in a given directory.
- Modified MarcStreamReader so that if an exception is thrown for an error in one record you can choose to catch the exception, discard the erroneous record and continue reading from the input file.
- Modified AnselToUnicode to fix some problems that would occur when trying to handle Chinese characters, to fix an infinite loop problem that would occur sometimes when extraneous characters appear within a MARC8 character set escape sequence, and made many changes to support the MarcPermissiveStreamReader to report and try to recover from encoding errors in the records being read.
- Modified CodeTable (which is used by AnselToUnicode) so that rather than reading and parsing a large XML file to create the hash tables for mapping MARC8 to Unicode at runtime, the parsing is done once at compile time, and a class that handles the mapping directly via switch statements is automatically generated.
- Made minor changes to the MarcXmlReader so that if an exception occurs in the MarcXmlParserThread that it starts, the exception is passed to the MarcXmlReader rather than simply hanging the parser thread.
- Added PermissiveReaderExample which demonstrates how to use the MarcPermissiveReader to examine and/or validate records for structural or encoding errors.