Speakers:
John Jansson, Chris Cowan, Bob Huggins and Kim Dail
Moderator: Patti Graziano
Reported by Michael Jesse
One of the hottest topics in the news library industry is the recent emergence of full digitization from microfilm as a mature technology.
Digitization projects have been around for years in other industries, but newspaper archives presented a much more difficult challenge, in terms of format, quality of reproduction and sheer volume. The Orlando Sentinel was a very-early pioneer and in 1997 Sandy Levy of the Baltimore Sun spoke at a News Division conference program about the Sun's ground-breaking digitization program.
At that time, the key vendor in the field was PTFS (Progressive Technology Federal Systems, Inc). The PTFS method of searching against minimally-corrected OCR text while displaying an image of the page as the search result became the foundation for much of the progress that followed. Although PFTS remains active in the field, other vendors have joined in.
In 2001 ProQuest announced a major digitization program involving The New York Times, Washington Post and Los Angeles Times. Soon, the Chicago Tribune would add its name to that list. Other digitization vendors -- Olive Software, Cold North Wind, Heritage Microfilm and others -- launched similar projects and by 2002 it was clear not just that the technology had arrived, but that there was a growing consensus among both publishers and vendors that the significant investment required would pay for itself over time.
Our program at the 2002 conference was moderated and organized by Patti Graziano of the Cleveland Plain Dealer. Patti brought in vendor representatives from ProQuest, Olive and Cold North Wind, and for the librarian's perspective she recruited John Jansson of the Chicago Tribune.
John described the Tribune's original digitization program with PTFS, and the difficult decision the paper had to make when it ended that relationship and joined in on the ProQuest project.
However, John's presentation was not primarily about any specific vendor, but about what he has learned from the experience and his advice for any news librarian considering such a project. It is particularly important, John said, for the news librarian to be and the center of the project -- the one who knows more about it that anyone else in the room. You have to plan more than anything you have planned before, you have to get full commitment from others in your organization who need to be involved, you have to be the one in charge, and you have to be willing to throw out the first results if they don't meet the quality standards you set.
Following John on the program were vendor reps Chris Cowan of ProQuest, Kim Dail of Olive Software and Bob Huggins of Cold North Wind. (Rhonda Hollimon of the Augusta Chronicle was also scheduled to speak but had to leave conference early because of a family emergency.)
Chris, Kim and Bob each gave an overview of the digitization projects their companies are selling. Chris described the decision-making process ProQuest went through with its customers in academic and public libraries, and with the publishers providing their content. They decided it was necessary to develop a system which could isolate individual articles and also unite them with their jumps on different pages. It was also decided to re-key certain fields if the OCR was imperfect. Headlines, bylines the lead paragraph and photo captions are retyped while the body of the document remains as it emerged from the OCR scan.
This is expensive and involves a workforce of about 2,500 people, mostly contracted overseas. It also takes close to a year to complete the full run of a newspaper from its first issue to date.
Bob Huggins showed a simpler and less expensive approach in which the OCR text is not corrected and the process does not go to the article level. Each newspaper page is searchable as one document and it is up to the end-user to make the logical connection from the beginning of the article to its jump on another page. Bob said the full run of a newspaper can be converted in one month.
The Olive product is similar to ProQuest in terms of going ot the article level. Kim's presentation focused on the differences between "page-based" and "object-based" (article) approaches.