Use jtidy to parse SGML and other badly formed XML
I finally have been able to use jtidy to clean up SGML and other kinds of almost-XML. The story is that I have a bajillion old-skool TEI SGML files that need to be inventoried, preferably not by having a person open each one and read it. These files are mostly un-parseable by modern xml parsers. Encoding errors abound. I thought about developing a pure regular expression matching solution, but that seemed like way too much work. I’m working in java, so Jtidy is an obvious solution, but it doesn’t seem to handle xml / sgml well as-is. So here’s what I came up with instead:
- Only parse the teiHeader part of the document. It has all the metadata I really need about the document, and is much more likely to be well-formed. However, even the teiHeaders have problems and need to be cleaned.
- Adapt jtidy to parse strings: First, Download jtidy – the 04aug2000r7-dev release. Newer, r8 releases won’t work. Open up the source code, and add this method to org.w3c.tidy.Tidy (Thanks to jruel’s post on sourceforge for getting me started!):
public String cleanString(String toClean) {
Node rootNode = this.parse(new ByteArrayInputStream(toClean.getBytes()), null);
OutputStream out = new ByteArrayOutputStream();
Out o = new OutImpl();
o.encoding = configuration.CharEncoding;
o.out = out;
PPrint pprint = null;
if(out != null) {
pprint = new PPrint(configuration);
Node currentNode = rootNode.content;
while(currentNode != null) {
if(configuration.XmlTags) {
pprint.printXMLTree(o, (short) 0, 0, null, currentNode);
} else {
pprint.printTree(o, (short) 0, 0, null, currentNode);
}
currentNode = currentNode.next;
}
pprint.flushLine(o, 0);
}
String result = out.toString();
try { out.close(); }
catch(IOException e) { /* Do nothing. */ }
return result;
}Now, Recompile jtidy and add the new .jar file to your classpath
- Read in your teiHeader line-by-line, clean it up with jtidy, and build it into a DOM:
private Document parseSGML(File f){
System.out.println(“Parsing SGML file”);
String s = null;
Document doc = null;
try {
System.out.println(“trying…” + f.getAbsolutePath());
FileInputStream fis = new FileInputStream(f);
BufferedReader stdInput = new BufferedReader(new InputStreamReader(fis));
boolean inHeader = false;
Pattern teiPattern = Pattern.compile(“teiHeader”);
Matcher matcher = null;
String teiHeader = new String(“”);
while ((s = stdInput.readLine()) != null) {
matcher = teiPattern.matcher(s);
if(matcher.find()){
inHeader = !inHeader;
}
if(inHeader){
teiHeader = teiHeader.concat(s.toString().trim().replaceAll(“\\b\\s{2,}\\b”, ” “)); // concat each string, but strip out excess whitespace
}
}
if(teiHeader.length()>1) teiHeader.concat(“</teiHeader>”); // Add that last closing tag!
SAXBuilder builder = new SAXBuilder(false);
Tidy tidy = new Tidy(); // obtain a new Tidy instance
tidy.setXmlOut(true); // set desired config options using tidy setters
tidy.setXmlTags(true);
tidy.setSpaces(1);
tidy.setIndentAttributes(false);
teiHeader = teiHeader.replace(‘"’, ‘\u0022′); // escape all the double quotes, otherwise the parser fails
String cleanString = tidy.cleanString(teiHeader);
System.out.println(cleanString);
doc = builder.build(new StringReader(cleanString));
return doc;
}catch (IOException e) {
System.out.println(“exception happened – here’s what I know: “ + e.getMessage());
e.printStackTrace();
return null;
}catch (Exception e){
System.out.println(e.getMessage());
e.printStackTrace();
return null;
}}
Posted: April 17th, 2006 under code4lib.
Comments: 3
Comments
Comment from Dorothea
Time: April 17, 2006, 5:38 pm
If you use Python at all, it still has an SGML library that acts like a SAX parser for SGML. It’d take care of this job in a jiffy.
Comment from Bjorn
Time: April 17, 2006, 5:55 pm
How do you get color syntax on your code? Is it a plugin? Thanks.
Comment from bess
Time: April 17, 2006, 8:12 pm
Dorothea: I did not know that about python. Hmmm. I have put off learning python, and I mainly wanted something in java because everything else in my inventory workflow is already in java. But if python does a good job of parsing sgml, maybe I finally need to break down a pick it up.

Write a comment