Main menu:

Site search

Feeds

Categories

December 2014
S M T W T F S
« Sep    
 123456
78910111213
14151617181920
21222324252627
28293031  

Archive

Use jtidy to parse SGML and other badly formed XML

I finally have been able to use jtidy to clean up SGML and other kinds of almost-XML. The story is that I have a bajillion old-skool TEI SGML files that need to be inventoried, preferably not by having a person open each one and read it. These files are mostly un-parseable by modern xml parsers. Encoding errors abound. I thought about developing a pure regular expression matching solution, but that seemed like way too much work. I’m working in java, so Jtidy is an obvious solution, but it doesn’t seem to handle xml / sgml well as-is. So here’s what I came up with instead:

  1. Only parse the teiHeader part of the document. It has all the metadata I really need about the document, and is much more likely to be well-formed. However, even the teiHeaders have problems and need to be cleaned.
  2. Adapt jtidy to parse strings: First, Download jtidy – the 04aug2000r7-dev release. Newer, r8 releases won’t work. Open up the source code, and add this method to org.w3c.tidy.Tidy (Thanks to jruel’s post on sourceforge for getting me started!):

    public String cleanString(String toClean) {
    Node rootNode = this.parse(new ByteArrayInputStream(toClean.getBytes()), null);
    OutputStream out = new ByteArrayOutputStream();
    Out o = new OutImpl();
    o.encoding = configuration.CharEncoding;
    o.out = out;
    PPrint pprint = null;
    if(out != null) {
    pprint = new PPrint(configuration);
    Node currentNode = rootNode.content;
    while(currentNode != null) {
    if(configuration.XmlTags) {
    pprint.printXMLTree(o, (short) 0, 0, null, currentNode);
    } else {
    pprint.printTree(o, (short) 0, 0, null, currentNode);
    }
    currentNode = currentNode.next;
    }
    pprint.flushLine(o, 0);
    }
    String result = out.toString();
    try { out.close(); }
    catch(IOException e) { /* Do nothing. */ }
    return result;

    }

    Now, Recompile jtidy and add the new .jar file to your classpath

  3. Read in your teiHeader line-by-line, clean it up with jtidy, and build it into a DOM:

    private Document parseSGML(File f){
    System.out.println("Parsing SGML file");

    String s = null;
    Document doc = null;

    try {

    System.out.println("trying..." + f.getAbsolutePath());
    FileInputStream fis = new FileInputStream(f);

    BufferedReader stdInput = new BufferedReader(new InputStreamReader(fis));
    boolean inHeader = false;
    Pattern teiPattern = Pattern.compile("teiHeader");
    Matcher matcher = null;
    String teiHeader = new String("");

    while ((s = stdInput.readLine()) != null) {

    matcher = teiPattern.matcher(s);
    if(matcher.find()){
    inHeader = !inHeader;
    }
    if(inHeader){
    teiHeader = teiHeader.concat(s.toString().trim().replaceAll("\\b\\s{2,}\\b", " ")); // concat each string, but strip out excess whitespace
    }
    }

    if(teiHeader.length()>1) teiHeader.concat(""); // Add that last closing tag!

    SAXBuilder builder = new SAXBuilder(false);
    Tidy tidy = new Tidy(); // obtain a new Tidy instance
    tidy.setXmlOut(true); // set desired config options using tidy setters
    tidy.setXmlTags(true);
    tidy.setSpaces(1);
    tidy.setIndentAttributes(false);
    teiHeader = teiHeader.replace('"', '\u0022'); // escape all the double quotes, otherwise the parser fails
    String cleanString = tidy.cleanString(teiHeader);
    System.out.println(cleanString);

    doc = builder.build(new StringReader(cleanString));

    return doc;

    }catch (IOException e) {
    System.out.println("exception happened - here's what I know: " + e.getMessage());
    e.printStackTrace();
    return null;
    }catch (Exception e){
    System.out.println(e.getMessage());
    e.printStackTrace();
    return null;
    }

    }

Comments

Comment from Dorothea
Time: April 17, 2006, 5:38 pm

If you use Python at all, it still has an SGML library that acts like a SAX parser for SGML. It’d take care of this job in a jiffy.

Comment from Bjorn
Time: April 17, 2006, 5:55 pm

How do you get color syntax on your code? Is it a plugin? Thanks.

Write a comment