Main menu:

Site search

Feeds

Categories

July 2014
S M T W T F S
« Mar    
 12345
6789101112
13141516171819
20212223242526
2728293031  

Archive

Use jtidy to parse SGML and other badly formed XML

I finally have been able to use jtidy to clean up SGML and other kinds of almost-XML. The story is that I have a bajillion old-skool TEI SGML files that need to be inventoried, preferably not by having a person open each one and read it. These files are mostly un-parseable by modern xml parsers. Encoding errors abound. I thought about developing a pure regular expression matching solution, but that seemed like way too much work. I’m working in java, so Jtidy is an obvious solution, but it doesn’t seem to handle xml / sgml well as-is. So here’s what I came up with instead:

  1. Only parse the teiHeader part of the document. It has all the metadata I really need about the document, and is much more likely to be well-formed. However, even the teiHeaders have problems and need to be cleaned.
  2. Adapt jtidy to parse strings: First, Download jtidy – the 04aug2000r7-dev release. Newer, r8 releases won’t work. Open up the source code, and add this method to org.w3c.tidy.Tidy (Thanks to jruel’s post on sourceforge for getting me started!):
    public String cleanString(String toClean) {
            Node rootNode = this.parse(new ByteArrayInputStream(toClean.getBytes()), null);
            OutputStream out = new ByteArrayOutputStream();
            Out o = new OutImpl();
            o.encoding = configuration.CharEncoding;
            o.out = out;
            PPrint pprint = null;
            if(out != null) {
            pprint = new PPrint(configuration);
            Node currentNode = rootNode.content;
            while(currentNode != null) {
            if(configuration.XmlTags) {
             pprint.printXMLTree(o, (short) 0, 0, null, currentNode);
            } else {
             pprint.printTree(o, (short) 0, 0, null, currentNode);
            }
            currentNode = currentNode.next;
            }
             pprint.flushLine(o, 0);
            }
            String result = out.toString();
            try { out.close(); }
            catch(IOException e) { /* Do nothing. */ }
            return result;
            
           }

    Now, Recompile jtidy and add the new .jar file to your classpath

  3. Read in your teiHeader line-by-line, clean it up with jtidy, and build it into a DOM:
    private Document parseSGML(File f){
                    System.out.println(“Parsing SGML file”);
                   
            String s = null;
            Document doc = null;
           
            try {
                   
                            System.out.println(“trying…” + f.getAbsolutePath());
                            FileInputStream fis = new FileInputStream(f);
               
                            BufferedReader stdInput = new BufferedReader(new InputStreamReader(fis));
                            boolean inHeader = false;
                            Pattern teiPattern = Pattern.compile(“teiHeader”);
                            Matcher matcher = null;
                            String teiHeader = new String(“”);
               
                while ((s = stdInput.readLine()) != null) {
                   
                        matcher = teiPattern.matcher(s);
                        if(matcher.find()){
                            inHeader = !inHeader;
                        }
                        if(inHeader){
                            teiHeader = teiHeader.concat(s.toString().trim().replaceAll(\\b\\s{2,}\\b”, ” “)); // concat each string, but strip out excess whitespace
                        }
                }
                           
                if(teiHeader.length()>1) teiHeader.concat(“</teiHeader>”); // Add that last closing tag!
                           
                SAXBuilder builder = new SAXBuilder(false);
                Tidy tidy = new Tidy(); // obtain a new Tidy instance
                tidy.setXmlOut(true); // set desired config options using tidy setters
                tidy.setXmlTags(true);
                tidy.setSpaces(1);
                tidy.setIndentAttributes(false);
                teiHeader = teiHeader.replace(‘"’, \u0022′); // escape all the double quotes, otherwise the parser fails
                String cleanString = tidy.cleanString(teiHeader);
                System.out.println(cleanString);
               
                doc = builder.build(new StringReader(cleanString));
                           
                return doc;
               
            }catch (IOException e) {
                System.out.println(“exception happened – here’s what I know: “ + e.getMessage());
                e.printStackTrace();
                return null;
            }catch (Exception e){
                            System.out.println(e.getMessage());
                            e.printStackTrace();
                            return null;
            }

            }

Comments

Comment from Dorothea
Time: April 17, 2006, 5:38 pm

If you use Python at all, it still has an SGML library that acts like a SAX parser for SGML. It’d take care of this job in a jiffy.

Comment from Bjorn
Time: April 17, 2006, 5:55 pm

How do you get color syntax on your code? Is it a plugin? Thanks.

Comment from bess
Time: April 17, 2006, 8:12 pm

Dorothea: I did not know that about python. Hmmm. I have put off learning python, and I mainly wanted something in java because everything else in my inventory workflow is already in java. But if python does a good job of parsing sgml, maybe I finally need to break down a pick it up.

Write a comment