Main menu:

Site search

Feeds

Categories

September 2014
S M T W T F S
« Mar    
 123456
78910111213
14151617181920
21222324252627
282930  

Archive

Some notes on Nokogiri::XML::Reader

I’m a fan of Nokogiri, and I use it for all of my ruby XML processing needs. Recently, however, I needed to use the Nokogiri::XML::Reader class and had some trouble finding good examples so I’m putting some here.

Why might you want to use Nokogiri::XML::Reader? Probably because you have a very large XML file to process and reading the whole thing into memory, the way you do using the more standard Nokogiri::XML class, is going to be slow. For example, I started using Reader when one of our workflow robots needed to process a 2Mb XML file. Trying to process this file via DOM methods (i.e., the standard Nokogiri::XML method) was taking over a minute per file. Once we switched to using Nokogiri::XML::Reader, processing the same file took about 4 seconds.

The official docs are available at http://nokogiri.org/Nokogiri/XML/Reader.html, but they don’t provide many examples beyond the most trivial ones. The crucial bit is this (adapted from the nokogiri docs):

reader = Nokogiri::XML::Reader(IO.read(my_xml_file))
reader.each do |node|
puts node.name
end

The idea is that the cursor moves forward through the document, and you have to get any info you want as the cursor moves past the part of the document you need. One pattern I encountered several times was this:


reader = Nokogiri::XML::Reader(IO.read(my_xml_file))
reader.each do |node|
case node.name
when 'PREMIS:premis'
# do PREMIS stuff
when 'something_else'
# do something else
end
end

That lets you switch on the node name, which is pretty useful, but falls short in some important cases. First of all, it’s not obvious that with this method both <PREMIS:premis> AND </PREMIS:premis> are going to match that when statement. It also isn’t clear how to use this to match nested elements, for example if you only want PREMIS data when it occurs inside a specific element.

After some experimentation, here’s a method I like more. Consider this psudo code. I’m leaving out things that seem irrelevant to the Nokogiri::XML::Reader structure. Also note that node.node_type==1 will restrict to matching only opening elements. This processes each <repInfo> node, and then within that node you can use node.read to move the cursor forward as needed.

def generate_jhove_hash
reader = Nokogiri::XML::Reader(File.read(jhove_file))
reader.each do |node|
if(node.name=='repInfo' and node.node_type==1)
# do some stuff
node.read until node.name=='format'
node.read # to get to the inside of the format tag
jhove_hash[current_uri]["format"] = node.value
node.read until node.name=='status'
node.read # to get to the inside of the status tag
jhove_hash[current_uri]["status"] = node.value
if(current_uri =~ /jp2/)
node.read until node.name=='mix:imageWidth'
node.read
jhove_hash[current_uri]["width"] = node.value
node.read until node.name=='mix:imageHeight'
node.read
jhove_hash[current_uri]["height"] = node.value
end
end
end
return jhove_hash
end

Comments

Comment from Jason Ronallo
Time: December 11, 2010, 8:37 pm

You may also want to check out the SAX parser which is a similar kind of interface and is also very fast. In that case all you have is the ability to have a start tag, end tag and characters in between. Because this kind of event driven parsing is so different, something like https://github.com/pauldix/sax-machine gives a nice syntax for defining classes.

Comment from Amol Pujari
Time: July 14, 2012, 1:52 am

check out this example – https://github.com/amolpujari/reading-huge-xml/blob/master/examples/item.rb

Write a comment