Main menu:

Site search

Feeds

Categories

February 2012
S M T W T F S
« Jan    
 1234
567891011
12131415161718
19202122232425
26272829  

Archive

I am the 99%

I just got an email from Elizabeth Warren’s campaign about President Obama’s appointment of Richard Cordray as the first director of the Consumer Financial Protection Bureau. This gives me hope that we can change this deeply broken system.

I am a native English speaker with a Master’s Degree. I am employed as a software engineer at Stanford University. Clearly, I have some advantages in my favor and I know how to use my brain effectively in many situations. And yet I signed a mortgage I could not understand because I trusted people who seemed nice to explain it to me and look out for my best interests. That was a terrible mistake that I’m paying for heavily right now.

I am one of the people whose home is in foreclosure. The stress of not being able to keep up with my house payments caused me tremendous grief, irreparable damage to my marriage, plenty of stress-related damage to my body, and has left some deep emotional scars that I am just starting to learn how to heal.

Let’s heal this country. Let’s elect people who give a damn about other people. Elect Elizabeth Warren: http://elizabethwarren.com/

Some notes on Nokogiri::XML::Reader

I’m a fan of Nokogiri, and I use it for all of my ruby XML processing needs. Recently, however, I needed to use the Nokogiri::XML::Reader class and had some trouble finding good examples so I’m putting some here.

Why might you want to use Nokogiri::XML::Reader? Probably because you have a very large XML file to process and reading the whole thing into memory, the way you do using the more standard Nokogiri::XML class, is going to be slow. For example, I started using Reader when one of our workflow robots needed to process a 2Mb XML file. Trying to process this file via DOM methods (i.e., the standard Nokogiri::XML method) was taking over a minute per file. Once we switched to using Nokogiri::XML::Reader, processing the same file took about 4 seconds.

The official docs are available at http://nokogiri.org/Nokogiri/XML/Reader.html, but they don’t provide many examples beyond the most trivial ones. The crucial bit is this (adapted from the nokogiri docs):

reader = Nokogiri::XML::Reader(IO.read(my_xml_file))
reader.each do |node|
  puts node.name
end

The idea is that the cursor moves forward through the document, and you have to get any info you want as the cursor moves past the part of the document you need. One pattern I encountered several times was this:

reader = Nokogiri::XML::Reader(IO.read(my_xml_file))
    reader.each do |node|
      case node.name
        when ‘PREMIS:premis’
          # do PREMIS stuff
        when ’something_else’
          # do something else
      end
    end

That lets you switch on the node name, which is pretty useful, but falls short in some important cases. First of all, it’s not obvious that with this method both <PREMIS:premis> AND </PREMIS:premis> are going to match that when statement. It also isn’t clear how to use this to match nested elements, for example if you only want PREMIS data when it occurs inside a specific element.

After some experimentation, here’s a method I like more. Consider this psudo code. I’m leaving out things that seem irrelevant to the Nokogiri::XML::Reader structure. Also note that node.node_type==1 will restrict to matching only opening elements. This processes each <repInfo> node, and then within that node you can use node.read to move the cursor forward as needed.

def generate_jhove_hash   
    reader = Nokogiri::XML::Reader(File.read(jhove_file))
    reader.each do |node|
      if(node.name==‘repInfo’ and node.node_type==1)
        # do some stuff
        node.read until node.name==‘format’
        node.read # to get to the inside of the format tag
        jhove_hash[current_uri][“format”] = node.value
        node.read until node.name==’status’
        node.read # to get to the inside of the status tag
        jhove_hash[current_uri][“status”] = node.value
        if(current_uri =~ /jp2/)
          node.read until node.name==‘mix:imageWidth’
          node.read
          jhove_hash[current_uri][“width”] = node.value
          node.read until node.name==‘mix:imageHeight’
          node.read
          jhove_hash[current_uri][“height”] = node.value
        end
      end
    end
    return jhove_hash
  end

Vampires vs Werewolves — Slides from my code4lib 2010 talk

Here are the slides from my code4lib 2010 talk, which was titled “Vampires vs Werewolves: Ending the War Between Developers and Sysadmins with Puppet”

download pdf of slides

Blacklight 2.4 release

It’s strange to do a release without creating a download package, but Blacklight 2.4 is available in github here: http://github.com/projectblacklight/blacklight/tree/v2.4.0. You can read the release notes at http://projectblacklight.org/?p=129 and make sure you also admire our new logo and style scheme at http://projectblacklight.org.

Dear Ruby,

Listen, we need to talk. I’ve been seeing someone else. Yes, it’s python. I know I said I didn’t like python, but I never really gave it a chance before, and now that I have there’s an undeniable attraction.

Of course I’m still attracted to you! With your integrated testing frameworks, hipster cache, and glossy web 2.0 mystique, how could I not be? But there are some areas where you just aren’t giving me what I need. How many times have we talked about linked data? I’ve told you how much I want to try it, but every time I mention that you try to get me to have a three-way with java, and I’ve told you before that’s just not my scene. I had the same conversation with python, and it was so easy, it just sort of happened. I don’t think I can let that go.

It isn’t as if you haven’t been playing the field too. Aside from your ongoing thing with java, every time I go to a ruby event these days it actually ends up being about erlang or maya. I know you have an adventurous streak, and that’s one of the things I love about you, but most of the time I just want to be able to get my work done. Python is going to help me do that. I hope you understand.

No, I’m not breaking up with you! We have so many great projects we’re working on together, and there’s so much we’ve planned to do together that I would never want to give up on. I just wanted to tell you about this python thing before you heard about it somewhere else, and let you know there’s room in my heart for both of you.