Main menu:

Site search



April 2014
« Mar    


Preliminary notes on Blacklight and LocalSolr

SearchCampDC was last week, and UVA was well represented. I think we even tied Library of Congress (the hosting institution) for the number of attendees. Among the many excellent presentations and conversations, Patrick O’Leary’s LocalSolr stood out for me as being of immediate interest to Blacklight, and Matt Mitchell and I started working on getting geospatial search into Blacklight immediately upon returning to Charlottesville.

The bad news is, it isn’t quite working yet. The good news is, we’re pretty close, and I think we just need someone with better Java mojo than I can muster, so I think I’m going to ask Bob Haschart to take a look at it next week. Here’s where things stand at the moment, though:

1. LocalSolr is not yet a part of the Solr trunk (although apparently this is on the horizon), so you have to download it separately, which you can read about here. There is a great example application on that site as well, and a good tutorial about putting LocalSolr into an existing solr instance here. The tutorial is a little rough around the edges and hasn’t been updated since Solr 1.3 was released (although the code does work with 1.3, with some minor modifications.) I worked though these issues by reading the mailing list, and in the spirit of scratching one’s own itch I’ve volunteered to help Patrick update the tutorial asap.

2. Matt wrote a clever little script to query the top 1000 place names mentioned in our library catalog and then query Yahoo’s geo-search to get lat and long values for each of these. Those values are now cached in a mysql database.

So now we just have to get those lat/long values into our solr index!

3. My first thought was that I’d just write a little custom method in SolrMarc, which is what we use to do our indexing. So, the properties file would look like this:

region_facet = custom, removeTrailingPunct(650z:651a)
lat = custom, getLat(650z:651a)
lng = custom, getLng(650z:651a)

where getLat and getLng would be methods that query the mysql database, and the code would live in, where our custom indexing methods live. However, this doesn’t work. It does indeed insert the lat and long values into the solr record, but they just sit there as ordinary numbers, and none of the geospatial behavior is enabled. That’s because SolrMarc isn’t really using Solr when it indexes, it’s using the lucene libraries to write directly to a lucene index (see update). This has the advantage of being speedy, but not being able to take advantage of solr-specific index-time behaviors is an obvious disadvantage. I tried to see if I could get my getLat and getLng methods to replicate the index-time behavior that Patrick has written into LocalSolr, but my skills just aren’t up to it. I’m still hopeful about this approach though, and I’m going to ask Bob to look into it next week. UPDATE: Hooray for Bob! Jonathan, you’ll be happy to know solrmarc is NOT bypassing solr, we’re just bypassing the HTTP layer, by using embedded solr. This means that some things that the HTTP ingest process does for you have to be called explicitly, like the localsolr update method that translates lat/long strings into geo-spatially searchable coordinates. Bob fixed this, we have a working demo, and as soon as it’s a bit more robust I’ll do a write-up.

4. The next thing I tried was a script to update an existing Solr document, like this:

require ‘rubygems’
require ’solr’

# connect to the solr instance
conn =‘http://localhost:8080/solr’, :autocommit => :o n)

# iterate through some hits
  conn.query(‘carolina’) do |hit|
    a =   
    hit.keys.each do |key|
      a[key.to_sym] = hit[key]
    a[:lat] = ‘40.6724′
    a[:lng] = ‘-73.5287′

So, get the Solr document, and only update the lat and lng values, keeping everything else the same by passing it back through. And again, it *almost* works, but it loses all of my un-stored index values. This approach would work for a smaller system that could afford to store all of its values, but this index is big enough that that’s not an option for us. I did have a fun time reading about the history of this issue on the Solr jira, though.

I feel like we’re very close to getting this working. What I would love to see is a map interface to our collections. We are already doing a pretty good job of getting all of our books and images into Blacklight, and increasingly we’re ingesting archaeological artifacts and GIS data sets as well. Particularly given the amount of place-specific history we support, it would be so useful to be able to draw a box on a map and have that form part of your library query. It looks like we’ll be presenting on Blacklight and on our GIS infrastructure at code4lib this year, so that also lights a fire under my feet to have this working before then.


Comment from Jonathan Rochkind
Time: December 7, 2008, 11:39 am

Sweet. Consider using the open source/open access geonames instead of/in addition to yahoo.

Comment from Jonathan Rochkind
Time: December 7, 2008, 12:12 pm

I’d also been thinking about that kind of updating of an exiting solr document (without having much solr experience), and was worried about needing to ‘lock’ it in some way to make sure you don’t over-write another procesesses new indexing? But I guess you just make sure not to run an indexer at the same time you are running this kind of updater?

But I didn’t even know enough about SOLR to be concerned about that “un-stored index value” issue. I don’t even know what that means, in fact. But I think this kind of updating of an existing index value could be useful in a variety of circumstances, so I hope there’s a way around it.

Comment from Jonathan Rochkind
Time: December 7, 2008, 12:13 pm

And I ALSO hadn’t realized that SOLRMarc was going right to lucene rather than through SOLR. I don’t really like the sound of that, hmm.

Write a comment