Main menu:

Site search

Feeds

Categories

July 2008
S M T W T F S
« Jun    
 12345
6789101112
13141516171819
20212223242526
2728293031  

Archive

java vs. javascript urlencoding

This is one of those ridiculous corner-cases one sometimes runs into, and chances are no one but me will ever encounter it, but it took me a couple of days to solve it so I want to write it down in case it ever comes up again.

I have an XML searching application that’s using XTF. 99% of the time the search has been working fine, even for most searches with non-Latin characters in them. However, some non-Latin characters in a search were causing a crash, with baffling error messages, (e.g., “An invalid XML character (Unicode: 0×1) was found in the element content of the document”) when I *know* and have verified over and over again that the document in question has no illegal unicode characters in it.

I’ll spare you the saga of my pursuit of this bug, and skip to the kill. Although most of the time it seems to be fine to url encode a string with javascript and then decode it with java URLDecoder (as you might if you’re passing a GET query to an XSL parser), when you’re dealing with unicode characters outside the Latin-1 range these encoding schemes seem to diverge. I wrote this little java program to prove it to myself (borrowing heavily from this bug report, and javascript url encoding courtesy of here)

import java.net.*;

public class Test {
    public static void main (String args[]){

            String control = “this is a control string?!”;
            String control_jencoded = URLEncoder.encode(control);
            String control_jsencoded = “this%20is%20a%20control%20string%3F%21″;
            System.out.println(\ncontrol url encoded by java = “ + control_jencoded);
            System.out.println(\ncontrol url encoded by javascript = “ + control_jsencoded);

            String ghazali = “Ghazālī”;
            String ghazali_jencoded = URLEncoder.encode(ghazali);
            String ghazali_jsencoded = “Ghaz%u0101l%u012B”;
            System.out.println(\nGhazālī url encoded by java = “ + ghazali_jencoded);

            System.out.println(\nGhazālī url encoded by javascript = “ + ghazali_jsencoded);

            String trevino = “Treviño”;
            String trevino_jencoded = URLEncoder.encode(trevino);
            String trevino_jsencoded = “Trevi%F1o”;
            System.out.println(\nTreviño url encoded by java = “ + trevino_jencoded);
            System.out.println(\nTreviño url encoded by javascript = “ + trevino_jsencoded);

        String s[] = {control_jencoded, control_jsencoded, ghazali_jencoded, ghazali_jsencoded, trevino_jencoded, trevino_jsencoded};
        for (int i = 0; i < s.length; i++) {
            try {
                String decoded = URLDecoder.decode(s[i]);
                System.out.println(\n\” + s[i] + \” –> \” + decoded + \”);
            } catch (Exception e) {
                System.out.println(\n\” + s[i] + \” –> “ + e);
            }
        }
    }
}

The output looks like this:

control url encoded by java = this+is+a+control+string%3F%21

control url encoded by javascript = this%20is%20a%20control%20string%3F%21

Ghazālī url encoded by java = Ghaz%C4%81l%C4%AB

Ghazālī url encoded by javascript = Ghaz%u0101l%u012B

Treviño url encoded by java = Trevi%C3%B1o

Treviño url encoded by javascript = Trevi%F1o

“this+is+a+control+string%3F%21″ –> “this is a control string?!”

“this%20is%20a%20control%20string%3F%21″ –> “this is a control string?!”

“Ghaz%C4%81l%C4%AB” –> “Ghazālī”

“Ghaz%u0101l%u012B” –> java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in escape (%) pattern - For input string: “u0″

“Trevi%C3%B1o” –> “Treviño”

“Trevi%F1o” –> “Trevi?o”

So you can see, java can decode the control string regardless of who encoded it, but fails to decode the other two javascript encoded strings. “Treviño” is just decoded wrong, but the attempt to decode “Ghazālī” throws an exception.

The solution is that I now use java to url encode any queries I need to pass, and then the browser doesn’t attempt to encode them, and everything can de-code just fine. And happy ending, The Journal of Islamic Philosophy is now searchable.

my clover lawn is doing great

… especially with all this rain we’re getting. I’ll post a photo soon, but in the meantime, read this comic and share my smug.

beCamp is coming up!

This is a repeat of a post I wrote on the beTech Blog

Special note for code4libbers: If you’re anywhere in driving distance of Charlottesville and you like technology, beer, and FUN, please consider yourself invited to beCamp. We’re going to have a pretty healthy code4lib contingent already, so chances are good that you will know someone else here. Please come!

Mark your calendars for beCamp! beCamp May 2nd-3rd 2008 @ Charlottesville Business Innovation Council (CBIC), 501 E. Main Street, Charlottesville, VA (on Downtown Mall)

We have 68 registered attendees so far. If you’re planning to come but haven’t yet registered, do it soon so you can get a t-shirt! You can register at http://barcamp.org/beCamp2008.

Last year this event was a huge success, with a huge range of topics from ruby, smalltalk, and solr (which you might expect) to 3D modeling, robotics, and reputation management (which you might not expect, which makes them even cooler). beCamp is whatever YOU want it to be, so bring your ideas for a session.

Event Details

When

  • Friday, May 2nd, 5:00PM-9:00PM
  • Saturday, May 3rd, 9:00AM-5:00PM

Where

  • Charlottesville Business Innovation Council
  • 501 East Main Street, Charlottesville, Virginia 22902

Bring

  • Brain
  • Laptop
  • Reference books
  • Games—cards, board games, Legos, video (bring a monitor as well)
  • Snacks, beverages
  • Camera (Flickr tags: beCamp, beCamp2008, BarCamp)
  • Business cards

Please comment on the Blacklight award nomination

Shameless self-promotion warning!

Blacklight has been nominated[1] for a Mellon Award for Technology Collaboration. You can read the nomination here: http://matc.mellon.org/nominate/university-of-virginia/project-blacklight. I feel a little abashed about some of the language, but I guess no one ever won an award by being modest.

If you think Blacklight is a worthwhile project, would you please go show us some love? I think it might help our nomination to have some supportive comments on there.

Thank you!

[1] Don’t you love the passive voice? It “has been nominated…” Don’t worry about by whom. ;)

How not to apply for a job

Update: Image of letter removed. I’m feeling paranoid.

Folks, I kid you not, this was an actual cover letter I just read, applying for a full-time professional librarian position. I don’t feel safe posting it here since you’ll notice in spite of the fact that it has no identifying information about it at all. It isn’t even signed.

Let me just echo the thoughts of the always eloquent Dorothea. People on search committees are busy, some of us are serving on committees outside our area of expertise, and we are almost always constrained by fair hiring practices (and rightly so) to consider only how closely your cover letter and your CV match the job ad. That means that I cannot say, “Oh, this person mentions open source software, I bet that means they know how to run apache even though they haven’t mentioned apache explicitly.” No, if the job ad says apache, I need to see the literal string “apache” before I can check the box off on my little mandated spreadsheet. Same with “project management,” “interest in continuing education,” “supervisory experience”, and everything else in the job ad. A close second, if you can’t honestly say those things in your CV, is to use your polished and well-written (and signed! Do I even need to say that?) cover letter to explain why “Although I do not have significant experience in X, I do have extensive experience with Y, which is similar, and I have taken Z steps to learn skill X.” Then I can give you partial credit, or maybe even full marks, depending on how well you’ve made your case.

When applying for a job anywhere in academia, please also disregard the terrible advice that someone somewhere keeps spouting, to keep your letter and CV under one page. There’s no need to run on un-necessarily, but I would much rather see a coherent discussion of your skills and experience that needs a paperclip than a truncated bulleted list that can’t possibly address what you’re actually capable of doing.

Please, recent graduates of the world, it is breaking my heart to read CVs of people who are probably talented, and have to give them big fat zeros all the way down that stupid little spreadsheet, for lack of the right buzzwords.