java vs. javascript urlencoding
This is one of those ridiculous corner-cases one sometimes runs into, and chances are no one but me will ever encounter it, but it took me a couple of days to solve it so I want to write it down in case it ever comes up again.
I have an XML searching application that’s using XTF. 99% of the time the search has been working fine, even for most searches with non-Latin characters in them. However, some non-Latin characters in a search were causing a crash, with baffling error messages, (e.g., “An invalid XML character (Unicode: 0×1) was found in the element content of the document”) when I *know* and have verified over and over again that the document in question has no illegal unicode characters in it.
I’ll spare you the saga of my pursuit of this bug, and skip to the kill. Although most of the time it seems to be fine to url encode a string with javascript and then decode it with java URLDecoder (as you might if you’re passing a GET query to an XSL parser), when you’re dealing with unicode characters outside the Latin-1 range these encoding schemes seem to diverge. I wrote this little java program to prove it to myself (borrowing heavily from this bug report, and javascript url encoding courtesy of here)
public class Test {
public static void main (String args[]){
String control = “this is a control string?!”;
String control_jencoded = URLEncoder.encode(control);
String control_jsencoded = “this%20is%20a%20control%20string%3F%21″;
System.out.println(“\ncontrol url encoded by java = “ + control_jencoded);
System.out.println(“\ncontrol url encoded by javascript = “ + control_jsencoded);
String ghazali = “Ghazālī”;
String ghazali_jencoded = URLEncoder.encode(ghazali);
String ghazali_jsencoded = “Ghaz%u0101l%u012B”;
System.out.println(“\nGhazālī url encoded by java = “ + ghazali_jencoded);
System.out.println(“\nGhazālī url encoded by javascript = “ + ghazali_jsencoded);
String trevino = “Treviño”;
String trevino_jencoded = URLEncoder.encode(trevino);
String trevino_jsencoded = “Trevi%F1o”;
System.out.println(“\nTreviño url encoded by java = “ + trevino_jencoded);
System.out.println(“\nTreviño url encoded by javascript = “ + trevino_jsencoded);
String s[] = {control_jencoded, control_jsencoded, ghazali_jencoded, ghazali_jsencoded, trevino_jencoded, trevino_jsencoded};
for (int i = 0; i < s.length; i++) {
try {
String decoded = URLDecoder.decode(s[i]);
System.out.println(“\n\”“ + s[i] + “\” –> \”“ + decoded + “\”“);
} catch (Exception e) {
System.out.println(“\n\”“ + s[i] + “\” –> “ + e);
}
}
}
}
The output looks like this:
control url encoded by javascript = this%20is%20a%20control%20string%3F%21
Ghazālī url encoded by java = Ghaz%C4%81l%C4%AB
Ghazālī url encoded by javascript = Ghaz%u0101l%u012B
Treviño url encoded by java = Trevi%C3%B1o
Treviño url encoded by javascript = Trevi%F1o
“this+is+a+control+string%3F%21″ –> “this is a control string?!”
“this%20is%20a%20control%20string%3F%21″ –> “this is a control string?!”
“Ghaz%C4%81l%C4%AB” –> “Ghazālī”
“Ghaz%u0101l%u012B” –> java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in escape (%) pattern - For input string: “u0″
“Trevi%C3%B1o” –> “Treviño”
“Trevi%F1o” –> “Trevi?o”
So you can see, java can decode the control string regardless of who encoded it, but fails to decode the other two javascript encoded strings. “Treviño” is just decoded wrong, but the attempt to decode “Ghazālī” throws an exception.
The solution is that I now use java to url encode any queries I need to pass, and then the browser doesn’t attempt to encode them, and everything can de-code just fine. And happy ending, The Journal of Islamic Philosophy is now searchable.
Posted: June 12th, 2008 under random.
Comments: 3
