Main menu:

Site search



August 2016
« Sep    


java vs. javascript urlencoding

This is one of those ridiculous corner-cases one sometimes runs into, and chances are no one but me will ever encounter it, but it took me a couple of days to solve it so I want to write it down in case it ever comes up again.

I have an XML searching application that’s using XTF. 99% of the time the search has been working fine, even for most searches with non-Latin characters in them. However, some non-Latin characters in a search were causing a crash, with baffling error messages, (e.g., “An invalid XML character (Unicode: 0x1) was found in the element content of the document”) when I *know* and have verified over and over again that the document in question has no illegal unicode characters in it.

I’ll spare you the saga of my pursuit of this bug, and skip to the kill. Although most of the time it seems to be fine to url encode a string with javascript and then decode it with java URLDecoder (as you might if you’re passing a GET query to an XSL parser), when you’re dealing with unicode characters outside the Latin-1 range these encoding schemes seem to diverge. I wrote this little java program to prove it to myself (borrowing heavily from this bug report, and javascript url encoding courtesy of here)


public class Test {
public static void main (String args[]){

String control = "this is a control string?!";
String control_jencoded = URLEncoder.encode(control);
String control_jsencoded = "this%20is%20a%20control%20string%3F%21";
System.out.println("\ncontrol url encoded by java = " + control_jencoded);
System.out.println("\ncontrol url encoded by javascript = " + control_jsencoded);

String ghazali = "Ghazālī";
String ghazali_jencoded = URLEncoder.encode(ghazali);
String ghazali_jsencoded = "Ghaz%u0101l%u012B";
System.out.println("\nGhazālī url encoded by java = " + ghazali_jencoded);

System.out.println("\nGhazālī url encoded by javascript = " + ghazali_jsencoded);

String trevino = "Treviño";
String trevino_jencoded = URLEncoder.encode(trevino);
String trevino_jsencoded = "Trevi%F1o";
System.out.println("\nTreviño url encoded by java = " + trevino_jencoded);
System.out.println("\nTreviño url encoded by javascript = " + trevino_jsencoded);

String s[] = {control_jencoded, control_jsencoded, ghazali_jencoded, ghazali_jsencoded, trevino_jencoded, trevino_jsencoded};
for (int i = 0; i < s.length; i++) {
try {
String decoded = URLDecoder.decode(s[i]);
System.out.println("\n\"" + s[i] + "\" --> \"" + decoded + "\"");
} catch (Exception e) {
System.out.println("\n\"" + s[i] + "\" --> " + e);

The output looks like this:

control url encoded by java = this+is+a+control+string%3F%21

control url encoded by javascript = this%20is%20a%20control%20string%3F%21

Ghazālī url encoded by java = Ghaz%C4%81l%C4%AB

Ghazālī url encoded by javascript = Ghaz%u0101l%u012B

Treviño url encoded by java = Trevi%C3%B1o

Treviño url encoded by javascript = Trevi%F1o

"this+is+a+control+string%3F%21" --> "this is a control string?!"

"this%20is%20a%20control%20string%3F%21" --> "this is a control string?!"

"Ghaz%C4%81l%C4%AB" --> "Ghazālī"

"Ghaz%u0101l%u012B" --> java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in escape (%) pattern - For input string: "u0"

"Trevi%C3%B1o" --> "Treviño"

"Trevi%F1o" --> "Trevi?o"

So you can see, java can decode the control string regardless of who encoded it, but fails to decode the other two javascript encoded strings. “Treviño” is just decoded wrong, but the attempt to decode “GhazālÄ«” throws an exception.

The solution is that I now use java to url encode any queries I need to pass, and then the browser doesn’t attempt to encode them, and everything can de-code just fine. And happy ending, The Journal of Islamic Philosophy is now searchable.


Comment from Jonathan Rochkind
Time: June 13, 2008, 10:09 am

Encoding and escaping issues are predictably the most frustrating bugs I run into, and I do regularly run into them. Very very confusing to deal with multiple char encodings, or multiple escaping needs, or a need for _double_ or even _triple_ escaping (with different escaping conventions at each level). Can quickly become a nightmare.

The other day I had to write XML that would be processed by XSL to create javascript that would, based on other information in the XML output by the XSL, create a URL, which itself included in it some escaped possibly non-ascii content. Nightmare!

Comment from Erik Hetzner
Time: June 13, 2008, 11:31 am

According to wikipedia it is a common though incorrect for unicode to data to be percent encoded with %uxxxx. ( The correct way is to encode it with UTF-8 & then percent encode those octets.

In the case of Treviño it is %-encoding the latin-1 representation.

But it looks like the page you link to is using the escape() function, which is apparently no longer for use with URIs. Have you tried using the encodeURI function instead? It seems to work correctly, at least in Firefox.

Comment from pbinkley
Time: June 13, 2008, 12:19 pm

I notice you’re not explicitly telling java what encoding to use (like URLEncoder.encode(control, “UTF-8″) ). That might help secure the process, if you can get the javascript to use a single encoding consistently. I had a case the other day where some java running in an XSL was urlencoding in latin-1, which Solr didn’t like, until I told it to stick to utf-8.

Comment from casey
Time: June 16, 2008, 4:08 pm

Just FYI, your problem on the javascript side was probably due to using the escape() function instead of encodeURIComponent().

Write a comment