URL decoding in Java for non-ASCII characters

Question

I'm trying in Java to decode URL containing % encoded characters

I've tried using java.net.URI class to do the job, but it's not always working correctly.

String test = "https://fr.wikipedia.org/wiki/Fondation_Alliance_fran%C3%A7aise";
URI uri = new URI(test);
System.out.println(uri.getPath());

For the test String "https://fr.wikipedia.org/wiki/Fondation_Alliance_fran%C3%A7aise", the result is correct "/wiki/Fondation_Alliance_française" (%C3%A7 is correctly replaced by ç).

But for some other test strings, like "http://sv.wikipedia.org/wiki/Anv%E4ndare:Lsjbot/Statistik#Drosophilidae", it gives an incorrect result "/wiki/Anv�ndare:Lsjbot/Statistik" (%E4 is replaced by � instead of ä).

I did some testing with getRawPath() and URLDecoder class.

System.out.println(URLDecoder.decode(uri.getRawPath(), "UTF8"));
System.out.println(URLDecoder.decode(uri.getRawPath(), "ISO-8859-1"));
System.out.println(URLDecoder.decode(uri.getRawPath(), "WINDOWS-1252"));

Depending on the test String, I get correct results with different encodings:

• For %C3%A7, I get a correct result with "UTF-8" encoding as expected, and incorrect results with "ISO-8859-1" or "WINDOWS-1252" encoding
• For %E4, it's the opposite.

For both test URL, I get the correct page if I put them in Chrome address bar.

How can I correctly decode the URL in all situations ? Thanks for any help

==== Answer ====

Thanks to the suggestions in McDowell answer below, it now seems to work. Here's what I now have as code:

private static void appendBytes(ByteArrayOutputStream buf, String data) throws UnsupportedEncodingException {
  byte[] b = data.getBytes("UTF8");
  buf.write(b, 0, b.length);
}

private static byte[] parseEncodedString(String segment) throws UnsupportedEncodingException {
  ByteArrayOutputStream buf = new ByteArrayOutputStream(segment.length());
  int last = 0;
  int index = 0;
  while (index < segment.length()) {
    if (segment.charAt(index) == '%') {
      appendBytes(buf, segment.substring(last, index));
      if ((index < segment.length() + 2) &&
          ("ABCDEFabcdef0123456789".indexOf(segment.charAt(index + 1)) >= 0) &&
          ("ABCDEFabcdef0123456789".indexOf(segment.charAt(index + 2)) >= 0)) {
        buf.write((byte) Integer.parseInt(segment.substring(index + 1, index + 3), 16));
        index += 3;
      } else if ((index < segment.length() + 1) &&
                 (segment.charAt(index + 1) == '%')) {
        buf.write((byte) '%');
        index += 2;
      } else {
        buf.write((byte) '%');
        index++;
      }
      last = index;
    } else {
      index++;
    }
  }
  appendBytes(buf, segment.substring(last));
  return buf.toByteArray();
}

private static String parseEncodedString(String segment, Charset... encodings) {
  if ((segment == null) || (segment.indexOf('%') < 0)) {
    return segment;
  }
  try {
    byte[] data = parseEncodedString(segment);
    for (Charset encoding : encodings) {
      try {
        if (encoding != null) {
          return encoding.newDecoder().
              onMalformedInput(CodingErrorAction.REPORT).
              decode(ByteBuffer.wrap(data)).toString();
        }
      } catch (CharacterCodingException e) {
        // Incorrect encoding, try next one
      }
    }
  } catch (UnsupportedEncodingException e) {
    // Nothing to do
  }
  return segment;
}

Answer 1

Anv%E4ndare

As PopoFibo says this is not a valid UTF-8 encoded sequence.

You can do some tolerant best-guess decoding:

public static String parse(String segment, Charset... encodings) {
  byte[] data = parse(segment);
  for (Charset encoding : encodings) {
    try {
      return encoding.newDecoder()
          .onMalformedInput(CodingErrorAction.REPORT)
          .decode(ByteBuffer.wrap(data))
          .toString();
    } catch (CharacterCodingException notThisCharset_ignore) {}
  }
  return segment;
}

private static byte[] parse(String segment) {
  ByteArrayOutputStream buf = new ByteArrayOutputStream();
  Matcher matcher = Pattern.compile("%([A-Fa-f0-9][A-Fa-f0-9])")
                          .matcher(segment);
  int last = 0;
  while (matcher.find()) {
    appendAscii(buf, segment.substring(last, matcher.start()));
    byte hex = (byte) Integer.parseInt(matcher.group(1), 16);
    buf.write(hex);
    last = matcher.end();
  }
  appendAscii(buf, segment.substring(last));
  return buf.toByteArray();
}

private static void appendAscii(ByteArrayOutputStream buf, String data) {
  byte[] b = data.getBytes(StandardCharsets.US_ASCII);
  buf.write(b, 0, b.length);
}

This code will successfully decode the given strings:

for (String test : Arrays.asList("Fondation_Alliance_fran%C3%A7aise",
    "Anv%E4ndare")) {
  String result = parse(test, StandardCharsets.UTF_8,
      StandardCharsets.ISO_8859_1);
  System.out.println(result);
}

Note that this isn't some foolproof system that allows you to ignore correct URL encoding. It works here because v%E4n - the byte sequence 76 E4 6E - is not a valid sequence as per the UTF-8 scheme and the decoder can detect this.

If you reverse the order of the encodings the first string can happily (but incorrectly) be decoded as ISO-8859-1.

---

Note: HTTP doesn't care about percent-encoding and you can write a web server that accepts http://foo/%%%%% as a valid form. The URI spec mandates UTF-8 but this was done retroactively. It is really up to the server to describe what form its URIs should be and if you have to handle arbitrary URIs you need to be aware of this legacy.

I've written a bit more about URLs and Java here.