Urlencoding data for post request body. Am I using wrong charset?

Question

I want to replicate a working POST request in Java. For testing purpose, lets take message like: 'äöõüäöõüäöõüäöõü'

Working POST request (with encoded message of 'äöõüäöõüäöõüäöõü'):

Header

POST http://www.mysite.com/newreply.php?do=postreply&t=477352 HTTP/1.1
Host: www.warriorforum.com
Connection: keep-alive
Content-Length: 403
Origin: http://www.mysite.com
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko)Chrome/14.0.835.202 Safari/535.1
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Accept: */*
Referer: http://www.mysite.com/test-forum/477352-test.html
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: bblastvisit=1319205053; bblastactivity=0; bbuserid=265374; bbpassword=1125e9ec1ab41f532ab8ec6f77ddaf94; bbsessionhash=91444317c100996990a04d6c5bbd8375;

Body

securitytoken=1319806096-618e5f9012901e2d818bf2c74c2121baa064be57&ajax=1&ajax_lastpost=1319806096&**message=%u00E4%u00F6%u00F5%u00FC%u00E4%u00F6%u00F5%u00FC%u00E4%u00F6%u00F5%u00FC%u00E4%u00F6%u00F5%u00FC**&wysiwyg=0&styleid=1&signature=1&fromquickreply=1&s=&do=postreply&t=477352&p=who%20cares&specifiedpost=0&parseurl=1&loggedinuser=265374

As we can see in the request body 'äöõüäöõüäöõüäöõü is encoded as: %u00E4%u00F6%u00F5%u00FC%u00E4%u00F6%u00F5%u00FC%u00E4%u00F6%u00F5%u00FC%u00E4%u00F6%u00F5%u00FC

Now i want to replicate it.

Lets Url encode the text with charset utf-8 in Java:

String userText = "äöõüäöõüäöõüäöõü";
String encoded = URLEncoder.encode(userText, "utf-8");

Result: %C3%A4%C3%B6%C3%B5%C3%BC%C3%A4%C3%B6%C3%B5%C3%BC%C3%A4%C3%B6%C3%B5%C3%BC%C3%A4%C3%B6%C3%B5%C3%BC%0A%0A%0A%5BSIZE%3D%221%22%5D%5BI%5D << NOT THE SAME

Lets try ISO-8859-1:

String userText = "äöõüäöõüäöõüäöõü";
String encoded = URLEncoder.encode(userText, "ISO-8859-1");

Result: %E4%F6%F5%FC%E4%F6%F5%FC%E4%F6%F5%FC%E4%F6%F5%FC%0A%0A%0A%5BSIZE%3D%221%22%5D%5BI%5D << NOT THE SAME

Neither of them produce the same encoded string as in the working example, but all of them have the same input. What am I missing here?

McDowell McDowell · Accepted Answer · 2011-10-28T13:57:01

%u00E4%u00F6%u00F5%u00FC%u00E4%u00F6%u00F5%u00FC%u00E4%u00F6%u00F5%u00FC%u00E4%u00F6%u00F5%u00FC

I don't know what the above data is encoded as, but it isn't application/x-www-form-urlencoded; charset=UTF-8 as the request claims. This is not legal data for this MIME type.

It looks like some UTF-16BE-encoded form.

URLEncoder.encode(userText, "utf-8"); would be the correct way to encode the application/x-www-form-urlencoded; charset=UTF-8 values if this was actually what the server was expecting. (ref)

Urlencoding data for post request body. Am I using wrong charset?

1 Answers