What's valid and what's not in a URI query?

Question

Background (question further down)

I've been Googling this back and forth reading RFCs and SO questions trying to crack this, but I still don't got jack.

So I guess we just vote for the "best" answer and that's it, or?

Basically it boils down to this.

3.4. Query Component

The query component is a string of information to be interpreted by the resource.

query = *uric

Within a query component, the characters ";", "/", "?", ":", "@", "&", "=", "+", ",", and "$" are reserved.

The first thing that boggles me is that *uric is defined like this

uric = reserved | unreserved | escaped

reserved = ";" | "/" | "?" | ":" | "@" | "&" | "=" | "+" | "$" | ","

This is however somewhat clarified by paragraphs such as

The "reserved" syntax class above refers to those characters that are allowed within a URI, but which may not be allowed within a particular component of the generic URI syntax; they are used as delimiters of the components described in Section 3.

Characters in the "reserved" set are not reserved in all contexts. The set of characters actually reserved within any given URI component is defined by that component. In general, a character is reserved if the semantics of the URI changes if the character is replaced with its escaped US-ASCII encoding.

This last excerpt feels somewhat backwards, but it clearly states that the reserved character set depends on context. Yet 3.4 states that all the reserved characters are reserved within a query component, however, the only things that would change the semantics here is escaping the question mark (?) as URIs do not define the concept of a query string.

At this point I've given up on the RFCs entirely but found RFC 1738 particularly interesting.

An HTTP URL takes the form:

http://<host>:<port>/<path>?<searchpart>

Within the <path> and <searchpart> components, "/", ";", "?" are reserved. The "/" character may be used within HTTP to designate a hierarchical structure.

I interpret this at least with regards to HTTP URLs that RFC 1738 supersedes RFC 2396. Because the URI query has no notion of a query string also the interpretation of reserved doesn't really let allow me to define query strings as I'm used to doing by now.

Question

This all started when I wanted to pass a list of numbers together with the request of another resource. I didn't think much of it, and just passed it as a comma separated values. To my surprise though the comma was escaped. The query page.html?q=1,2,3 encoded turned into page.html?q=1%2C2%2C3 it works, but it's ugly and didn't expect it. That's when I started going through RFCs.

My first question is simply, is encoding commas really necessary?

My answer, according to RFC 2396: yes, according to RFC 1738: no

Later I found related posts regarding the passing of lists between requests. Where the csv approach was poised as bad. This showed up instead, (haven't seen this before).

page.html?q=1;q=2;q=3

My second question, is this a valid URL?

My answer, according to RFC 2396: no, according to RFC 1738: no (; is reserved)

I don't have any issues with passing csv as long as it's numbers, but yes you do run into the risk of having to encode and decode values back and forth if the comma suddenly is needed for something else. Anyway I tried the semi-colon query string thing with ASP.NET and the result was not what I expected.

Default.aspx?a=1;a=2&b=1&a=3

Request.QueryString["a"] = "1;a=2,3"
Request.QueryString["b"] = "1"

I fail to see how this greatly differs from a csv approach as when I ask for "a" I get a string with commas in it. ASP.NET certainly is not a reference implementation but it hasn't let me down yet.

But most importantly -- my third question -- where is specification for this? and what would you do or for that matter not do?

Answer 1

That a character is reserved within a generic URL component doesn't mean it must be escaped when it appears within the component or within data in the component. The character must also be defined as a delimiter within the generic or scheme-specific syntax and the appearance of the character must be within data.

The current standard for generic URIs is RFC 3986, which has this to say:

2.2. Reserved Characters

URIs include components and subcomponents that are delimited by characters in the "reserved" set. These characters are called "reserved" because they may (or may not) be defined as delimiters by the generic syntax, by each scheme-specific syntax, or by the implementation-specific syntax of a URI's dereferencing algorithm. If data for a URI component would conflict with a reserved character's purpose as a delimiter [emphasis added], then the conflicting data must be percent-encoded before the URI is formed.

   reserved    = gen-delims / sub-delims

   gen-delims  = ":" / "/" / "?" / "#" / "[" / "]" / "@"

   sub-delims  = "!" / "$" / "&" / "'" / "(" / ")"
               / "*" / "+" / "," / ";" / "="

3.3. Path Component

[...]

pchar         = unreserved / pct-encoded / sub-delims / ":" / "@"

[...]

3.4 Query Component

[...]

      query       = *( pchar / "/" / "?" )

Thus commas are explicitly allowed within query strings and only need to be escaped in data if specific schemes define it as a delimiter. The HTTP scheme doesn't use the comma or semi-colon as a delimiter in query strings, so they don't need to be escaped. Whether browsers follow this standard is another matter.

Using CSV should work fine for string data, you just have to follow standard CSV conventions and either quote data or escape the commas with backslashes.

As for RFC 2396, it also allows for unescaped commas in HTTP query strings:

2.2. Reserved Characters

Many URI include components consisting of or delimited by, certain special characters. These characters are called "reserved", since their usage within the URI component is limited to their reserved purpose. If the data for a URI component would conflict with the reserved purpose, then the conflicting data must be escaped before forming the URI.

Since commas don't have a reserved purpose under the HTTP scheme, they don't have to be escaped in data. The note from § 2.3 about reserved characters being those that change semantics when percent-encoded applies only generally; characters may be percent-encoded without changing semantics for specific schemes and yet still be reserved.

Answer 2

To answer what's valid in a query string I checked which special characters get replaced by chrome when making a request:

Space -> %20
! -> !
" -> %22
# -> removed, marks the end of the query string
% -> %
& -> &
' -> %27
( -> (
) -> )
* -> *
+ -> + (this usually means blank when received at the server, so encode if necessary)
, -> ,
- -> -
. -> .
/ -> /
: -> :
; -> ;
< -> %3C
= -> =
> -> %3E
? -> ?
@ -> @
[ -> [
\ -> \
] -> ]
^ -> ^
_ -> _
` -> `
{ -> {
| -> |
} -> }
~ -> ~

Extended ASCII (like °) -> Every character from this set is encoded

Note: That probably doesn't mean you shouldn't escape characters that didn't get replaced when you generate URIs for links. For instance it is often recommended not to use ~ in URIs because of compatibility issues but it's still a valid character.

Another example would be the plus-sign which is valid but usually treated as encoded blank when a server receives it as part of a request. Thus it should be encoded even if valid when it's purpose is to represent a plus and not a space.

So to answer what should be encoded: Invalid characters and characters that you want to treat literally but have a special meaning or can cause trouble at the server end.

Answer 3

Just use ?q=1+2+3

I am answering here a fourth question :) that did not ask but all started with: how do i pass list of numbers a-la comma-separated values? Seems to me the best approach is just to pass them space-separated, where spaces will get url-form-encoded to +. Works great, as longs as you know the values in the list contain no spaces (something numbers tend not to).

Answer 4

page.html?q=1;q=2;q=3

is this a valid URL?

Yes. The ; is reserved, but not by an RFC. The context that defines this component is the definition of the application/x-www-form-urlencoded media type, which is part of the HTML standard (section 17.13.4.1). In particular the sneaky note hidden away in section B.2.2:

We recommend that HTTP server implementors, and in particular, CGI implementors support the use of ";" in place of "&" to save authors the trouble of escaping "&" characters in this manner.

Unfortunately many popular server-side scripting frameworks including ASP.NET do not support this usage.

Answer 5

I would like to note that page.html?q=1&q=2&q=3 is a valid url as well. This is a completely legitimate way of expressing an array in a query string. Your server technology will determine how exactly that is presented.

In Classic ASP, you check Response.QueryString("q").Count and then use Response.QueryString("q")(0) (and (1) and (2)).

Note that you saw this in your ASP.NET, too (I think it was not intended, but look):

Default.aspx?a=1;a=2&b=1&a=3

Request.QueryString["a"] = "1;a=2,3"
Request.QueryString["b"] = "1"

Notice that the semicolon is ignored, so you have a defined twice, and you got its value twice, separated by a comma. Using all ampersands Default.aspx?a=1&a=2&b=1&a=3 will yield a as "1,2,3". But I am sure there is a method to get each individual element, in case the elements themselves contain commas. It is simply the default property of the non-indexed QueryString that concatenates the sub-values together with comma separators.

Answer 6

I had the same issue. The URL that was hyperlinked was a third party URL and was expecting a list of parameters in format page.html?q=1,2,3 ONLY and the URL page.html?q=1%2C2%2C3 did not work. I was able to get it working using javascript. May not be the best approach but can check out the solution here if it helps anyone.

Answer 7

If you are sending the ENCODED characters to FLASH/SWF file, then you should ENCODE the character twice!! (because of Flash parser)