Take this line as just one example:
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
</configuration>
Some brief research has taught me that nothing necessarily has to be hosted at that URL. Therefore, if it's just a reference to a namespace contained in a local assembly, why use URLs instead of regular namespaces like the ones typically used by .NET (System.Data, for example)?
When you ask why a standard is the way it is, there are two possible interpretations of your question: (a) what are the perceived benefits of the design choice that was made, and (b) what was the historical sequence of events that led to this design being adopted over other designs.
Clearly, using HTTP-like URIs gives uniqueness. But that could also have been achieved in other ways, for example by inventing a new URI scheme. I think that if you trace the arguments that were made at the time, you will find that many of the people advocating the use of HTTP-style URIs were also advocating that namespace URIs should be dereferencable, perhaps to a schema or to some other kind of document. Indeed, it's W3C policy for its own namespaces that there is always a document that can be retrieved when you type the namespace URI into a browser, and that it should tell you something useful about the namespace.
So I suspect the fact that we use URIs that look deferenceable but don't define any semantics for what happens when they are dereferenced is in fact the result a committee compromise between two or more opposing camps. But you'd have to do some proper historical research to confirm that.
Just like namespaces in .NET, namespaces in XML documents are designed to help prevent and resolve name collisions, especially if XML documents are from different organizations and/or domains of knowledge.
XML document format has been stylized by the same people responsible for HTML, so you may find syntactical similarities in elements of the format. Semantically, the dotNET and XML namespaces serve the same goal.
The core of the problem is that namespaces have to be URIs. There is no good reason for such a thing since all it needs to be is a unique identifier to give context for the defined elements.
Since they have to be URIs, all possibilities are senseless, because a scheme is needed (e.g. http) even though a namespace has nothing to do with such a thing.
So to answer the question of why: like all bad ideas, someone thought it was a good idea at the time.
