How does XPath deal with XML namespaces?

Question

How does XPath deal with XML namespaces?

If I use

/IntuitResponse/QueryResponse/Bill/Id

to parse the XML document below I get 0 nodes back.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<IntuitResponse xmlns="http://schema.intuit.com/finance/v3" 
                time="2016-10-14T10:48:39.109-07:00">
    <QueryResponse startPosition="1" maxResults="79" totalCount="79">
        <Bill domain="QBO" sparse="false">
            <Id>=1</Id>
        </Bill>
    </QueryResponse>
</IntuitResponse>

However, I'm not specifying the namespace in the XPath (i.e. http://schema.intuit.com/finance/v3 is not a prefix of each token of the path). How can XPath know which Id I want if I don't tell it explicitly? I suppose in this case (since there is only one namespace) XPath could get away with ignoring the xmlns entirely. But if there are multiple namespaces, things could get ugly.

Your XPath should not return any node : INFO - XPath returned 0 items (compiled in 0ms, evaluated in 1ms). How did you execute the XPath? — har07
@har07 I did it in Java using import javax.xml.xpath.XPath. I agree it doesn't work using an online tester. That was one of the perplexing things. — Adam
Excellent question! XPath itself provides no way to specify a default namespace or the binding of a namespace prefix to a namespace. Fortunately, however, hosting languages and libraries do. See my answer below for details... — kjhughes
I for one was impressed with this question because, unlike most previous askers, Adam not only included a minimal reproducible example, he sensed and conveyed the need for XPath to deal with XML namespaces somehow. Most such questions merely post an XPath, maybe some XML (and if we're lucky it's not an image or a link to a humongous off-site resource), and state that it "doesn't work." Adam sensed it had to do with namespaces, nailed the title, and wrote what I considered to be a question worthy of a canonical answer. — kjhughes
Possible duplicate of How to query XML using namespaces in Java with XPath? — rogerdpack

kjhughes kjhughes · Accepted Answer · 2016-11-25T00:58:49

Defining namespaces in XPath _{^{(recommended)}}

XPath itself doesn't have a way to bind a namespace prefix with a namespace. Such facilities are provided by the hosting library.

It is recommended that you use those facilities and define namespace prefixes that can then be used to qualify XML element and attribute names as necessary.

Here are some of the various mechanisms which XPath hosts provide for specifying namespace prefix bindings to namespace URIs.

^{_{(OP's original XPath, /IntuitResponse/QueryResponse/Bill/Id, has been elided to /IntuitResponse/QueryResponse.)}}

C#:

XmlNamespaceManager nsmgr = new XmlNamespaceManager(doc.NameTable);
nsmgr.AddNamespace("i", "http://schema.intuit.com/finance/v3");
XmlNodeList nodes = el.SelectNodes(@"/i:IntuitResponse/i:QueryResponse", nsmgr);

Java (SAX):

NamespaceSupport support = new NamespaceSupport();
support.pushContext();
support.declarePrefix("i", "http://schema.intuit.com/finance/v3");

Java (XPath):

xpath.setNamespaceContext(new NamespaceContext() {
    public String getNamespaceURI(String prefix) {
      switch (prefix) {
        case "i": return "http://schema.intuit.com/finance/v3";
        // ...
       }
    });

Remember to call DocumentBuilderFactory.setNamespaceAware(true).
See also: Java XPath: Queries with default namespace xmlns

JavaScript:

See Implementing a User Defined Namespace Resolver:

function nsResolver(prefix) {
  var ns = {
    'i' : 'http://schema.intuit.com/finance/v3'
  };
  return ns[prefix] || null;
}
document.evaluate( '/i:IntuitResponse/i:QueryResponse', 
                   document, nsResolver, XPathResult.ANY_TYPE, 
                   null );

Perl (LibXML):

my $xc = XML::LibXML::XPathContext->new($doc);
$xc->registerNs('i', 'http://schema.intuit.com/finance/v3');
my @nodes = $xc->findnodes('/i:IntuitResponse/i:QueryResponse');

Python (lxml):

from lxml import etree
f = StringIO('<IntuitResponse>...</IntuitResponse>')
doc = etree.parse(f)
r = doc.xpath('/i:IntuitResponse/i:QueryResponse', 
              namespaces={'i':'http://schema.intuit.com/finance/v3'})

Python (ElementTree):

namespaces = {'i': 'http://schema.intuit.com/finance/v3'}
root.findall('/i:IntuitResponse/i:QueryResponse', namespaces)

Python (Scrapy):

response.selector.register_namespace('i', 'http://schema.intuit.com/finance/v3')
response.xpath('/i:IntuitResponse/i:QueryResponse').getall()

PhP:

Adapted from @Tomalak's answer using DOMDocument:

$result = new DOMDocument();
$result->loadXML($xml);

$xpath = new DOMXpath($result);
$xpath->registerNamespace("i", "http://schema.intuit.com/finance/v3");

$result = $xpath->query("/i:IntuitResponse/i:QueryResponse");

See also @IMSoP's canonical Q/A on PHP SimpleXML namespaces.

Ruby (Nokogiri):

puts doc.xpath('/i:IntuitResponse/i:QueryResponse',
                'i' => "http://schema.intuit.com/finance/v3")

Note that Nokogiri supports removal of namespaces,

doc.remove_namespaces!

but see the below warnings discouraging the defeating of XML namespaces.

VBA:

xmlNS = "xmlns:i='http://schema.intuit.com/finance/v3'"
doc.setProperty "SelectionNamespaces", xmlNS  
Set queryResponseElement =doc.SelectSingleNode("/i:IntuitResponse/i:QueryResponse")

VB.NET:

xmlDoc = New XmlDocument()
xmlDoc.Load("file.xml")
nsmgr = New XmlNamespaceManager(New XmlNameTable())
nsmgr.AddNamespace("i", "http://schema.intuit.com/finance/v3");
nodes = xmlDoc.DocumentElement.SelectNodes("/i:IntuitResponse/i:QueryResponse",
                                           nsmgr)

SoapUI (doc):

declare namespace i='http://schema.intuit.com/finance/v3';
/i:IntuitResponse/i:QueryResponse

xmlstarlet:

-N i="http://schema.intuit.com/finance/v3"

XSLT:

<xsl:stylesheet version="1.0"
                xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:i="http://schema.intuit.com/finance/v3">
   ...

Note that if the default namespace has an associated namespace prefix defined, using the nsResolver() returned by Document.createNSResolver() can obviate the need for a customer nsResolver().

Once you've declared a namespace prefix, your XPath can be written to use it:

/i:IntuitResponse/i:QueryResponse

Defeating namespaces in XPath _{^{(not recommended)}}

An alternative is to write predicates that test against local-name():

/*[local-name()='IntuitResponse']/*[local-name()='QueryResponse']

Or, in XPath 2.0:

/*:IntuitResponse/*:QueryResponse

Skirting namespaces in this manner works but is not recommended because it

Under-specifies the full element/attribute name.
Fails to differentiate between element/attribute names in different namespaces (the very purpose of namespaces). Note that this concern could be addressed by adding an additional predicate to check the namespace URI explicitly¹:
```
 /*[    namespace-uri()='http://schema.intuit.com/finance/v3' 
    and local-name()='IntuitResponse']
 /*[    namespace-uri()='http://schema.intuit.com/finance/v3' 
    and local-name()='QueryResponse']
```
¹Thanks to Daniel Haley for the namespace-uri() note.
Is excessively verbose.

How does XPath deal with XML namespaces?

2 Answers

Defining namespaces in XPath (recommended)

Defeating namespaces in XPath (not recommended)

Defining namespaces in XPath _{^{(recommended)}}

Defeating namespaces in XPath _{^{(not recommended)}}