XQuery for remote XML

Question

Populating the baseX database with locally stored data is straight forward. But how would remote XML be fetched?

XQuery

The HTML Module provides a function for converting HTML to XML documents.

Documents can also be converted by specifying the parser and additional options in the query prolog:

declare option db:parser "html";
declare option db:htmlparser "html=false,nodefaults=true";
doc("index.html")

https://basex.readthedocs.io/en/search/Parsers/#xquery

To make concrete:

thufir@dur:~/basex$ 
thufir@dur:~/basex$ cat html_fetch_parse.xq 


fetch:xml("http://books.toscrape.com/", map {
  'parser': 'html',
  'htmlparser': map { 'html': false(), 'nodefaults': true() }
})

thufir@dur:~/basex$

But what if the document is already XML?

HTML is fetched reasonably easily. Surely XML is even simpler.

As far as I understand it, the fetch:xml does simply returns fetched nodes and it doesn't insert the nodes in any data base. As for simply loading XML, you can also use the doc function, e.g. use doc('http://example.com/foo.xml'). To work with data bases from XQuery, see the module docs.basex.org/wiki/Database_Module, for instance docs.basex.org/wiki/Database_Module#db:add or docs.basex.org/wiki/Database_Module#db:create e.g. db:create("DB", doc('http://example.com/foo.xml'), 'foo.xml'). — Martin Honnen

Thufir Thufir · Accepted Answer · 2019-01-02T03:54:41

This code populates a database in baseX correctly:

package org.basex.examples.local;

import java.net.MalformedURLException;
import java.net.URL;
import java.util.Iterator;
import java.util.Properties;
import java.util.logging.Logger;
import org.basex.core.BaseXException;
import org.basex.core.Context;
import org.basex.core.Databases;
import org.basex.core.cmd.CreateDB;
import org.basex.core.cmd.DropDB;
import org.basex.core.cmd.List;
import org.basex.core.cmd.Set;
import org.basex.util.list.StringList;

public class ScraperForXML {

    private static final Logger LOG = Logger.getLogger(App.class.getName());
    private Properties properties = new Properties();

    private ScraperForXML() {
    }

    public ScraperForXML(Properties properties) {
        this.properties = properties;
        LOG.fine(properties.toString());
    }

    public void fetch() throws BaseXException, MalformedURLException   {
        URL url = new URL(properties.getProperty("xmlURL"));
        String databaseName = properties.getProperty("databaseName");

        Context context = new Context();
        LOG.info(new List().execute(context));

        new Set("parser", "xml").execute(context);
        new CreateDB(databaseName, url.toString()).execute(context);


        Databases databases = context.databases();
        StringList stringListOfDatabases = databases.listDBs();
        String currentDatabaseName = null;

        Iterator<String> databaseIterator = stringListOfDatabases.iterator();

        while (databaseIterator.hasNext()) {
            currentDatabaseName=databaseIterator.next();
            LOG.info(currentDatabaseName);
            //not quite sure how to query a database...
        }


        new DropDB(databaseName).execute(context);
        context.close();
    }

}

although it's not an actual XQuery which is the goal.

XQuery for remote XML

1 Answers