Xquery recursive query does not set variable

Question

[edited with fuller example of success vs. error]

In Xquery 3 (eXist 4.7) I am working with a public API (Zotero) that provides a bibliographic list using this GET request : https://api.zotero.org/groups/2304628/items?format=atom&content=tei&v=3

The provider chunks the responses into 25 items each response (3202 expected, as indicated in the first response), so that I have to fetch the next 25, 25, 25... in a loop using parameters. The API response helpfully provides the full URLs with parameters to make the next request:

 <link rel="self" type="application/atom+xml"
            href="https://api.zotero.org/groups/2304628/items?content=tei&amp;format=atom"/>
 <link rel="next" type="application/atom+xml"
            href="https://api.zotero.org/groups/2304628/items?content=tei&amp;format=atom&amp;start=25"/>
 <link rel="last" type="application/atom+xml"
            href="https://api.zotero.org/groups/2304628/items?content=tei&amp;format=atom&amp;start=3200"/>

I am trying to build a query which recursively sends a GET for the next URL, and each 'recursion' checks to see if the $current-url is the same as the $last-url. When they match, end the recursion.

The following produces the error err:XPDY0002 variable '$next-url' is not set

xquery version "3.1";

module namespace zotero="/db/apps/thema/modules/zotero";
declare namespace tei="http://www.tei-c.org/ns/1.0";
declare namespace atom = "http://www.w3.org/2005/Atom";

declare function zotero:get-recursive($current-url as xs:string)
{
  let $APIdoc := httpclient:get($current-url,true(),<headers/>)
  let $next-url := $APIdoc//atom:link[@rel="next"]/data(@href)
  let $last-url := $APIdoc//atom:link[@rel="last"]/data(@href)

  (: perform db insert from API data:)
  let $bibdoc := doc("db/apps/myapp/data/list_bibliography.xml")
  let $insert-doc := for $content in $APIdoc//atom:content
                let $x := parse-xml($content/text())
                return update insert $x//tei:biblStruct into $bibdoc//tei:listBibl

  return 
        if ($current-url = $last-url)
            then "finished"
            else zotero:get-recursive($next-url)         
};

Removing the recursive function successfully inserts the data and returns the correct next-url:

xquery version "3.1";

module namespace zotero="/db/apps/thema/modules/zotero";
declare namespace tei="http://www.tei-c.org/ns/1.0";
declare namespace atom = "http://www.w3.org/2005/Atom";

declare function zotero:get-recursive($current-url as xs:string)
{
  let $APIdoc := httpclient:get($current-url,true(),<headers/>)
  let $next-url := $APIdoc//atom:link[@rel="next"]/data(@href)
  let $last-url := $APIdoc//atom:link[@rel="last"]/data(@href)

  let $bibdoc := doc("db/apps/myapp/data/list_bibliography.xml")
  let $insert-doc := for $content in $APIdoc//atom:content
                let $x := parse-xml($content/text())
                return update insert $x//tei:biblStruct into $bibdoc//tei:listBibl 
  return ($insert-doc, $next-url)
};

Is there something in xquery recursion that interferes with variable setting/use? Or am I approaching this entirely wrong?

Many thanks.

Do you get that error with the code as shown? Or do you only get it if the comment (: perform my db insert here :) is spelled out with some XQuery code you have not shown that perhaps changes the scope of the previous lets so that the last return has a different meaning? — Martin Honnen
@MartinHonnen I've updated the examples to be more explicit. The insert does not appear to change the scope of the let statements. — jbrehr
I tried working through query recursion per a demo by @joewiz at gist.github.com/joewiz/6762f1d8826fc291c3884cce3634eb77 but I can't identify how to integrate the call back with I'm trying to do. — jbrehr
I have tried your original code (in a slightly different variant using the local prefix for the function and not declaring a module) at the eXide online tester with the main query local:get-recursive('https://api.zotero.org/groups/2304628/items?format=atom&content=tei&v=3') and although it takes a long time to finish it does finally return. I can't see anything wrong with your edited code either so I guess you have to wait until someone with eXist db experience shows up to tell you more. — Martin Honnen
I'm not sure what to say - I tried it in local just now and it ran fine. I rebuilt the function and now it works too. — jbrehr

line-o line-o · Accepted Answer · 2019-08-09T14:39:23

I would switch to a different http-client: http://expath.org/modules/http-client/

This one is recommended by the community to use since exist version 4.1+.


declare function zotero:get-recursive($current-url as xs:string)
{
  let $response := http:send-request(<http:request href="{$current-url}" method="get" />)
  (: try catch or other error handling would be good here :)
  (: assuming status 200 :)
  let $APIdoc := $response[2]
  let $next-url := $APIdoc//atom:link[@rel="next"]/data(@href)
  let $last-url := $APIdoc//atom:link[@rel="last"]/data(@href)

  (: perform db insert from API data:)
  let $bibdoc := doc("db/apps/myapp/data/list_bibliography.xml")
  let $insert-doc := for $content in $APIdoc//atom:content
                let $x := parse-xml($content/text())
                return update insert $x//tei:biblStruct into $bibdoc//tei:listBibl

  return 
        if ($current-url = $last-url)
            then "finished"
            else zotero:get-recursive($next-url)         
};

Xquery recursive query does not set variable

1 Answers