Use output_collections key in MLCP transform module

Question

I'm trying to use map:get($context,"collections") parameter in my MLCP transform function (input parameters described in MLCP Guide here). I want to use the collection specified in the -output_collections parameter so that I can insert it into my insert-update() function.

The doc loads to the database & mlcp output doesn't indicate an error, but when I look in the Query Console it shows (no collections) instead of COLLTEST. The transform also works correctly when I hardcode the collection (which I don't want to do). Am I using the map keys incorrectly?

mlcp-context-transform-test.xqy:

xquery version "1.0-ml";
module namespace tx = "http://transform-test";
import module namespace dls = 'http://marklogic.com/xdmp/dls' at '/MarkLogic/dls.xqy';

(:Function to Add document to DLS Library Services:)
declare function tx:insert-update($filename,$doc,$coll) {
let $i := dls:document-is-managed($filename) 
return
 if ($i = fn:false()) then     
    dls:document-insert-and-manage($filename,fn:true(),$doc/*,(),
(xdmp:permission('rest-reader', 'read'), xdmp:permission('rest-writer', 'insert')),($coll))
 else     
    if ($i = fn:true()) then
        dls:document-checkout-update-checkin(
      $filename,
      $doc/*,
      'CHECKIN-UPDATE-CHECKOUT',
      fn:true(),
      (xdmp:permission('rest-reader', 'read'), xdmp:permission('rest-writer', 'insert')),
      ($coll))
else    
    ()
};

declare function tx:transform(
 $content as map:map,
 $context as map:map
) as map:map* {

let $docnode := map:get($content, "value")
let $collections := map:get($context, "collections")
return
if (fn:empty($docnode/element()))
    then $content
    else 
        let $root := $docnode/*
        let $_:= 
            map:put($content, "value",
            document {$root/preceding-sibling::node(), 
            element {fn:name($root)} {
            $root/@*,
            $root/node(),
            element { xs:QName("metadata")} {
                    namespace {"docprop"} {"http://mynamespace"},
                    'foobarfoo'
                    }
            },
            $root/following-sibling::node()
            } )          
    
   return (map:put($content,"uri", 
    tx:insert-update(map:get($content, "uri"),map:get($content,"value"),
    map:get($context, "collections"))
            ),
            $content
            )
};

mlcp command:

mlcp.sh IMPORT -mode local \
-host localhost \
-port 8007 \
-username admin -password **** \
-input_file_path /MLCP-testdocs/testdocname.xml \
-output_uri_replace "/MLCP-testdocs,''" \
-output_uri_prefix /content/docs \
-output_uri_suffix .xml \
-output_collections COLLTEST \
-output_permissions rest-reader,read,rest-writer,insert \
-database top-songs \
-xml_repair_level full \
-transform_module /mlcp-context-transform-test.xqy \
-transform_namespace "http://transform-test" \
-document_type xml

I've also tried map:get($context,"output_collections"). The only $context parameter that ever works for me is "transform_param". I'm using mlcp 8.0.6.3 if that helps.

Looks like a correct approach to me. Can you try latest version of MLCP? You can use MLCP 9 against MarkLogic 8 and below.. — grtjn
@grtjn I tried using MLCP 9 and it still didn't work. I took a look at the hadoop.xqy module at /MarkLogic/hadoop.xqy and I noticed that $collections was not included in the $context map. So I tried adding let $dummy := map:put($context, "collections", fn:tokenize($collections, ',')[. ne '']) to hadoop.xqy and it's been working, but I'm not sure if it's the right approach or if it's ok to be editing this module. — finchmontana
What version of MakLogic do you have? Maybe try upgrading to latest patch release of your version.. — grtjn

Jane Chen Jane Chen · Accepted Answer · 2017-10-27T16:33:28

mlcp 8.0-6.3 does not allow a user to access/modify the document collection inside a transform.

https://github.com/marklogic/marklogic-contentpump/issues/34

The fix has a dependency on the server. So when you use mlcp 9.0-x to do this, you still need to connect to a server that's 8.0-6.4 or above.

The work-around on 8.0-6.3 is to perform the document insert inside your transform function and return empty sequence from the transform function.

Use output_collections key in MLCP transform module

1 Answers