MarkLogic : DHF Harmonize flow for writing multiple documents into FINAL db for each document in STAGING DB

Question

MarkLogic version : 9.0-6.2

I am using MLCP to ingest documents into STAGING DB. For each document in STAGING DB, I need to update multiple documents in the FINAL DB.

For example, the staging DB document has email with a bounce back indicator. I need to update bounce back indicator in all documents in a collection in FINAL that has the same email id.

Can I use Harmonize flow to do this? How can I code in content.sjs to call writer.sjs multiple times?

I tried to use a loop in writer.sjs to return each document in FINAL and apply documentInsert (basically, a FOR loop in writer.sjs), but the behavior is not consistent. Also, I am not sure if this is the right usage of Harmonize flow.

I tried to write custom code and invoke through REST API call. This is working as expected.

Please suggest the best way to handle my requirement (Harmonize Vs Custom REST).

Answer 1

There is one fairly elegant way of doing this, but it won't work from QuickStart UI. You can run harmonize flows with Final as source database, and use a cts:values as collector source.

You run your input and harmonize flows without side-effects, and once done, you start a separate harmonize flow that will look for all unique email addresses in the Final database. You pass each unique email address around as id (which normally contains a document uri, but any other string works too). You gather all you need in the various steps, and then in the writer you query and loop over all relevant docs to set the indicator as needed.

The gradle task to run a flow has a sourceDB property, but you can likely do the same with Data Hub REST-api as well: https://marklogic.github.io/marklogic-data-hub/refs/gradle-tasks/#marklogic-data-hub-flow-management-tasks

HTH!