0
votes

I have data in Mongo DB in following collections: Client, Client_Organization, Payroll_Run & Worker. The relation among the Objects is Client & Olient_Organization are 1:M, Client_Organization & Payroll_Execution are 1:M. The Worker element is an array under the Client_Organization

The sample JSON to be generated is as below:

"Client": {
  "name":"",
  "ID":"", // other fields omitted for brevity
  "Client_Organization": [{
    "ID":"",
    "name":"",
    "address":"",
    "Payroll_Execution": [{
       "ID":"",
       "date":"" // other fields
        }] // end of Payroll_Execution
    "worker": [{
        // several nested JSON elements
        "ID":"",
        "payStubDetails": {
            },
        "dependents": [{
            }],
        "benefits": {
            }
    }]
    }] // end Client_Org 
} // end of Client

For a large client the number of elements in Worker array can be as much as 300K. I am assessing Spring Batch to generate such a JSON file. How can I apply chunk based processing on the Worker array element so that I do not overflow the memory.

Per my understanding if I join the data from the collections then I would only be processing 1 client's data and commit-interval would work on the root element of the JSON object (i.e. Client) passed to the writer. What can be other approaches to address this use case using Spring Batch?

1
I added an answer that I believe answers your question. Please accept it or upvote it: stackoverflow.com/help/someone-answers. Otherwise, please let me know what it still needs to be accepted. Thank you. - Mahmoud Ben Hassine

1 Answers

1
votes

The JsonFileItemWriter generates a json file with the following format:

[
   {jsonObject1},
   {jsonObject2},
   ...
   {jsonObjectN}
]

Items will be marshalled to json using a JsonObjectMarshaller (Spring Batch provides marshallers based on Jackson and Gson out-of-the-box). So if you are planning to use this writer, you need to make sure that items correspond to "root" json objects in the output file. In your case, if items are Client instances, you need to do a join to prepare items with all details that need to be marshalled as part of root json objects.