I have data in Mongo DB in following collections: Client, Client_Organization, Payroll_Run & Worker. The relation among the Objects is Client & Olient_Organization are 1:M, Client_Organization & Payroll_Execution are 1:M. The Worker element is an array under the Client_Organization
The sample JSON to be generated is as below:
"Client": {
"name":"",
"ID":"", // other fields omitted for brevity
"Client_Organization": [{
"ID":"",
"name":"",
"address":"",
"Payroll_Execution": [{
"ID":"",
"date":"" // other fields
}] // end of Payroll_Execution
"worker": [{
// several nested JSON elements
"ID":"",
"payStubDetails": {
},
"dependents": [{
}],
"benefits": {
}
}]
}] // end Client_Org
} // end of Client
For a large client the number of elements in Worker array can be as much as 300K. I am assessing Spring Batch to generate such a JSON file. How can I apply chunk based processing on the Worker array element so that I do not overflow the memory.
Per my understanding if I join the data from the collections then I would only be processing 1 client's data and commit-interval would work on the root element of the JSON object (i.e. Client) passed to the writer. What can be other approaches to address this use case using Spring Batch?