Kettle (pdi) - Open multiple files with Stax Step

Question

I have millions of xmls I need to read, transform and save to a database.

Using StAX Step I managed to open an xml file and do what I needed.

But the filename only let me open one file per step. Was there any way I can do the Stax Step open more than one xml file at a time with a regular expression, or other solution?

PS: Do not necessarily need to be parallel openings of xmls, just need a transformation that alone can open all xmls from a folder and run what I need.

hugs

Answer 1

I don't think you can do that because you can't just concatenate XML documents together. What you need to do make your transform load one file and then run that transform once for each file. See this answer:

https://stackoverflow.com/a/14613787/1515537

Brian

Answer 2

Open multiple files with Stax Step

Hello everybody,

Using the XML Input deprecated I could use regular expression in opening files.

thank you