I am new to Kettle. I want to know if the table input step will load all the data from the table into memory at once. If yes, how do I avoid a memory overflow?
1 Answers
0
votes
If it worked like that how would people use it in a big data scenario? :) Don't worry it will be fine. PDI doesn't the data into local memory if it can be avoided. That's the same for all input steps. It creates a cursor (like a stream) and just gets the data you work currently on and then releases it from memory when it reaches the end of the pipeline.
