Dynamically reading column headers to pass to ETL Metadata Injection step

0
votes

Having difficulty understanding how to dynamically pass the headers of any csv to the Metadata Injection step.

I mostly get how it works. However, the examples out there create the field definitions from the 'Meta' tab of a data grid, and pass the column headers via the 'Data' tab of the same data grid. Thus, you have to know the column names, which defeats my purpose.

How can I continue to pass the field definitions from the data grid, but pass the header names dynamically?

1
pentahopentaho-data-integration
Data Grid step is more appropriate for small static data and illustration purpose. In a production environment, I'd suggest you avoid Data Grid step and use YAML Input step with which you can easily maintain all configuration in one place. also add the YAML filename as a parameter for more flexibility. – jxc

1 Answers

2
votes

You need to read the file in two passes:

  1. The injector transformation reads the header line only, and parses it to determine the list of available fields (and possibly with a data dictionary to define data types, formats, etc.)

  2. A template transformation that reads from the file, but doesn't have any field information. this will be injected on run time.

Here's a working example: http://ubiquis.co.uk/pdi/loading-csv-files-with-pdi-metadata-injection/