Mixed record formats in a flat file with mule

Question

I have managed to confuse myself once more in mule.

I must accept a flat file from a vendor and cannot get the file reformatted. The file has a large number of records with fixed-width fields, perfect for the mule data-mapper. This works fine, mapping to a java POJO to create a collection of records to process.

The issue: The last record in the file is a summary record of a completely different format. The problem is, the format not only changes the data layout, the record length also changes. This exceptions the data-mapper as the record is too short for the format definition.

Are there any ideas on how to allow for this type of mixed formatting within a file, or instead of standard mule features do I need to make custom transformers to split the records and react to record lengths to separate formats?

Yes, I am aware this type of use of flat files is outdated, but dealing with legacy systems requires accepting the data as produced until the system can be retired.

Maybe you could just remove that last record before passing to datamapper, or do a custom transformer like you said. — Avanaur
The files are relatively straight forward, 20-30,000 fixed length records. Each start with a record control value and then a number of fixed length fields, perfect for a data mapper, until the last record. It starts with a special control value, and then instead of being 200 characters long is only 20-24, and does not retain the fixed length of the record, or even its last field. Not seeing any way for the out of package data mapper to handle this. — dlb

dlb dlb · Accepted Answer · 2015-08-19T18:36:26

I remain open to other solutions, but I doubt that the out of box data-mapper can achieve this. A bit too esoteric for a general solution I would think.

Pending hearing a brilliant work around, my current approach is to use a java component to split the file by record and extract the control value and pass through a choice control to separate the records by type. Java components are then called to map the fields and perform the business logic in each branch for the special record and the body records separately such as:

<file:connector name="File" autoDelete="true" streaming="true" 
         validateConnections="true" doc:name="File"/>
<spring:beans>  
    <spring:bean name="sumRec" class="com.mypackage.SummaryRecord" />                
    <spring:bean name="detailRec" class="com.mypackage.DetailRecord" />
</spring:beans>

<flow name="FlatToFile">
    <file:inbound-endpoint path="/SourceDir/Work"
        moveToDirectory="/SourceDir/Archive" 
        doc:name="FlatFileIn" connector-ref="File" >
        <file:filename-wildcard-filter pattern="*.txt" caseSensitive="false" />
    </file:inbound-endpoint>

    <component class="com.mypackage.FlatFileReader" doc:name="Split file and dispatch to VM" />

</flow>

<flow name="processRecords">
    <vm:inbound-endpoint exchange-pattern="one-way" path="in" doc:name="VM" />

    <object-to-string-transformer doc:name="Object to String"/>
    <set-variable variableName="recType" value="#message.payload.substring(0,7)]" doc:name="Variable"/>

    <choice doc:name="Choice">
        <when expression="flowVars.recType == '9999999'" >
            <expression-component doc:name="Expression">
                message.outboundProperties.recCount =     app.registry.sumRec.process(message.payload);
            </expression-component>
        </when>
        <otherwise>
            <expression-component doc:name="Expression">
                app.registry.detailRec.process(message.payload);
            </expression-component>
        </otherwise>
    </choice>
</flow>
</mule>

The VM endpoint is called from Java by the FlatFileReader.

Mixed record formats in a flat file with mule

2 Answers