Spring Batch Footer Validation

0
votes

I am using Spring batch for processing a file with a header, detail and footer records. The footer contains the total number of records in the file. If the detail record count dosent match the count in the footer, the file should not be processed.

I am using a Custom Line Tokenizer that processes the header, detail and footer record. When the footer record is encountered, if the count dosent match the detail record count, I am throwing an exception.

But the problem I am facing is if the chunk size is set to small numbers like 10 and the file has 20 records, the first 10 detail records are being persisted into the DB, even though the footer count dosent match the total number of records.

Is there a way to validate the footer count with the number of records in the file before the call to the Writer?

Thanks.

2
spring-batch
If file is small you can set an high commit-interval to force a single chunk processione, but is a dirty way – Luca Basso Ricci
Any other better way like prevalidation ? I dont want to set the chunk size too high. – jyn
Do you know how many bytes long is the last record? – Luca Basso Ricci
@jyn there are many options. I would write a tasklet that just reads the file and make the validation, if validation passes I would go to next step the one you have now and make a normal insert. – user1121883
What happen if number of found records matches footer's number but write of some record fails? Is the file still correct? – Luca Basso Ricci

2 Answers

0
votes

What you need is a reader with a footer callback handler defined. I had faced a similar problem and this link helped me a lot! See the last post by Atefeh Zareh. He has also included the xml configuration.

And regarding the first ten being persisted, you can have another validation step before the main processing step which will just check the header and trailer counts. Do not write any persisting logic in the writer. If the count fails, stop the job so that it does not go into the processing step.

0
votes

By writing our own Item Reader as well as Item classes to handle Header,Footer,Data records and finding the counts of Header,Footer,Data records

ItemReader Class

public class AggregateItemReader<T> implements ItemStreamReader<ResultHolder> {


private ItemStreamReader<AggregateItem<T>> itemReader;

@Override
public ResultHolder read() throws Exception {
    ResultHolder holder = new ResultHolder();

    while (process(itemReader.read(), holder)) {
        continue;
    }

    if (!holder.isExhausted()) {
        return holder;
    }
    else {
        return null;
    }
}

private boolean process(AggregateItem<T> value, ResultHolder holder) {
    // finish processing if we hit the end of file
    if (value == null) {
        LOG.debug("Exhausted ItemReader");
        holder.setExhausted(true);
        return false;
    }

    // start a new collection
    if (value.isHeader()) {
        LOG.debug("Header Record detected");
        holder.addHeaderRecordCount();
        return true;
    }

    // mark we are finished with current collection
    if (value.isFooter()) {
        LOG.debug("Tailer Record detected");
        holder.addTailerRecordCount();
        holder.setFiledRecordCount(value.getFieldSet().readInt(3));
        System.out.println("###########################################"+holder.getDataRecordCount()+"############################################");
        return false;
    }

    // add a simple record to the current collection

    holder.addDataRecordCount();
    return true;
}

And Item Class is

public class AggregateItem<T> {

@SuppressWarnings("unchecked")
public static <T> AggregateItem<T> getData(FieldSet fs) {
    return new AggregateItem(fs, false, false, true);
}

@SuppressWarnings("unchecked")
public static <T> AggregateItem<T> getFooter(FieldSet fs) {
    return new AggregateItem(fs, false, true, false);
}


@SuppressWarnings("unchecked")
public static <T> AggregateItem<T> getHeader(FieldSet fs) {
    return new AggregateItem(fs, true, false, false);
}

private boolean data = false;
private FieldSet fieldSet;

private boolean footer = false;

private boolean header = false;

private T item;

public AggregateItem(FieldSet fs, boolean header, boolean footer, boolean data) {
    this(null);
    this.header = header;
    this.footer = footer;
    this.data = data;
    this.fieldSet = fs;
}


public AggregateItem(T item) {
    super();
    this.item = item;
}

public FieldSet getFieldSet() {
    return fieldSet;
}


public T getItem() {
    return item;
}

public boolean isData() {
    return data;
}


public boolean isFooter() {
    return footer;
}


public boolean isHeader() {
    return header;
}

}

And ResultHolder class is

public class ResultHolder implements {
private Integer headerRecordCount = 0;
private Integer dataRecordCount = 0;
private Integer tailerRecordCount = 0;
private Integer filedRecordCount;//this is to save record count given in source File
private boolean exhausted = false;//setters & getters

}

If any doubts feel free to mail at [email protected]