Spring Integration File splitter lock

Question

I have a integration flow that reads files from a message channel called "promo", it then splits the file and does a transformation to a specific line of each file, it is then written by an ouboundAdapter line by line in append mode to a directory called "generated". I have then an inbound adapter that reads from the "generated" directory with a poller on fixed delay and moves the files to the destination location to be picked by a different application. How can I ensure that the files picked from the second flow are complete? I'm seeing cases were the inbound adapter moves to the destination files still being written by the first flow. Can a lock be added to the file that is being written to the "generated" directory so the second flow?

  private IntegrationFlow promoChannelSubscriber(File generatedDir, GenericSelector storesFilter) {
    return IntegrationFlows
      .from("promo")
      .filter(storesFilter)
      .split(
        Files.splitter()
          .applySequence(true)
      )
      .transform(this::enrichXstoreFileHeader)
      .handle(
        Files.outboundAdapter(generatedDir)
          .autoCreateDirectory(true)
          .fileExistsMode(APPEND)
          .appendNewLine(true)
      )
      .get();
  }

  private IntegrationFlow movePromotions(File generatedDir, Path destinationDirectory) {
    return IntegrationFlows.from(
      Files.inboundAdapter(generatedDir, comparingLong(File::lastModified))
        .regexFilter(fileNamePattern),
      cas -> cas.poller(
        Pollers.fixedDelay(2, TimeUnit.SECONDS)
          .maxMessagesPerPoll(promoConfig.getMaxMessagesPerPoll())
          .errorChannel(errorChannelName)
      ))
      .handle(
        Files.outboundAdapter(destinationDirectory.toFile())
          .deleteSourceFiles(true)
          .temporaryFileSuffix(".tmp")
      )
      .get();
  }

Thanks

Gary Russell Gary Russell · Accepted Answer · 2019-03-22T00:33:49

Why use 2 flows? Why not execute the second flow when the last split is received?

If you must use 2 flows, you need to write to a temporary file name that doesn't match the filter for the second flow.

Then, when the file is complete, rename the file to its final name.

You can use marker messages and a router to detect the end of file situation.

See the file-split-ftp sample application for an example.

Spring Integration File splitter lock

1 Answers