Is there a technique for “streaming” to an HTTP Reply Node in an IIB message flow?

0
votes

I'm a newbie and I have an IIB message flow that looks like this:

HTTP Input Node -> File Read Node -> HTTP Reply Node.

(1) HTTP Input Node - flow is started by a user initiating an https request. (2) File Read Node - message flow gets a file from our ftp server (local file directory, file name, sftp host and credentials, and sftp server directory all temporarily hardcoded just for ease in troubleshooting a solution, but in real life this info would get extracted from the http request). (3) HTTP Reply Node - reply to user with contents of file retrieved from the ftp server.

The problem is this: The message flow causes a "java/lang/OutOfMemoryError" heap dump when the File Read Node reads large files (I've been testing by running multiple instances of the message flow with a 124 MG file).

Background: This is a legacy ASP/VBScript application that was rewritten in IIB. Not by me. But I've been tasked with fixing the memory error.

More Info: From reading, I understand why the committed memory gets all used up. I've read all I can on large message processing but because this application's main purpose is to allow users to pick up files from our ftp server via ** https **, I seem to be tied to that HTTP Reply Node and thus the entire file has to be available in the MBMessageAssembly object (and therefore in memory) when the flow hits the "in" terminal of the HTTP Reply Node.

My question is this: Is there a technique for "streaming" to the HTTP Reply Node? Or, is it possible to remove the HTTP Reply Node altogether and do the SFTP and the HTTP Reply in a Java Compute Node (I can't use a Compute Node/ESQL since we aren't licensed for that)? Or, any other ideas?

1
ibm-integration-bus

1 Answers

0
votes

Good problem statement, and thanks for doing the legwork before posting your question. The fundamental problem here is that IIB is designed for processing messages, not (huge) files. You can experiment with increasing the heap size for all of

  • the HTTP listener (broker-wide or EG-specific, be sure to adjust the correct one).
  • the execution group
  • the JVM.

Note that IIB does not store the message tree itself in the JVM heap, even when you are using the JavaCompute node to manipulate it. But the FileRead node uses the JVM, and it clearly needs more Java heap.