(question tldr at end)
So my task for the Mule "Transform Message" component is to take a bunch of user info from LDAP Directory Service and provide it to an old database endpoint using SOAP. Fairly simple transform stuff.
The main ! about this operation is the size of the message that has to be provided to the endpoint. The entire payload has to be provided in a single message, otherwise the service will remove all entries that are not part of the payload (there is no explicit 'delete' service). This is an issue because the amount of users in the directory is roughly 20,000 causing every message to be 5MB or so in size.
My flow in Mule Studio currently works with a low amount of users being returned from the LDAP component. Successful return from the endpoint and I can see the data updated in the legacy environment. When applying this to a more 'production-realistic' load the Web Service Consumer (SOAP) craps out with an odd exception (unexpected EOF/character).
So I stuck a File component in the middle to dumpcheck the message that was being sent to the Consumer. The message is indeed getting cut before it can finish, which is where the EOF is coming from.
This is the transform script in Dataweave.
%output application/xml
%namespace ns0 test.namespace.com
---
{
ns0#updateContact: {
ns0#ContactType: "Primary",
ns0#ContactDetails: {
(payload map {
(ns0#ContactDetailElem: {
ns0#personID: $.personID,
ns0#contactDetail: $.desc
}) when $.personID != null
})
}
}
}
The expected output is below and successfully occurs with a lesser payload.
<?xml version='1.0' encoding='windows-1252'?>
<ns0:updateContact xmlns:ns0="test.namespace.com">
<ns0:ContactType>Primary</ns0:ContactType>
<ns0:ContactDetails>
<../>
<ns0:ContactDetailElem>
<ns0:personID>{Integer}</ns0:personID>
<ns0:contactDetail>{String.detail}</ns0:contactDetail>
</ns0:ContactDetailElem>
<../>
</ns0:ContactDetails>
</ns0:updateContact>
On the big payload the following happens at the end of the file
<?xml version='1.0' encoding='windows-1252'?>
<ns0:updateContact xmlns:ns0="test.namespace.com">
<ns0:ContactType>Primary</ns0:ContactType>
<ns0:ContactDetails>
<../>
<ns0:ContactDetailElem>
<ns0:personID>{Integer}</ns0:personID>
<ns0:contactDetail>{String.detail}</ns0:contactDetail>
</ns0:ContactDeta
Which looks like a typo but is what looks like the message being cut before it can finish. The file size is always stopped at 3,553,099 characters. Of course this makes the flow crap out as the xml is invalid.
The question then is there a limit on the message size that the Dataweave transformer can create? If not a legitimate bug but a configuration issue, where would I find this setting? I've had a look around but can't find anybody encounter this type of issue.
TL;DR: Do Dataweave transform messages have a size limit around 3.38MB?
Exception caused by: com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
PS: I've found the documentation on dataweave streaming after typing this up, will see if this can help my situation. Otherwise i'm considering implementing a workaround to construct the message outside dataweave and then passing it to the Consumer.
