WSO2 Enterprise integrator (ESB) stuck every week

Question

I am using WSO2 Enterprise integrator (ESB) V6.3.0 to move files from one FTP server to another. There are around 10 very small files which needs to be moved daily at midnight.

But I found that, EI somehow stuck somewhere while moving the files and stops rest of the operation though it remains in running status. I have to restart EI every week on PROD which is not a good sign.

Kindly suggest if there are any configuration settings or any other thing which could fix this problem.

Attaching below one of the sample proxy service

 <?xml version="1.0" encoding="UTF-8"?>
<proxy name="SFTPFileTransferProxy" startOnLoad="true"
  trace="disable" transports="vfs" xmlns="http://ws.apache.org/ns/synapse">
  <target>
    <inSequence>
      <property expression="$trp:FILE_NAME" name="FILE_NAME"
        scope="default" type="STRING"/>
      <log level="custom">
        <property name="sequence" value="File transfer Proxy"/>
        <property expression="$ctx:FILE_NAME" name="FL2"/>
      </log>
    </inSequence>
    <outSequence/>
    <faultSequence/>
  </target>
  <parameter name="transport.PollInterval">300</parameter>
  <parameter name="transport.vfs.FileURI">vfs:sftp://server_name/out/xyz?transport.vfs.AvoidPermissionCheck=true</parameter>
  <parameter name="transport.vfs.ContentType">text/plain</parameter>
  <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
  <parameter name="transport.vfs.MoveAfterFailure">file:///C:/files/failed/xyz_in</parameter>
  <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
  <parameter name="transport.vfs.Locking">disable</parameter>
  <parameter name="transport.vfs.FileNamePattern">.*\.csv</parameter>
  <parameter name="transport.vfs.MoveAfterProcess">smb://server_in/files/xyz_in</parameter>
</proxy>

Attaching thread dump from server on github directory.

https://github.com/gadginir/wso2-esb

Shanaka Premarathna Shanaka Premarathna · Accepted Answer · 2020-07-06T18:31:53

You need to identify what is causing the ESB to stuck. A thread dump will help you to identify this. Please capture several thread dumps in the ESB node [1] and check whether the threads are continuously stuck.

Also please check in the carbon.log file if the error "java.lang.OutOfMemoryError: GC overhead limit exceeded" is present. This indicates whether the server has gone in to out of memory state. If it is present it will create a heap dump in the logs directory and analyzing the heap dump will let you know the possible cause of the issue.

[1]-https://docs.wso2.com/display/CLUSTER420/Troubleshooting+in+Production+Environments

Updated

I have analyzed the provided thread dumps (29072020_1.txt) and I could observe the vfs-Worker-9 has locked the resource 0x00000006c5842030 and it seems to be stuck in a loop on Session. There are several VFS threads waiting to lock the resource 0x00000006c5842030 and thus they are blocked.

vfs-Worker-9" #228 prio=5 os_prio=0 tid=0x000000002344f000 nid=0x1494 waiting on condition [0x0000000034eae000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at com.jcraft.jsch.Session.write(Session.java:1260)
    at com.jcraft.jsch.ChannelSftp.sendPacketPath(ChannelSftp.java:2602)
    at com.jcraft.jsch.ChannelSftp.sendPacketPath(ChannelSftp.java:2586)
    at com.jcraft.jsch.ChannelSftp.sendSTAT(ChannelSftp.java:2506)
    at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:2213)
    at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:2242)
    at com.jcraft.jsch.ChannelSftp.stat(ChannelSftp.java:2199)
    at org.apache.commons.vfs2.provider.sftp.SftpFileObject.statSelf(SftpFileObject.java:127)
    at org.apache.commons.vfs2.provider.sftp.SftpFileObject.doGetType(SftpFileObject.java:95)
    at org.apache.commons.vfs2.provider.AbstractFileObject.getType(AbstractFileObject.java:1285)
    - locked <0x00000006c5842030> (a org.apache.commons.vfs2.provider.sftp.SftpFileSystem)
    at org.apache.commons.vfs2.provider.sftp.SftpFileObject.refresh(SftpFileObject.java:79)
    at org.apache.commons.vfs2.provider.AbstractFileSystem.resolveFile(AbstractFileSystem.java:337)
    - locked <0x00000006c5842030> (a org.apache.commons.vfs2.provider.sftp.SftpFileSystem)

Update 2 Can you try to modify the vfs connection URL as follows. Here we have added vfs.passive=true [2].

vfs:sftp://server_name/out/xyz?transport.vfs.AvoidPermissionCheck=true&amp;vfs.passive=true

[2]-https://docs.wso2.com/display/EI650/VFS+Transport

WSO2 Enterprise integrator (ESB) stuck every week

1 Answers