spring xd connection pool for module/job/filejdbc

0
votes

My use case is to process 50 csv files and load the same to MySQL DB using spring xd filejdbc job module.

For this I have configured datasource connection pool in spring-xd-1.0.0.RELEASE\xd\config\servers.yml like:

spring: datasource: url: jdbc:mysql://localhost:3306/spring_xd username: springxd password: springxd driverClassName: com.mysql.jdbc.Driver maxActive: 75 maxIdle: 10 minIdle: 10 initialSize: 10 maxWait: 30000 validationQuery: select 1 validationInterval: 30000 testOnBorrow: true testOnReturn: false testWhileIdle: false timeBetweenEvictionRunsMillis: 10000 minEvictableIdleTimeMillis: 60000 removeAbandoned: true removeAbandonedTimeout: 300 logAbandoned: true

When singlenode starts spring xd is initializing this pool.

But the problem is that, when the filejdbc job launches its once again initializing a connection pool with the above configs. Since \spring-xd-1.0.0.RELEASE\xd\config\modules\job\filejdbc\filejdbc.properties is using the same connection pool configs from servers.yml

The csv files data are loaded to the DB only using the filejdbc connectionpool and not the spring xd.

Because of this my db connections are running out.

Now, when I reduce the connection in servers.yml to max active 20, and when a job launches, spring xd is throwing connection pool exhaust exception, since i have 50 csv files to load.

I simply can't configure the servers.xml max active to 10 for the spring xd database and filejdbc maxactive to 60 for my custom database, so that 50 files can be loaded when the filejdbc job launches. This always throws connection exhaust exception.

Please advise.

Details as below: spring xd version - spring-xd-1.0.0.RELEASE in singlenode java version - jdk 7 environment - windows

1
spring-xd
How many containers is this job being run in? Also, why do you need one connection per file? - Michael Minella

1 Answers

0
votes

There is a bug in the local message bus where all partitions get executed simultaneously. In your case that causes the connection pool to get exhausted. We have opened a bug for this - https://jira.spring.io/browse/XD-2868.

Until this is fixed you could run with an another message bus like Redis or RabbitMQ and you should see your job complete. You can try by starting a Redis server and then start your singlenode with:

xd-singlenode --transport redis