With Kafka JDBC source connector getting only 1000 records/sec. How to improve the record pulling rate

3
votes

I am using kafka connect with JDBC source connector. The connector is working fine but i able to get only 1000 messages/sec. to the topic from Oracle DB. I tried most of the configuration settings but no luck. i tried in both standalone and distributed modes. Pls. help on this. Below is my JDBC Source connector configuration:

curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '{"name": "ORA_SRC_DEVDB",
"config": {                "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",                 
"connection.url": "jdbc:oracle:thin:@xxxxxxx/DBDEV",                 
"connection.user": "xxxxxx",                 
"connection.password": "xxxxxx",                 
"query": "select * from A.LOG_AUDIT",               
"topic.prefix": "Topic_POC", 
"tasks.max": "1",  
"poll.interval.ms": "5000", 
"batch.max.rows": "1000", 
"table.poll.interval.ms": "60000",  
"mode": "timestamp",    
"timestamp.column.name": "MODIFIED_DATEnTIME"                 }        
 }'

And also the destination Topic "Topic_POC" created with 3 partitions & 3 replicas.

1
apache-kafkaapache-kafka-connectconfluent-platform
batch.max.rows is 1000, and table.poll.interval.ms is 60000. What have you tried changing so far, and is the "1000" limitation your question/issue? Your 'problem' isn't clearly outlined, but we can assume you want to process more messages than you currently are? - gravity
curious - did you ever devise a configuration that would successfully poll more than 1000/sec ? - sairn

1 Answers

1
votes

poll.interval.ms: Frequency in ms to poll for new data in each table(default 5000)

batch.max.rows: Maximum number of rows to include in a single batch(default 100)

In your case every 5 seconds you are polling max 1000 record from DB. Trying to decrease poll.interval.ms and increase batch.max.rows could improve fetch rate.

Not only that below factors also impact on your fetch rate

  1. Rate of incoming data into the Database that also depends
  2. I/O rate from DB to JDBC connector to Kafka
  3. DB table performance if you have a proper index on the time column.
  4. After all its uses JDBC to fetch data from the database so implies all that you face on a single JDBC application

As per my experience JDBC connector is pretty fas