I am running development environment for Confluent Kafka, Community edition on Windows, version 3.0.1-2.11. I am trying to achieve load balancing of tasks between 2 instances of connector. I am running Kafka Zookepper, Server, REST services and 2 instance of Connect distributed on the same machine. Only difference between properties file for connectors is rest port since they are running on the same machine. I don't create topics for connector offsets, config, status. Should I? I have custom code for sink connector.
When I create worker for my sink connector I do this by executing POST request
POST http://localhost:8083/connectors
toward any of the running connectors. Checking is there loaded worker is done at URL
GET http://localhost:8083/connectors
My sink connector has System.out.println() lines in code with which I can follow output of my code in the console log. When my worker is running I can see that only one instance of connector is executing code. If I terminate one connector another instance will take over the worker and execution will resume. However this is not what I want. My goal is that both connector instances are running worker code so that they can share the load between them. I've tried to got over some open source connectors to see is there specifics in writing code of connectors but with no success.
I've made some different attempts to tackle this problem but with no success. I could rewrite my business code to come around this but I'm pretty sure I'm missing on something not obvious for me. Recently I commented on Robin Moffatt's answer of this question.
