How to externalize Landoop Kafka connector common properties

0
votes

I am using Landoop Kafka docker image. I am using its UI to create jdbc source and jdbc sink connector or I can also use Kafka Rest Api to create connectors. I have number of jdbc source connectors using same connection url, username, password. Whenever the source database credentials are changed, currently I have to change in all connectors individually. Is there a way I can externalize only these properties and let each connector have its specific properties like topic, query, etc.

1
apache-kafkaapache-kafka-connect

1 Answers

0
votes

Added in Kafka 2.0, you can externalize secrets, at least. Ref KIP-297.

I have yet to try it myself, and it might only be available for ConfigDef.Type.PASSWORD configurations, not just general ones...

Extract from the KIP

The patterns for variable substitutions are of the form ${provider:[path:]key}, where only one level of indirection is followed during substitutions.

The path in the variable is optional.

This means if you have the following:

foo=${file:bar}
bar=${file:baz}

and your file contains

bar=hello
baz=world

then the result will be

foo=hello
bar=world

So, in Docker, you could volume mount a config file, then reference it inside the connector config payload