Working on setting up Kafka running from our RDS Postgres 9.6 to Redhift. Using the guidelines at https://blog.insightdatascience.com/from-postgresql-to-redshift-with-kafka-connect-111c44954a6a and we have the all of the infrastructure set up, and am working on fully setting up Confluent. I'm getting the error of ava.lang.IllegalArgumentException: Number of groups must be positive. when trying to set stuff up. Here's my config file:
name=source-postgres
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=16
connection.url= ((correct url and information here))
mode=timestamp+incrementing
timestamp.column.name=updated_at
incrementing.column.name=id
topic.prefix=postgres_
Full error:
/usr/local/confluent$ /usr/local/confluent/bin/connect-standalone /usr/local/confluent/etc/schema-registry/connect-avro-standalone.properties /usr/local/confluent/etc/kafka-connect-jdbc/source-postgres.properties SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/confluent/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/confluent/share/java/kafka-connect-elasticsearch/slf4j-simple-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/confluent/share/java/kafka-connect-hdfs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/confluent/share/java/kafka/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] [2018-01-29 16:49:49,820] INFO StandaloneConfig values: access.control.allow.methods = access.control.allow.origin = bootstrap.servers = [localhost:9092] internal.key.converter = class org.apache.kafka.connect.json.JsonConverter internal.value.converter = class org.apache.kafka.connect.json.JsonConverter key.converter = class io.confluent.connect.avro.AvroConverter offset.flush.interval.ms = 60000 offset.flush.timeout.ms = 5000 offset.storage.file.filename = /tmp/connect.offsets rest.advertised.host.name = null rest.advertised.port = null rest.host.name = null rest.port = 8083 task.shutdown.graceful.timeout.ms = 5000 value.converter = class io.confluent.connect.avro.AvroConverter (org.apache.kafka.connect.runtime.standalone.StandaloneConfig:180) [2018-01-29 16:49:49,942] INFO Logging initialized @549ms (org.eclipse.jetty.util.log:186) [2018-01-29 16:49:50,301] INFO Kafka Connect starting (org.apache.kafka.connect.runtime.Connect:52) [2018-01-29 16:49:50,302] INFO Herder starting (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:70) [2018-01-29 16:49:50,302] INFO Worker starting (org.apache.kafka.connect.runtime.Worker:113) [2018-01-29 16:49:50,302] INFO Starting FileOffsetBackingStore with file /tmp/connect.offsets (org.apache.kafka.connect.storage.FileOffsetBackingStore:60) [2018-01-29 16:49:50,304] INFO Worker started (org.apache.kafka.connect.runtime.Worker:118) [2018-01-29 16:49:50,305] INFO Herder started (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:72) [2018-01-29 16:49:50,305] INFO Starting REST server (org.apache.kafka.connect.runtime.rest.RestServer:98) [2018-01-29 16:49:50,434] INFO jetty-9.2.15.v20160210 (org.eclipse.jetty.server.Server:327) Jan 29, 2018 4:49:51 PM org.glassfish.jersey.internal.Errors logErrors WARNING: The following warnings have been detected: WARNING: The (sub)resource method listConnectors in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation. WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation. WARNING: The (sub)resource method listConnectorPlugins in org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource contains empty path annotation. WARNING: The (sub)resource method serverInfo in org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty path annotation. [2018-01-29 16:49:51,385] INFO Started o.e.j.s.ServletContextHandler@5aabbb29{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:744) [2018-01-29 16:49:51,409] INFO Started ServerConnector@54dab9ac{HTTP/1.1}{0.0.0.0:8083} (org.eclipse.jetty.server.ServerConnector:266) [2018-01-29 16:49:51,409] INFO Started @2019ms (org.eclipse.jetty.server.Server:379) [2018-01-29 16:49:51,410] INFO REST server listening at http://127.0.0.1:8083/, advertising URL http://127.0.0.1:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:150) [2018-01-29 16:49:51,410] INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect:58) [2018-01-29 16:49:51,412] INFO ConnectorConfig values: connector.class = io.confluent.connect.jdbc.JdbcSourceConnector key.converter = null name = source-postgres tasks.max = 16 value.converter = null (org.apache.kafka.connect.runtime.ConnectorConfig:180) [2018-01-29 16:49:51,413] INFO Creating connector source-postgres of type io.confluent.connect.jdbc.JdbcSourceConnector (org.apache.kafka.connect.runtime.Worker:159) [2018-01-29 16:49:51,416] INFO Instantiated connector source-postgres with version 3.1.2 of type class io.confluent.connect.jdbc.JdbcSourceConnector (org.apache.kafka.connect.runtime.Worker:162) [2018-01-29 16:49:51,419] INFO JdbcSourceConnectorConfig values: batch.max.rows = 100 connection.url = incrementing.column.name = id mode = timestamp+incrementing poll.interval.ms = 5000 query = schema.pattern = null table.blacklist = [] table.poll.interval.ms = 60000 table.types = [TABLE] table.whitelist = [] timestamp.column.name = updated_at timestamp.delay.interval.ms = 0 topic.prefix = postgres_ validate.non.null = true (io.confluent.connect.jdbc.source.JdbcSourceConnectorConfig:180) [2018-01-29 16:49:52,129] INFO Finished creating connector source-postgres (org.apache.kafka.connect.runtime.Worker:173) [2018-01-29 16:49:52,130] INFO SourceConnectorConfig values: connector.class = io.confluent.connect.jdbc.JdbcSourceConnector key.converter = null name = source-postgres tasks.max = 16 value.converter = null (org.apache.kafka.connect.runtime.SourceConnectorConfig:180) [2018-01-29 16:49:52,209] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:102) java.lang.IllegalArgumentException: Number of groups must be positive. at org.apache.kafka.connect.util.ConnectorUtils.groupPartitions(ConnectorUtils.java:45) at io.confluent.connect.jdbc.JdbcSourceConnector.taskConfigs(JdbcSourceConnector.java:123) at org.apache.kafka.connect.runtime.Worker.connectorTaskConfigs(Worker.java:193) at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.recomputeTaskConfigs(StandaloneHerder.java:251) at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.updateConnectorTasks(StandaloneHerder.java:281) at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:163) at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:96) [2018-01-29 16:49:52,210] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:68) [2018-01-29 16:49:52,210] INFO Stopping REST server (org.apache.kafka.connect.runtime.rest.RestServer:154) [2018-01-29 16:49:52,213] INFO Stopped ServerConnector@54dab9ac{HTTP/1.1}{0.0.0.0:8083} (org.eclipse.jetty.server.ServerConnector:306) [2018-01-29 16:49:52,218] INFO Stopped o.e.j.s.ServletContextHandler@5aabbb29{/,null,UNAVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:865) [2018-01-29 16:49:52,224] INFO REST server stopped (org.apache.kafka.connect.runtime.rest.RestServer:165) [2018-01-29 16:49:52,224] INFO Herder stopping (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:76) [2018-01-29 16:49:52,224] INFO Stopping connector source-postgres (org.apache.kafka.connect.runtime.Worker:218) [2018-01-29 16:49:52,225] INFO Stopping table monitoring thread (io.confluent.connect.jdbc.JdbcSourceConnector:137) [2018-01-29 16:49:52,225] INFO Stopped connector source-postgres (org.apache.kafka.connect.runtime.Worker:229) [2018-01-29 16:49:52,225] INFO Worker stopping (org.apache.kafka.connect.runtime.Worker:122) [2018-01-29 16:49:52,225] INFO Stopped FileOffsetBackingStore (org.apache.kafka.connect.storage.FileOffsetBackingStore:68) [2018-01-29 16:49:52,225] INFO Worker stopped (org.apache.kafka.connect.runtime.Worker:142) [2018-01-29 16:49:57,334] INFO Reflections took 6952 ms to scan 263 urls, producing 12036 keys and 80097 values (org.reflections.Reflections:229) [2018-01-29 16:49:57,346] INFO Herder stopped (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:86) [2018-01-29 16:49:57,346] INFO Kafka Connect stopped (org.apache.kafka.connect.runtime.Connect:73)
We were using DMS between our RDS Postgres (9.6) to Redshift. It has been failing, and simply miserable, as well as almost at this point almost unweidly expensive, so we are moving into this as a possible solution. I am kind of at a wall here, and would really like to get some help on this.
