(Submitting on behalf of a Snowflake client...)
.........................
I am trying to connect Kafka to snowflake using Snowflake Connector for Kafka.
Referring to this document: https://docs.snowflake.net/manuals/user-guide/kafka-connector.html
When I am running Kafka, it is initializing the Snowflake plugins .
eg:
[2019-08-31 21:52:09,448] INFO Added aliases 'SnowflakeSinkConnector' and 'SnowflakeSink' to plugin 'com.snowflake.kafka.connector.SnowflakeSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:396)
[2019-08-31 21:52:09,456] INFO Added aliases 'SnowflakeJsonConverter' and 'SnowflakeJson' to plugin 'com.snowflake.kafka.connector.records.SnowflakeJsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:396)
But after that it is unable to read other worker config attributes.
[2019-08-31 21:52:10,373] WARN The configuration 'connector.class' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)
[2019-08-31 21:52:10,373] WARN The configuration 'snowflake.topic2table.map' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)
[2019-08-31 21:52:10,375] WARN The configuration 'tasks.max' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)
[2019-08-31 21:52:10,378] WARN The configuration 'topics' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)
[2019-08-31 21:52:10,381] WARN The configuration 'snowflake.private.key.passphrase' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)
[2019-08-31 21:52:10,385] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)
[2019-08-31 21:52:10,386] WARN The configuration 'buffer.flush.time' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)
[2019-08-31 21:52:10,386] WARN The configuration 'snowflake.url.name' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)
[2019-08-31 21:52:10,387] WARN The configuration 'value.converter.basic.auth.credentials.source' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)
[2019-08-31 21:52:10,387] WARN The configuration 'snowflake.database.name' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)
[2019-08-31 21:52:10,387] WARN The configuration 'snowflake.schema.name' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)
[2019-08-31 21:52:10,387] WARN The configuration 'value.converter.schema.registry.url' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)
[2019-08-31 21:52:10,389] WARN The configuration 'offset.storage.file.filename' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)
[2019-08-31 21:52:10,392] WARN The configuration 'value.converter.basic.auth.user.info' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)
[2019-08-31 21:52:10,392] WARN The configuration 'buffer.count.records' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)
[2019-08-31 21:52:10,393] WARN The configuration 'snowflake.private.key' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)
[2019-08-31 21:52:10,393] WARN The configuration 'snowflake.user.name' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)
[2019-08-31 21:52:10,393] WARN The configuration 'name' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)
[2019-08-31 21:52:10,394] WARN The configuration 'value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)
[2019-08-31 21:52:10,394] WARN The configuration 'key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)
[2019-08-31 21:52:10,394] WARN The configuration 'buffer.size.bytes' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)
I realize these are warnings but after that, we're getting failures. So I assume it is failing as it is unable to initialize the above config values.
WARNING: A provider org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource will be ignored.
Sep 04, 2019 11:55:52 AM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
WARNING: A provider org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource will be ignored.
Sep 04, 2019 11:55:52 AM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
WARNING: A provider org.apache.kafka.connect.runtime.rest.resources.RootResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.apache.kafka.connect.runtime.rest.resources.RootResource will be ignored.
Sep 04, 2019 11:55:52 AM org.glassfish.jersey.internal.Errors logErrors
WARNING: The following warnings have been detected: WARNING: The (sub)resource method listConnectors in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method listConnectorPlugins in org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource contains empty path annotation.
WARNING: The (sub)resource method serverInfo in org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty path annotation.
[2019-09-04 11:55:52,788] INFO Started o.e.j.s.ServletContextHandler@2be818da{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:850)
[2019-09-04 11:55:52,800] INFO Started http_8083@798deee8{HTTP/1.1,[http/1.1]}{0.0.0.0:8083} (org.eclipse.jetty.server.AbstractConnector:292)
[2019-09-04 11:55:52,801] INFO Started @9514ms (org.eclipse.jetty.server.Server:408)
[2019-09-04 11:55:52,802] INFO Advertised URI: http://10.10.25.86:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:267)
[2019-09-04 11:55:52,802] INFO REST server listening at http://10.10.25.86:8083/, advertising URL http://10.10.25.86:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:217)
[2019-09-04 11:55:52,802] INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect:55)
[2019-09-04 11:55:52,807] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:113)
org.apache.kafka.common.config.ConfigException: Must configure one of topics or topics.regex
at org.apache.kafka.connect.runtime.SinkConnectorConfig.validate(SinkConnectorConfig.java:96)
at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:269)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:189)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:107)
[2019-09-04 11:55:52,808] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:65)
[2019-09-04 11:55:52,808] INFO Stopping REST server (org.apache.kafka.connect.runtime.rest.RestServer:223)
[2019-09-04 11:55:52,820] INFO Stopped http_8083@798deee8{HTTP/1.1,[http/1.1]}{0.0.0.0:8083} (org.eclipse.jetty.server.AbstractConnector:341)
[2019-09-04 11:55:52,821] INFO node0 Stopped scavenging (org.eclipse.jetty.server.session:167)
[2019-09-04 11:55:52,827] INFO Stopped o.e.j.s.ServletContextHandler@2be818da{/,null,UNAVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:1040)
[2019-09-04 11:55:52,829] INFO REST server stopped (org.apache.kafka.connect.runtime.rest.RestServer:241)
[2019-09-04 11:55:52,829] INFO Herder stopping (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:95)
[2019-09-04 11:55:52,829] INFO Worker stopping (org.apache.kafka.connect.runtime.Worker:184)
[2019-09-04 11:55:52,829] INFO Stopped FileOffsetBackingStore (org.apache.kafka.connect.storage.FileOffsetBackingStore:66)
[2019-09-04 11:55:52,830] INFO Worker stopped (org.apache.kafka.connect.runtime.Worker:205)
[2019-09-04 11:55:52,830] INFO Herder stopped (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:112)
[2019-09-04 11:55:52,830] INFO Kafka Connect stopped (org.apache.kafka.connect.runtime.Connect:70)
I understand that this may be a configuration issue in that in the newer version of Kafka, the configuration for "topic" was updated to "topics", but are there any other/additional explanations, corrective actions or recommended work-arounds?
Thank you!
