Kafka to Snowflake connection issue

2
votes

I am trying to connect from local standalone Confluent Kafka topics to Snowflake tables. I am using the following connector config via ksqldb.

CREATE SINK CONNECTOR `snowflake_sink` WITH(
"name"='snowflake_sink',
"tasks.max"='1',
"connector.class"='com.snowflake.kafka.connector.SnowflakeSinkConnector',
"topics"='USERPROFILE',
"snowflake.url.name"='https://mybu.mycompany.us-east-1.aws.privatelink.snowflakecomputing.com',
"snowflake.user.name”=‘myuser’,
"snowflake.database.name"='DEMO_DB',
"snowflake.topic2table.map"='USERPROFILE:UK_SF_DB1_Table1',
"snowflake.schema.name"='PUBLIC',
"snowflake.private.key”=‘<valid private key>’,
"snowflake.private.key.passphrase”=‘<valid pass phrase>’,
"key.converter"='org.apache.kafka.connect.storage.StringConverter',
"value.converter"='com.snowflake.kafka.connector.records.SnowflakeJsonConverter',
"key.converter.schema.registry.url"='http://schema-registry:8081',
"value.converter.schema.registry.url"='http://schema-registry:8081');

Ksqldb throws error saying snowflake url, username, private key is not valid. They are all valid as I can login using the same via Snowflake web ui.

1
apache-kafkasnowflake-cloud-data-platformksqldb
This is the same question as: stackoverflow.com/questions/64049202/… - Robert

1 Answers

1
votes

Some ideas on troubleshooting this:

  1. it looks like you have some "smart quotes" in your config, please remove the smart single and double quotes and replace them with proper text quotes. This may have happened during your preparing of this question, so just double check them.

  2. set your table name to all capital letters.

  3. your snowflake account should not include the https:// prefix. Examples at these links:

https://community.snowflake.com/s/article/Docker-Compose-Setting-up-Kafka-using-Snowflake-Sink-Connector-and-testing-Streams

https://docs.snowflake.com/en/user-guide/kafka-connector-install.html

I hope this helps...Rich

p.s. If this (or another) answer helps you, please take a moment to "accept" the answer that helped by clicking on the check mark beside the answer to toggle it from "greyed out" to "filled in".