Install Kafka connector manually

0
votes

I'm trying install twitter source connector and found the following links link1 and link2. The first one shows how to install twitter source connector and the second generally how to install manually a connector (always Confluent Platform).

However I'm confused from the entire procedure. I followed the steps in here, and so far it's working fine. However since the twitter source connector is not in the list of the Connector Hub it needs to be installed manually. According to link1, I clone a git repo somewhere - Question: where do I clone it? In the docker-compose.yml there are a bunch of services. In particular, a docker-compose ps after I spin up the yaml shows 

       Name                    Command                  State                         Ports
  ------------------------------------------------------------------------------------------------------------

   broker            /etc/confluent/docker/run        Up             0.0.0.0:9092->9092/tcp
   connect           /etc/confluent/docker/run        Up             0.0.0.0:8083->8083/tcp, 9092/tcp
   control-center    /etc/confluent/docker/run        Up             0.0.0.0:9021->9021/tcp
   ksql-datagen      bash -c echo Waiting for K ...   Up
   ksqldb-cli        /bin/sh                          Up
   ksqldb-server     /etc/confluent/docker/run        Up (healthy)   0.0.0.0:8088->8088/tcp
   rest-proxy        /etc/confluent/docker/run        Up             0.0.0.0:8082->8082/tcp
   schema-registry   /etc/confluent/docker/run        Up             0.0.0.0:8081->8081/tcp
   zookeeper         /etc/confluent/docker/run        Up             0.0.0.0:2181->2181/tcp, 2888/tcp, 3888/tcp

I don't understand to which one of the above I should carry out the instructions of link1. Is it in the zookeeper, the broker or in the connect? Any help is welcomed, please if someone has time to write out a few words around the steps in link1 more thoroughly, please feel free. Cheers

1
dockerapache-kafkaapache-kafka-connectconfluent-platform

1 Answers

2
votes

Connectors are installed in Kafka Connect workers. They must be installed before the worker is started.

There is a newer version of that blog article, here: https://www.confluent.io/blog/stream-processing-twitter-data-with-ksqldb. Associated with that is a new demo, which you can find here: https://github.com/confluentinc/demo-scene/tree/master/ksqldb-twitter

As part of the demo, it installs the twitter connector at runtime. It does this by overriding the default command of the container image and instead runs the connector installation before running the connect worker: 

    command: 
      - bash 
      - -c 
      - |
        echo "Installing connector plugins"
        confluent-hub install --no-prompt jcustenborder/kafka-connect-twitter:0.3.33
        #
        echo "Launching Kafka Connect worker"
        /etc/confluent/docker/run & 
        #
        sleep infinity