Confluent HDFS Connector

1
votes

I want to move kafka log file to hadoop log file. So i make follow HDFS Connector Configuration

/quickstart-hdfs.properties

name=hdfs-sink 
connector.class=io.confluent.connect.hdfs.HdfsSinkConnector
tasks.max=1 
topics=kafka_log_test 
hdfs.url=hdfs://10.100.216.60:9000 
flush.size=100000 
hive.integration=true 
hive.metastore.uris=thrift://localhost:9083 
schema.compatibility=BACKWARD 
format.class=io.confluent.connect.hdfs.parquet.ParquetFormat 
partitioner.class=io.confluent.connect.hdfs.partitioner.Hour‌​lyPartitioner

/connect-avro-standalone.properties

bootstrap.servers=localhost:9092

key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://localhost:8081
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://localhost:8081

internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false

offset.storage.file.filename=/tmp/connect.offsets

When i run the HDFS Connector, just writes avro schema in .avro file. Not data.

/kafka_log_test+0+0000000018+0000000020.avro

 avro.schema {"type":"record","name":"myrecord","fields":[{"name":"f1","type":"string"}],"connect.version":1,"connect.name":"myrecord"}

Topic have lots of data but confluent hdfs connector doesn't move data to hdfs.

How can i do that to resolve this problem ?

1
hadoopapache-kafkaavroapache-kafka-connectconfluent-platform

1 Answers

0
votes

By definition, unless the messages are otherwise compacted or expired between offsets 18 and 20 then the file containing the name 0+0000000018+0000000020 will have 2 records from partition 0.

You should use tojson command of avro-tools rather than getmeta.

Or you can use Spark or Pig to read that file.

You might also want to verify the connectors is continuing to run after starting it because setting hive.metastore.uris=thrift://localhost:9083 on a machine that is not the Hive Metastore Server will cause the Connect task to fail. The URI should be the actual host for Hive, just as you've done for the NameNode.

Also, it shouldn't be possible to get a .avro file extension with format.class=io.confluent.connect.hdfs.parquet.ParquetFormat anyway, so you might want to verify you are looking in the correct HDFS path. Note: Connect writes to a +tmp location temporarily before writing the final output files.