create stream on kafka topic using KSQL

0
votes

Below is my sample logline from kafka topic

2019-03-04T08:53:03.023Z "cd8cbe" 100.212.212.212 - - [20/Feb/2019:12:13:33 +0000] "GET http://dl-mysite.com/drm/PRIORITY1080/HINDI_MOVIES/somemovie.mp4/video/avc1/4/seg-1281.m4s HTTP/1.1" 200 325040 "-" "Dalvik/2.1.0 (Linux; U; Android 6.0; Le X509 Build/DHXOSOP5801911241S)" "256" "0.000"

I am trying to create stream on above topic using KSQL.Below is the script which I used to create stream. After running the below stream creation script it returns me 'Stream created' message, however the select statement (select * from test_duplicate_stream;)doesn't return anything.

CREATE STREAM test_duplicate_stream (logArrivalTime varchar,edgeid varchar,ip varchar,col1_empty varchar,col2_empty varchar, eventdate varchar,url varchar,response_code int,response_length BIGINT,col3_empty varchar,user_agent varchar,request_length varchar, response_time varchar) WITH (kafka_topic='test_duplicate',VALUE_FORMAT='DELIMITED');

I believe 'DELIMITED' is not the right value to use here as my fields are not comma separated but space separated. What is the right way to create the stream for my logline?

1
apache-kafkaksqldb
What is the format of your original topic? – Giorgos Myrianthous
My topic is space delimited – hampi2017

1 Answers

0
votes

KSQL currently supports:

  • JSON
  • Comma-separated (delimited)
  • Avro

If your data's not in one of the above formats then you won't be able to process it without changing the serialisation first.

See also Notes on troubleshooting KSQL for when your query doesn't return data.