separating key and value in a line using Spark Streaming

0
votes

I have a stream of files and each file contains lines composed of key:value for example, a file would be like:

key1:value1
key2:value2
key3:value3

therefore, I'm using Spark Streaming to detect the arrival of the file in HDFS and what I need to do is put each of the values of these lines in HBase (the columns of HBase are composed of the keys) If the type of the line were "String" separating the key from the value would be easy but what I have is a DStream[String], if I apply DStream.flatmap(_.split(":")) I would get the words and I won't be able to do :

val separated = String.split(":")
val key = separated(0)
val value = separated(1)

I tried converting the DStream to String by using .ToString, when I execute the script using spark-submit I get an error "ArrayOutOfBoundException".

1
apache-sparkspark-streaming

1 Answers

1
votes
val rdd = sc.parallelize(Seq("key1:value1", "key2:value2"))

rdd.map(_.split(":")).collect
// Array[Array[String]] = Array(Array(key1, value1), Array(key2, value2))

rdd.map(_.split(":")(0)).collect
// Array[String] = Array(key1, key2)