I have this JSON String that comes from a Kafka topic
{"id": 12345, "items": {"unit17": 0, "unit74": 0, "unit42": 0, "unit96": 0, "unit13": 0, "unit16": 0, "unit11": 0, "z10": 0, "z0": 1}}
By using spray-json (version 1.3.5), I want to parse it so I do:
val parsedStream = stream.map( event => event.parseJson )
This is working well BUT when using "parseJson" the nested JSON for items is alphabetically ordered and within a List:
parsedStream.print()
--> List(12345,{"unit11": 0, "unit13": 0, "unit16": 0, "unit17":0, "unit42": 0, "unit74": 0, "unit96": 0, "z0": 1, "z10": 0}}
Any ideas why spray-json is behaving like this and ordering it automatically? Any settings to avoid this or options to apply?
I have tried to do the same with another library, play-json (play.api.libs.json.Json) and it is properly working so I could use this one but I was curious if I was missing something with spray-json:
val parsedStream = stream.map( event => Json.parse(event))
parsedStream.print()
--> {"id":12345,"items":{"unit17":0,"unit74":0,"unit42":0,"unit96":0,"unit13":0,"unit16":0,"unit11":0,"z10":0,"z0":1}}
Finally, I would like to feed a case class with the values of that stream; for this I've implemented the next elements:
case class MyCaseClass(id:Int,items:Map[String,Int])
val parsedStream = stream.map{ value => MyCaseClass( (value \ "id").as[Int],
(value \ "items").as[Map[String,Int]] ) }
That works smoothly BUT since I'm using a Map for "items" attribute, the order in the JSON is not forcefully kept, and I need it to be, for this will be sent to a model to predict some stuff and the model training has been done with the same items order that is produced by the Kafka topic. I know scala provides a ListMap that is a kind of "ordered" map with an interface of List but when using it like this:
case class MyCaseClass(id:Int,items:ListMap[String,Int])
val parsedStream = stream.map{ value => MyCaseClass( (value \ "id").as[Int],
(value \ "items").as[ListMap[String,Int]] ) }
the compiler states "No Json deserializer found for type scala.collection.immutable.ListMap[String,Int]. Try to implement an implicit Reads or Format for this type.", I guess because there is not a convenient Play's automatic JSON macros for this type and it should be implemented. Do you have any hint on how to do that for a ListMap or any other way I could achieve to have the items in my case class with the same order they come in the JSON? Maybe change the "items" JSON String to a JSON Array could do the trick, any hint on how I could achieve that and if it is a good idea?
EDIT - Resolved with circe-json
After some tests with play-json it was a bit tricky to use with ListMap for the Reads macros, so I read and went to experiment about circe-json and with this last it was quite straight forward for both, the order in the parsing and the conversion to ListMap. It is enough with just some imports and one line of code to have the parsing done and the conversion to the case class:
import io.circe._
import io.circe.parser._
import io.circe.generic.auto._
[...]
case class MyCaseClass(id:Int,items:ListMap[String,Int])
[...]
val streamParsed = stream.map{ event => parser.decode[MyCaseClass](event) match {
case Left(failure) => [...]
case Right(myCaseClassInstance) => { myCaseClassInstance }
}}
[...]
That will automatically parse keeping the order and generate a DataStream[MyCaseClass] using the ListMap to still keep the order in the generated map. The mapping from JSON to ListMap is possible with the "io.circe.generic.auto._" that supports ListMap and transparently takes care of this conversion.
