Deserialize JSON distinguising missing and null values

Question

I have a requirement to parse a JSON object, using play-json and distinguish between a missing value, a string value and a null value.

So for example I might want to deserialize into the following case class:

case class MyCaseClass(
  a: Option[Option[String]]
)

Where the values of 'a' mean:

None - "a" was missing - normal play-json behavipr
Some(Some(String)) - "a" had a string value
Some(None) - "a" had a null value

So examples of the expected behavior are:

{}

should deserialize to myCaseClass(None)

{
  "a": null
} 

should deserialize as myCaseClass(Some(None))

{
  "a": "a"
}

should deserialize as myCaseClass(Some(Some("a"))

I've tried writing custom formatters, but the formatNullable and formatNullableWithDefault methods don't distinguish between a missing and null value, so the code I've written below cannot generate the Some(None) result

object myCaseClass {
  implicit val aFormat: Format[Option[String]] = new Format[Option[String]] {
    override def reads(json: JsValue): JsResult[Option[String]] = {
      json match {
        case JsNull => JsSuccess(None) // this is never reached
        case JsString(value) => JsSuccess(Some(value))
        case _ => throw new RuntimeException("unexpected type")
      }
    }
    override def writes(codename: Option[String]): JsValue = {
      codename match {
        case None => JsNull
        case Some(value) =>  JsString(value)
      }
    }
  }

  implicit val format = (
      (__ \ "a").formatNullableWithDefault[Option[String]](None)
  )(MyCaseClass.apply, unlift(MyCaseClass.unapply))
}

Am I missing a trick here? How should I go about this? I am very much willing to encode the final value in some other way than an Option[Option[Sting]] for example some sort of case class that encapsulates this:

case class MyContainer(newValue: Option[String], wasProvided: Boolean)

And, yes - I am aware that I could write an entirely custom read method for the entire object. I was trying to avoid that. — iandotkelly
Type Option[Option[_]] can hardly make any sensé whatever is the case. BTW I cannot see benefit from such null/missing distinction. — cchantep
@cchantep .. there is a bit of a code smell for sure - but these are my requirements. There is a somewhat plausible use case. In any case its the json parsing that is the side i'm struggling with. For the purpose of the question if we consider undefined and null to be separate values in json ... how do I parse that in play without just traversing the json myself. — iandotkelly
You are assuming that undefined or null are différent, whereas they both the same using different format: represent the fact there is no value for a spécifiéd field — cchantep
i absolutely see null vs omitted as two completely different intents. it is unfortunate that there isn't less ambiguity about this in the JSON spec, but intuitively it makes sense to me to do what @iandotkelly is proposing here (especially when considering PATCH). I have opened a discussion about this with the core play-json team here: discuss.lightbend.com/t/… — kflorence

Dave Gurnell Dave Gurnell · Accepted Answer · 2019-05-13T18:52:32

I recently found a reasonable way to do this. I'm using Play 2.6.11 but I'm guessing the approach will transfer to other recent versions.

The following snippet adds three extension methods to JsPath, to read/write/format fields of type Option[Option[A]]. In each case a missing field maps to a None, a null to a Some(None), and a non-null value to a Some(Some(a)) as the original poster requested:

import play.api.libs.json._

object tristate {
  implicit class TriStateNullableJsPathOps(path: JsPath) {
    def readTriStateNullable[A: Reads]: Reads[Option[Option[A]]] =
      Reads[Option[Option[A]]] { value =>
        value.validate[JsObject].flatMap { obj =>
          path.asSingleJsResult(obj) match {
            case JsError(_)           => JsSuccess(Option.empty[Option[A]])
            case JsSuccess(JsNull, _) => JsSuccess(Option(Option.empty[A]))
            case JsSuccess(json, _)   => json.validate[A]
                                             .repath(path)
                                             .map(a => Option(Option(a)))
          }
        }
      }

    def writeTriStateNullable[A: Writes]: OWrites[Option[Option[A]]] =
      path.writeNullable(Writes.optionWithNull[A])

    def formatTriStateNullable[A: Format]: OFormat[Option[Option[A]]] =
      OFormat(readTriStateNullable[A], writeTriStateNullable[A])
  }
}

Like previous suggestions in this thread, this method requires you to write out a JSON format in full using the applicative DSL. It's unfortunately incompatible with the Json.format macro, but it gets you close to what you want. Here's a use case:

import play.api.libs.json._
import play.api.libs.functional.syntax._
import tristate._

case class Coord(col: Option[Option[String]], row: Option[Option[Int]])

implicit val format: OFormat[Coord] = (
  (__ \ "col").formatTriStateNullable[String] ~
  (__ \ "row").formatTriStateNullable[Int]
)(Coord.apply, unlift(Coord.unapply))

Some examples of writing:

format.writes(Coord(None, None))
// => {}

format.writes(Coord(Some(None), Some(None)))
// => { "col": null, "row": null }

format.writes(Coord(Some(Some("A")), Some(Some(1))))
// => { "col": "A", "row": 1 }

And some examples of reading:

Json.obj().as[Coord]
// => Coord(None, None)

Json.obj(
  "col" -> JsNull, 
  "row" -> JsNull
).as[Coord]
// => Coord(Some(None), Some(None))

Json.obj(
  "col" -> "A", 
  "row" -> 1
).as[Coord]
// => Coord(Some(Some("A")), Some(Some(1)))

As a bonus exercise for the reader, you could probably combine this with a little shapeless to automatically derive codecs and replace the Json.format macro with a different one-liner (albeit one that takes longer to compile).

Deserialize JSON distinguising missing and null values

3 Answers