deserialization issue, with json.net, in F#

Question

I have a simple object to deserialize, but I do not understand the error I get.

The code is the following:

open System
open Newtonsoft.Json

type r =
    {
        Timestamp:           DateTime
        Currency:            string

        PreviousDeposited:   int64 option
        PreviousWithdrawn:   int64 option
        PreviousTransferIn:  int64 option
        PreviousTransferOut: int64 option
        PreviousAmount:      int64 option

        TransferIn:          int64 option
        TransferOut:         int64 option

        Amount:              int64 option

        PendingCredit:       int64 option
        PendingDebit:        int64 option
        ConfirmedDebit:      int64 option
    }


let a =
    "{
    \"account\": 117122,
    \"currency\": \"XBt\",
    \"prevDeposited\": 747841316,
    \"prevWithdrawn\": 2160000,
    \"prevTransferIn\": 1000000,
    \"prevTransferOut\": 0,
    \"prevAmount\": 656893723,
    \"prevTimestamp\": \"2020-06-13T12:00:00.005Z\",
    \"deltaDeposited\": 0,
    \"deltaWithdrawn\": 0,
    \"deltaTransferIn\": 0,
    \"deltaTransferOut\": 0,
    \"deltaAmount\": 0,
    \"deposited\": 747841316,
    \"withdrawn\": 2160000,
    \"transferIn\": 1000000,
    \"transferOut\": 0,
    \"amount\": 656893723,
    \"pendingCredit\": 0,
    \"pendingDebit\": 0,
    \"confirmedDebit\": 0,
    \"timestamp\": \"2020-06-13T12:00:00.643Z\",
    \"addr\": \"2NBMEXRW4oCiNzVUq4uVFRSsK2jtTLbtfc7\",
    \"script\": \"532102c10be2f0dc20f4285c25156aa22a0c46d2b89ccc4d1c8eaed92ea0c1a8f40c002102ceba29da1af96a0f2ef7cda6950b8be2baeb1adf12c0d5efebb70dbcaa086ba021034ab762f4ede40311e9f8bf01db0bbea578497ac6ccc8aa94a74394b05a53d94b2103d5a42b90e9d7156155661979530a09d2e12e252ef4104e5611274a7ae7e2b09454ae\",
    \"withdrawalLock\": []
    }"


JsonConvert.DeserializeObject<r> a

and I get this error:

Newtonsoft.Json.JsonSerializationException: Unexpected property 'transferOut' found when reading union. Path 'transferOut', line 18, position 18.] at Newtonsoft.Json.Converters.DiscriminatedUnionConverter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType) at ...

I do not understand what makes the property 'TransferOut' so special that it stops on this one, and not on any of the other identical ones before.

I have a fiddle here: https://dotnetfiddle.net/HGiia5

You might want to consider FsPickler for this. Good for communication, but not for persistence. — Bent Tranberg
@BentTranberg, it looks like an interesting library. but it can't work in this scenario since I'm getting data from a 3rd party, processing it and sending it to a queue with a several consumers. — Thomas

dbc dbc · Accepted Answer · 2020-06-13T15:16:33

You have a few problems here.

Firstly, the JSON syntax you are using for an option field does not match Json.NET's syntax. If we simplify your type as follows:

type r =
    {
        TransferIn:          int64 option
        TransferOut:         int64 option
    }

And serialize an instance as follows:

let item : r = { TransferIn = Some 1000000L; TransferOut = None}
let json = JsonConvert.SerializeObject(item,Formatting.Indented)
printfn "%s" json
let item2 = JsonConvert.DeserializeObject<r> json // No exception

The result is:

{
  "TransferIn": {
    "Case": "Some",
    "Fields": [
      1000000
    ]
  },
  "TransferOut": null
}

Which round-trips successfully. Demo fiddle #1 here.

The simple syntax "transferIn": 1000000 you are using for option fields is not implemented by DiscriminatedUnionConverter, the converter that Json.NET uses for serializing discriminated unions including optional fields. This mismatch is causing an exception while reading the JSON.

Relatedly, see Serializing F# Option types which has a suggestion for a nuget package that provides a JsonConverter for option<_> that supports this simplified syntax.

Secondly, many of the JSON property names do not match your f# record names. Json.NET uses an ordinal-case-insensitive algorithm to match JSON property names to f# constructor arguments and member names, but many of your JSON names do not match:

"prevDeposited" does not match PreviousDeposited.
"prevWithdrawn" does not match PreviousWithdrawn.
And several others.

In fact the first property in the JSON that actually matches an option field is "transferIn". You are receiving an error about "transferOut" because it immediately follows the value of "transferIn" which was not deserialized successfully.

Finally, the error message that Json.NET is throwing for invalid JSON values for option fields is non-useful when the field does not appear at the end of a JSON object. If I simplify the input JSON as follows:

{
    "transferIn": 1000000,
}

The we get a much more useful error message

Newtonsoft.Json.JsonSerializationException: No 'Case' property with union name found. Path '', line 3, position 1.

Demo fiddle #2 here.

But when "transferIn" is followed by another JSON key/value pair the error message becomes the less useful message shown in your question. You might open an issue with Newtonsoft asking them to improve the error message that DiscriminatedUnionConverter throws when the JSON value for an option field does not match the expected schema and there are subsequent JSON properties in the containing object.

deserialization issue, with json.net, in F#

1 Answers