Avro schema issue when record missing a field

Question

I am using the NiFi (v1.2) processor ConvertJSONToAvro. I am not able to parse a record that only contains 1 of 2 elements in a "record" type. This element is also allowed to be missing entirely from the data. Is my Avro schema incorrect?

Schema snippet:

"name": "personname",
"type": [
  "null":,
  {
    "type": "record",
    "name": "firstandorlast",
    "fields": [
        {
          "name": "first",
          "type": [
            "null",
            "string"
          ]
        },
        {
          "name": "last",
          "type": [
            "null",
            "string"
          ]
        }
      ]
  }
]

If "personname" contains both "first" and "last" it works, but if it only contains one of the elements, it fails with the error: Cannot convert field personname: cannot resolve union: {"last":"Smith"} not in "type": [ "null":, { "type": "record", "name": "firstandorlast", "fields": [ { "name": "first", "type": [ "null", "string" ] }, { "name": "last", "type": [ "null", "string" ] } ] } ]

hlagos hlagos · Accepted Answer · 2017-11-05T01:24:33

You are missing the default value

https://avro.apache.org/docs/1.8.1/spec.html#schema_record

Your schema should looks like

"name": "personname",
"type": [
  "null":,
  {
    "type": "record",
    "name": "firstandorlast",
    "fields": [
        {
          "name": "first",
          "type": [
            "null",
            "string"
          ],
          "default": "null"
        },
        {
          "name": "last",
          "type": [
            "null",
            "string"
          ],
          "default": "null"
        }
      ]
  }
]

Avro schema issue when record missing a field

1 Answers