Avro schema GenericRecord missing key

Question

I am using Avro schema to dynamically produce message to a Kafka cluster from a C# application, using the Confluent Kafka client. The data types are not known on compile time so I am using the GenericRecord class from the Avro.Generic namespace, as described here: https://www.confluent.io/blog/decoupling-systems-with-apache-kafka-schema-registry-and-avro/.

I however have an issue - if the schema has a field that can contain null value, it is still required to add the field to the GenericRecord by using the Add method, giving null as value. My application is not aware of the fields that can be null and I don't think it should be - because that would defy the purpose of nullable fields in the schema.

Avro schema:

{
  "namespace": "Test",
  "type": "record",
  "doc": "Test bool type",
  "name": "BoolType",
  "version": "1",
  "fields": [
    {
      "name": "Data",
      "type": [ "null", "boolean" ],
      "default": null
    },
    {
      "name": "Source",
      "type": "string"
    }
  ]
}

C# code:

var valueRecord = new GenericRecord( valueAvroSchema );
valueRecord.Add( "Data", null );
valueRecord.Add( "Source", "Test app .NET" );

var messageToSend = new Message<GenericRecord, GenericRecord>
{
   Key = keyRecord,
   Value = valueRecord
};

await _producer.ProduceAsync( _topicName, messageToSend );

If the line:

valueRecord.Add( "Data", null );

is not present, the ProduceAsync method throws a Confluent.Kafka.ProduceException, as seen in the screenshot below.

Is there any way I can automatically populate the fields that can be null in the GenericRecord? The same would apply if I would have to populate the fields with their default value.

Is there any way of doing this in a standard way or do I need to write my own code to read the schema and if there are any nullable fields that are not already set by my application to add them at the end, before publishing?

Thank you!

OneCricketeer OneCricketeer · Accepted Answer · 2019-11-22T03:20:44

1

votes

Avro defaults are only relevant to consumers. The producer must always set each field

Avro schema GenericRecord missing key

1 Answers