is schema mandatory for spark structured streaming processing Json record

Question

I was going through the spark structured streaming in the below blog.

he is first creating schema variable using below piece of code.

val cloudTrailSchema = new StructType()
  .add("Records", ArrayType(new StructType()
    .add("additionalEventData", StringType)
    .add("apiVersion", StringType)
    .add("awsRegion", StringType)

Then below is the actual spark code

val rawRecords = spark.readStream
  .schema(cloudTrailSchema)
  .json("s3n://mybucket/AWSLogs/*/CloudTrail/*/2017/*/*")

As json record will have schema by default, Why should we provide the schema. For example in spark batch streaming we dont provide any schema in below line of code.

val peopleDF = spark.read.json(path)

The code infers the schema of Dataframe directly from Json record.

So can we process json records in spark structured streaming without using schema ?

tstites tstites · Accepted Answer · 2018-03-27T13:09:12

Schema is required.

This is easily testable. Just remove ".schema()" option when running your structured streaming query and it should fail with a message saying a schema is required.

Schema inference and partition of streaming DataFrames/Datasets

is schema mandatory for spark structured streaming processing Json record

1 Answers