Glue Crawler to read Json Array

Question

I have a Json array file of the format:- [[{Key1:Value1},{Key2:Value2},{Key3:Value3}],[{Key1:Value4},{Key2:Value5},{Key3:Value6}]]

I need to crawl the above file using AWS glue and read the json schema with each key as a column in the schema. I tried using the standard json classifier but it does not seem to work and the schema loads as an array. I need to read the json file from S3 and load it in an RDS Database. Any suggestions on how to make the transformation either through glue job or classifiers would be helpful.

Answer 1

Create a custom JSON classifier and specify the JSON path as $[*]

For more details, here is the link to the AWS documentation - https://docs.aws.amazon.com/glue/latest/dg/custom-classifier.html#custom-classifier-json

Answer 2

I encountered this very same problem. A solution for me was to format the json file using jq and then re-upload that file to S3. Using any other kind of json formatting tool will also help here.

Any other method is yet unknown to me, hope this helps!