How to do schema evolution in aws glue?

2
votes

I created a table in aws glue using the crawler, then added a partition in s3 with a different schema and ran the crawler again.

When I look at the partition's schema from the partitions view, it shows the new schema, but that table's schema remains the same.

How does schema evolution in glue work? When does glue update the table's schema, if at all?

1
aws-glue

1 Answers

3
votes

It all Depends on how you Configure your crawler. The last Screen ( Configure the crawler's output) of your crawler would have a Configuration options drop down where you have to specify your requirements depending on you needs. Here is the dialogue below: enter image description here