MSCK Repair Command on AWS Glue Catalog job

2
votes

Can we have an AWS Glue job scheduled to perform MSCK repair commands so that the metadata for newly added partition gets added to Glue Catalog?

Can Glue ETL script perform MSCK REPAIR TABLE command without calling Athena?

2
amazon-web-servicesaws-glue

2 Answers

0
votes

This is achieved by Glue Crawlers. If you create a crawler it will update the table based on new fields and add new partitions.

0
votes

You can call batch_create_partition() API to do it. It doesn't require expensive operations like MSCK REPAIR TABLE or re-crawling. Below is my detailed answer with code sample -

https://stackoverflow.com/a/52239022/2414855