AWS Glue - using Crawlers or not

1
votes

For a job to run on a data from an S3 bucket in parquet format, there are two ways:

  1. Create a crawler to create a schema table, use glueContext.create_dynamic_frame.from_catalog(dbname, tablename) to form the dynamic frame inside a Glue job.

  2. Read directly from S3 using glueContext.create_dynamic_frame.from_options("s3", {"paths": [full_s3_path] }, format="parquet")

Since my data scheme will NOT change in time, are there any advantages (performance-wise or else) to use a Crawler? Why do I need a Crawler in that case?

1
amazon-web-servicesapache-sparkpysparkaws-glue

1 Answers

3
votes

If your data is not partitioned or you don't want to use predicate-pushdown feature then you don't need to run a Crawler.

However, if it is partitioned and you want to be able to load data partially with predicate-pushdown then new partitions should be registered in Data Catalog and Crawler is one of the easiest ways to do that (there are alternatives though)