I'm setting up a simple AWS Glue ETL job in my AWS Dashboard which is mapping data from parquet files which are located in an AWS S3 bucket to a Redshift database. If I set my Redshift JDBC connection as a requirement to my ETL job, it'll be immediately failed with this message: "An error occurred (403) when calling the HeadObject operation: Forbidden" and if I remove it from job's requirements, it'll face connection timeout after a while. When I test my connection using the same IAM Role, it connects without any problems. Is there something that I'm missing here?
I have already tried even adding an ALL Ports inbound rule to the related security groups of the VPC. I have already tried checking S3 region with Redshift region to be the same. (Although S3 buckets are belonged to global) I even have other ETL jobs converting data from CSV to parquet on the same region with the same IAM role but they're working perfectly fine.
I'd just simply expect the job to transfer my parquet files data to a new Redshift table.
