How to Use EMR with EKS-Fargate? All serverless?

Question

So I am deploying an EMR application using EKS, following the EMR-EKS workshop by AWS, declaring the Fargate profile.

I tried everything to be serverless, so even my EKS Cluster runs on Fargate (kube-sytem, default, etc).

I created a custom namespace with a Fargate profile, and submit a Spark job. The job stayed at Pending status.

When I added some managed nodegroups, the job was submitted successfully.

I tried submitting very light jobs and by the time I removed the managed node groups, spark jobs stay at pending status.

Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  33s (x7 over 5m43s)  default-scheduler  0/2 nodes are available: 2 node(s) had taint {eks.amazonaws.com/compute-type: fargate}, that the pod didn't tolerate.

eksworkshop-eksctl.yaml

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: eksworkshop-eksctl
  region: ${AWS_REGION}
  version: "1.21"
    
availabilityZones: ["${AZS[0]}", "${AZS[1]}", "${AZS[2]}", "${AZS[3]}"]

fargateProfiles:
  - name: default
    selectors:
      # All workloads in the "default" Kubernetes namespace will be
      # scheduled onto Fargate:
      - namespace: default
      # All workloads in the "kube-system" Kubernetes namespace will be
      # scheduled onto Fargate:
      - namespace: kube-system

Create the EKS cluster:

eksctl create cluster -f eksworkshop-eksctl.yaml

Create namespace and fargate profile:

kubectl create namespace spark
eksctl create fargateprofile --cluster eksworkshop-eksctl --name emr \
--namespace spark --labels type=etl

Create virtual cluster:

I have already added the labels under sparkSubmitParameters, but it is still stuck in Pending state. :( Is there additional configuration I need to add when creating virtual cluster:

aws emr-containers create-virtual-cluster \
--name eksworkshop-eksctl \
--container-provider '{
    "id": "eksworkshop-eksctl",
    "type": "EKS",
    "info": {
        "eksInfo": {
            "namespace": "spark"
        }
    }
}'

Submit the spark job:

aws emr-containers start-job-run --virtual-cluster-id $VIRTUAL_CLUSTER_ID --name spark-pi-logging --execution-role-arn $EMR_ROLE_ARN --release-label emr-6.2.0-latest --job-driver '{
    "sparkSubmitJobDriver": {
        "entryPoint": "s3://aws-data-analytics-workshops/emr-eks-workshop/scripts/pi.py",
        "sparkSubmitParameters": "--conf spark.kubernetes.driver.label.type=etl --conf spark.kubernetes.executor.label.type=etl --conf spark.executor.instances=2 --conf spark.executor.memory=2G --conf spark.executor.cores=2 --conf spark.driver.cores=1"
        }
    }' --configuration-overrides '{
    "applicationConfiguration": [
      {
        "classification": "spark-defaults",
        "properties": {
          "spark.driver.memory":"2G"
         }
      }
    ],
    "monitoringConfiguration": {
      "cloudWatchMonitoringConfiguration": {
        "logGroupName": "/emr-containers/jobs",
        "logStreamNamePrefix": "emr-eks-workshop"
      },
      "s3MonitoringConfiguration": {
        "logUri": "'"$s3DemoBucket"'/logs/"
      }
    }
}'

For EKS cluster, is NodeGroup must be declared and is mandatory, and we can't run Spark Job just be using Fargate only?

Can you do a kubectl describe to the pending job and paste to your question? — gohm'c
Hi @gohm'c here is the result. I am using the same namespace for the fargate profile. — Charmee Lee
Hi @gohm'c . I'm testing Spark jobs with EMR virtual cluster. Below are my outcomes: created fargate-profile with the same namespace: Job completed created fargate-profile with the same namespace but with labels in executor and driver: Job stucked in Pending so I tried having 2 fargate profile with the same namespace=spark, one has label=etl, one does not have label. Is it required to have 2 profiles, one with label, and one with no label? I just expect that a Fargate profile with a label as long as it met the spark configuration shall work. — Charmee Lee

gohm'c gohm'c · Accepted Answer · 2021-09-30T08:43:26

Your situation:

...created fargate-profile with the same namespace: Job completed

... created fargate-profile with the same namespace but with labels...Job stucked in Pending

If a namespace is associated with >1 profile, EKS will randomly choose a profile. When it picks a profile that requires a label and your spark job don't have it - it goes into pending state since there is no other node group in your cluster.

Is it required to have 2 profiles, one with label, and one with no label

No, in fact you should not do this. To run job on a profile with labels your job sparkSubmitParameters must specify the label:

"sparkSubmitParameters": "...--conf spark.kubernetes.driver.label.<label key>=<value> --conf spark.kubernetes.executor.label.<label key>=<value>..."

You don't need >1 Fargate profile unless you need to distinguish spark jobs by Fargate profile selector.

How to Use EMR with EKS-Fargate? All serverless?

1 Answers