Mounting an elastic file system to AWS Batch Computer Enviroment

Question

I'm trying to get my elastic file system (EFS) to be mounted in my docker container so it can be used with AWS batch. Here is what I did:

Create a new AMI that is optimized for Elastic Container Services (ECS). I followed this guide here to make sure it had ECS on it. I also put the mount into /etc/fstab file and verified that my EFS was being mounted (/mnt/efs) after reboot.
Tested an EC2 instance with my new AMI and verified I could pull the docker container and pass it my mount point via

docker run --volume /mnt/efs:/home/efs -it mycontainer:latest

Interactively running the docker image shows me my data inside efs

Set up a new compute enviorment with my new AMI that mounts EFS on boot.

Create a JOB definition File:

{
"jobDefinitionName": "MyJobDEF",
"jobDefinitionArn": "arn:aws:batch:us-west-2:#######:job-definition/Submit:8",
"revision": 8,
"status": "ACTIVE",
"type": "container",
"parameters": {},
"retryStrategy": {
    "attempts": 1
},
"containerProperties": {
    "image": "########.ecr.us-west-2.amazonaws.com/mycontainer",
    "vcpus": 1,
    "memory": 100,
    "command": [
        "ls",
        "/home/efs",
    ],
    "volumes": [
        {
            "host": {
                "sourcePath": "/mnt/efs"
            },
            "name": "EFS"
        }
    ],
    "environment": [],
    "mountPoints": [
        {
            "containerPath": "/home/efs",
            "readOnly": false,
            "sourceVolume": "EFS"
        }
    ],
    "ulimits": []
}

}

Run Job, view log

Anyway, while it does not say "no file /home/efs found" it does not list anything in my EFS which is populated, which I'm inerpreting as the container mounting an empty efs. What am I doing wrong? Is my AMI not mounting the EFS in the compute environment?

Sean B Sean B · Accepted Answer · 2020-03-14T02:23:38

I covered this in a recent blog post

https://medium.com/arupcitymodelling/lab-note-002-efs-as-a-persistence-layer-for-aws-batch-fcc3d3aabe90

You need to set up a launch template for your batch instances, and you need to make sure that your subnets/security groups are configured properly.

Mounting an elastic file system to AWS Batch Computer Enviroment

1 Answers