How to submit multiple commands to AWS Batch using Boto3?

Question

I'm trying to run multiple shell commands through Docker using AWS Batch and boto3. When I try to submit multiple commands using the & sign as follows, the job fails.

My attempt

import boto3

client = boto3.client("batch")
response = client.submit_job(
    jobName='AndrewJob',
    jobQueue='AndrewJobQueue',
    jobDefinition='AndrewJobDefinition',
    containerOverrides={
        'command': 'ls & python myjob.py'.split(),
    },
    timeout = {'attemptDurationSeconds': 100}
)
print(response)

The error is:

ls: cannot access '&': No such file or directory

According to the Docker Docs here https://docs.docker.com/engine/reference/builder/#cmd and this post here docker run <IMAGE> <MULTIPLE COMMANDS> it seems like this should be possible in shell form.

@badger0053 - yes, I've tried && and get the same error (but with '&&') — maurera

Nathan Collins Nathan Collins · Accepted Answer · 2020-04-23T00:19:33

It appears that Batch is behaving like subprocess.Popen in that it executes the command as one command where the first argument is the command name and the remaining arguments are the command arguments. I got this to work with subprocess.Popen, so I bet it would work with Batch:

subprocess.Popen(["/bin/bash", "-c", "ls && echo \"Hello world\""])

How to submit multiple commands to AWS Batch using Boto3?

1 Answers