Schedule to start an EC2 instance and run a python script within it

Question

I am trying to schedule my python script in AWS, however I don't want the instances to be running all the time. So, trying to automate the process of:

1. Start the EC2 instance on a specific time
2. Run the python script within it
3. Stop the EC2 instance once the job is completed.

I cannot run this script directly as a Lambda function because the script does some parallel processing which requires more RAM, so choosing a bigger AWS instance rather than writing it as a lambda function. Also, don't want this instance to be running all the time as it is expensive.

So far, I followed Automatic starting and stopping of AWS EC2 instances with Lambda and CloudWatch · matoski.com and created a Lambda function to start and stop the instance at specific time, however I couldn't find a way to run the python script once the instance is started.

Can anyone point me in the right direction?

Answer 1

MY application runs an instance @ 13:39 UST everyday and self shuts down after processing is complete. It uses below

1. A scheduled lambda function using cloud watch event rule

Cloud watch Event/rules config

2. The lambda trigger will start an instance (with hardcoded id)

import boto3
def lambda_handler(event, context):
    ec2 = boto3.client('ec2', region_name='ap-south-1')
    ec2.start_instances(InstanceIds=['i-xxxxxxx'])
    print('started your instances: ' + str('i-xxxxxx'))
    return

3. This triggers an instance which has a cron running to execute Python script

   @reboot python /home/Init.py

4. Once script completes, python job shuts down itself using below snippet

import boto.ec2
import boto.utils
import logging
logger=logging.getLogger()
def stop_ec2():
    conn = boto.ec2.connect_to_region("ap-south-1") # or your region
    # Get the current instance's id
    my_id = boto.utils.get_instance_metadata()['instance-id']
    logger.info(' stopping EC2 :'+str(my_id))
    conn.stop_instances(instance_ids=[my_id])

Answer 2

For future developers, who come to this question, a newer approach to this is:

1. Create your EC2 with a role containing AmazonEC2RoleforSSM policy
2. Create a lambda to do the wake-up, run command, shutdown
3. Use a Cloudwatch Event to trigger the lambda

So:

1. Follow the steps here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html

2. Use the following lambda skeleton:

import time
import boto3

REGION_NAME = 'us-east-1'

WORKING_DIRECTORY = '<YOUR WORKING DIRECTORY, IF ANY>'

COMMAND = """
    echo "Hello, world!"
    """

INSTANCE_ID = '<YOUR INSTANCE ID>'


def start_ec2():
    ec2 = boto3.client('ec2', region_name=REGION_NAME)
    ec2.start_instances(InstanceIds=[INSTANCE_ID])

    while True:
        response = ec2.describe_instance_status(InstanceIds=[INSTANCE_ID], IncludeAllInstances=True)
        state = response['InstanceStatuses'][0]['InstanceState']

        print(f"Status: {state['Code']} - {state['Name']}")

        # If status is 16 ('running'), then proceed, else, wait 5 seconds and try again
        if state['Code'] == 16:
            break
        else:
            time.sleep(5)

    print('EC2 started')


def stop_ec2():
    ec2 = boto3.client('ec2', region_name=REGION_NAME)
    ec2.stop_instances(InstanceIds=[INSTANCE_ID])

    while True:
        response = ec2.describe_instance_status(InstanceIds=[INSTANCE_ID], IncludeAllInstances=True)
        state = response['InstanceStatuses'][0]['InstanceState']

        print(f"Status: {state['Code']} - {state['Name']}")

        # If status is 80 ('stopped'), then proceed, else wait 5 seconds and try again
        if state['Code'] == 80:
            break
        else:
            time.sleep(5)

    print('Instance stopped')


def run_command():
    client = boto3.client('ssm', region_name=REGION_NAME)

    time.sleep(10)  # I had to wait 10 seconds to "send_command" find my instance 

    cmd_response = client.send_command(
        InstanceIds=[INSTANCE_ID],
        DocumentName='AWS-RunShellScript',
        DocumentVersion="1",
        TimeoutSeconds=300,
        MaxConcurrency="1",
        CloudWatchOutputConfig={'CloudWatchOutputEnabled': True},
        Parameters={
            'commands': [COMMAND],
            'executionTimeout': ["300"],
            'workingDirectory': [WORKING_DIRECTORY],
        },
    )

    command_id = cmd_response['Command']['CommandId']
    time.sleep(1)  # Again, I had to wait 1s to get_command_invocation recognises my command_id

    retcode = -1
    while True:
        output = client.get_command_invocation(
            CommandId=command_id,
            InstanceId=INSTANCE_ID,
        )

        # If the ResponseCode is -1, the command is still running, so wait 5 seconds and try again
        retcode = output['ResponseCode']
        if retcode != -1:
            print('Status: ', output['Status'])
            print('StdOut: ', output['StandardOutputContent'])
            print('StdErr: ', output['StandardErrorContent'])
            break

        print('Status: ', retcode)
        time.sleep(5)

    print('Command finished successfully') # Actually, 0 means success, anything else means a fail, but it didn't matter to me
    return retcode


def lambda_handler(event, context):
    retcode = -1
    try:
        start_ec2()
        retcode = run_command()
    finally:  # Independently of what happens, try to shutdown the EC2
        stop_ec2()

    return retcode

3. Follow the steps here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/RunLambdaSchedule.html

Answer 3

I was having problems starting and stopping the instance using the solutions in this post. Then I followed the instructions on https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/ and it was really easy. Basically:

1. Go to https://console.aws.amazon.com/iam/home#/home and on the left side, click Policies and click Create Policy. Then click on the JSON tab. Then copy paste this to create a new policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:Start*",
        "ec2:Stop*"
      ],
      "Resource": "*"
    }
  ]
}

2. Go to https://console.aws.amazon.com/iam/home#/home and on the left choose Roles. Make sure you choose Lambda as your AWS Service, and attach the policy you created in Step 1.

3. Then go to the Lambda console, click Create Function. Choose Python 3.7 and then click the dropdown next to Permissions and Use An Existing Role and attach the IAM role you created in Step 2.

4. Use this as your code:

import boto3
region = 'us-west-1' # Dont use the specific, like instead of us-east-1d just write us-east-1
instances = ['i-xxxxxxxxxxxx']
ec2 = boto3.client('ec2', region_name=region)

def lambda_handler(event, context):
    ec2.start_instances(InstanceIds=instances)
    print('started your instances: ' + str(instances))

5. Start your EC2 instance, and type which python to find your path to python and write this down. Then, type in crontab -e to edit your CRON jobs. Don't use sudo...because sometimes sudo messes things up when you haven't been using it to run the Python files. In my instance, I had a pgpass file storing my password that sudo couldn't see , but removing sudo worked!
6. Within the crontab editor after the commented lines, type in @reboot /path/to/python /path/to/file.py For example, for me this was @reboot /home/init/python /home/init/Notebooks/mypredictor.py
7. At the end of your Python file, you need to stop your instance. You can do it like this:

import boto3
region = 'us-west-1' # Dont use the specific, like instead of us-east-1d just write us-east-1
instances = ['i-xxxxxxxxxxxx']
ec2 = boto3.client('ec2', region_name=region)

ec2.stop_instances(InstanceIds=instances)