Call a Lambda Function with AWS Glue

Question

Im trying to use boto3 in a job of AWS Glue to call a Lambda Function but without results.

I upload a zip with the libraries:

Like the examples by AWS

and without a zip.

The error is this " Unable to load data for: endpoints".

Im trying to invoke without zip but this go to a timeout exception.

import boto3
client = boto3.client('lambda' , region_name='us-east-1')
r_lambda = client.invoke(FunctionName='S3GlueJson')

Can someone help me ?

Answer 1

In Python, use Boto3 Lambda client 'invoke()'. For example, you can create a Lambda container, then call that from a Glue Job:

import boto3
import pandas as pd

lambda_client = boto3.client('lambda',region_name='us-east-1') 

def get_predictions( df ):
    # Call getPredictions Lambda container
    response = lambda_client.invoke(
        FunctionName='getPredictions',
        InvocationType='RequestResponse',
        LogType='Tail',
        Payload=df
    )
    logger.info('Received response from Lambda container.')
    data = response["Payload"].read().decode('utf-8')
    x = json.loads(data)
    df_pred = pd.DataFrame.from_dict(x)
    return df_pred

dfjson = df.to_json()
df_pred = get_predictions( dfjson )
df_pred.head()

Answer 2

If you want to call a Glue Jobs from Lambda Function, can do it like this:

import boto3
glue = boto3.client(service_name='glue', region_name='us-east-1',
              endpoint_url='https://glue.us-east-1.amazonaws.com')

#Start Job
myNewJobRun = glue.start_job_run(JobName=JOB_NAME)

#Get current state of Job, to be sure it's running
status = glue.get_job_run(JobName=JOB_NAME, RunId=myNewJobRun['JobRunId'])
logger.info('JOB State {}: {}'.format(
JOB_NAME, status['JobRun']['JobRunState']))

As Job execution can late some time to finish, it's better to don't wait on Lambda function for it to finish.