Issue while executing Python script on AWS Lambda function

Question

I tried running the below script on AWS Lambda, but unfortunately, it's not executing successfully. Can anyone help me to get this fix or correct me if there is an issue with the script which needs some change executing it from Lambda?

#!/usr/bin/env python3
import boto3
client = boto3.client('athena')
def run_query(query, database, s3_output):
    response = client.start_query_execution(
        QueryString=query,
        QueryExecutionContext={
            'Database': 'database'
            },
        ResultConfiguration={
            'OutputLocation': s3_output,
            }
        )
    print('Execution ID: ' + response['QueryExecutionId'])
    return response     
  
#Athena configuration
s3_input = 's3://smathena/cf-ant-prod/'
s3_ouput = 's3://smathena/athenatest/'
database = 's3_accesslog'
table = 'test_output1'

#Athena database and table definition
create_database = "CREATE DATABASE IF NOT EXISTS %s;" % (database)
delete_table = "drop table %s.%s;" % ( database, table )
create_table = \
  """CREATE EXTERNAL TABLE IF NOT EXISTS %s.%s (
  `Date` DATE,
   ScContentLen BIGINT,
   ScRangeStart BIGINT,
   ScRangeEnd BIGINT
   )
   ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
   LOCATION '%s'
   TBLPROPERTIES ('skip.header.line.count' = '2');""" % ( database, table, s3_input )

#Query definitions
query_1 = "SELECT * FROM %s.%s where CAST(status AS VARCHAR) like '404';" % (database, table)

#Execute all queries
queries = [ create_database, delete_table, create_table, query_1 ]
for q in queries:
   print("Executing query: %s" % (q))
   res = 'run_query(q, database, s3_ouput)'

Error while testing on AWS Lambda:

  Response:
{
  "errorMessage": "run_query() missing 1 required positional argument: 's3_output'",
  "errorType": "TypeError",
  "stackTrace": [
    "  File \"/var/runtime/bootstrap.py\", line 131, in handle_event_request\n    response = request_handler(event, lambda_context)\n"
  ]
}

Request ID: "2cb2175c-8838-470d-a8dd-efdf4c051312"

Function logs: START RequestId: 2cb2175c-8838-470d-a8dd-efdf4c051312 Version: $LATEST [ERROR] TypeError: run_query() missing 1 required positional argument: 's3_output' Traceback (most recent call last): File "/var/runtime/bootstrap.py", line 131, in handle_event_request response = request_handler(event, lambda_context) END RequestId: 2cb2175c-8838-470d-a8dd-efdf4c051312

import boto3
client = boto3.client('athena')
def run_query(event, context):
    response = client.start_query_execution(
        QueryString=query,
        QueryExecutionContext={
            'Database': 'database'
            },
        ResultConfiguration={
            'OutputLocation': s3_output,
            }
        )
    print('Execution ID: ' + response['QueryExecutionId'])
    return event

Getting the below error:

START RequestId: 55dbf703-f30c-4106-8873-c685f3d06e4d Version: $LATEST [ERROR] NameError: name 's3_output' is not defined Traceback (most recent call last): File "/var/task/lambda_function.py", line 12, in run_query 'OutputLocation': s3_output, END RequestId: 55dbf703-f30c-4106-8873-c685f3d06e4d REPORT RequestId: 55dbf703-f30c-4106-8873-c685f3d06e4d Duration: 14.30 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 75 MB Init Duration: 649.66 ms

Thanks for the response. Tried adding the handler attached above still getting NameError: name 's3_output' is not defined. Any suggestions would be appreciated. — Arunkumar
How is the function being triggered? How are the values for query, database, s3_output being provided to the Lambda function? — John Rotenstein

John Rotenstein John Rotenstein · Accepted Answer · 2020-08-27T04:06:21

This is the same problem as in your earlier question: Error while running a python script on AWS Lambda - Stack Overflow

An AWS Lambda function has this format:

import boto3

def lambda_handler(event, context):
    
    print(event)
    print(context)

When the Lambda function is invoked, it calls the lambda_handler() function. The name of this function can be changed if desired, but it will always receive those two incoming parameters: event and context

Depending upon how the Lambda function is invoked, the contents of the event contains information that is "passed into" the Lambda function. For example:

If the function is invoked from Amazon S3, it contains the name of the bucket and object that triggered the event
If the function is invoked from Amazon SNS, it contains the message that was sent to the SNS topic

It appears that you are wanting to use three values in the function: query, database, s3_output

These values will need to be passed to the Lambda function through whatever means the Lambda function is being invoked. The function can then retrieve those values via the event parameter.

Issue while executing Python script on AWS Lambda function

1 Answers