Making network/API calls from a Kinesis Firehose transformation Lambda

Question

I have a use-case in which I need to make an API call for the payload sent to my Kinesis Firehose stream before storing it in S3.

The flow would be: Kinesis Data Stream -> Kinesis Firehose -> Transformation Lambda -> API call to get additional data relating to current records -> Kinesis Firehose -> S3.

Basically, for a record that is consumed by my Kinesis Firehose stream, I need to call another backend service to get additional data related to the record before storing in S3 for our EMR jobs to consume and write queries on.

My question is, is it possible to make network calls from a Kinesis Firehose transformation Lambda. I think it should be since it's just another Lambda function. I would also like to understand if it's against best practices to make API calls in a Kinesis Firehose transformation Lambda.

Any insight is appreciated!

Answer 1

I dont think there is any problem in making network calls in transformation lambda function. Only thing you need to make sure is that you are returning all the recordIds back to firehose after transformation in firehose accepted format as shown in aws doc