AWS lambda function to submit spark job

1
votes

I would like to use Spark-sumit REST apis in AWS lambda function to submit jobs.I have looked into Spark-Jobserver but looks like it needs to be deployed in a\n Ec2 Instance,but i would like to submit jobs using AWS Lambda function using Spark REST interface.Please suggest.

1
apache-sparkaws-lambda

1 Answers

2
votes

You can add step to the EMR cluster from the lambda. The Step can be shell script,hive, or spark job.

The function that helps to do that is

emr.addJobFlowSteps(params, function(err, data)

I am adding the link for the api calls available on EMR http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/EMR.html