ApplicationMaster (YARN-Spark) for each job submit?

Question

This probably is more like a "theoretical" or "good practices" question rather than strictly practical (no problem with codes, or cluster configuration files).

So, following this simple scenario:

• Submit via RESt (Apache Livy), say, 10 spark jobs to a YARN-SPARK cluster,
• due to resource management configurations, 5 of them are running and 5 accepted/pending,

would this result in 10 AM instances running concurrently in the Master node (consuming a lot of ram), right ?

If thats the case, is there any other approach ? Considering this:

• The job requests are fast,
• each time, the cluster would receive almost 1000 requests,
• each job takes an aprox. of 15 secs long to complete (sometimes less depending on the amount of data received to process in each call),
• limited ammount of resources (3 workers with 6gb and 4 cores each + master)

Answer 1

Let me know if i have correctly understood your question. With separate spark-submit , you mean to invoke separate requests. Thus, all requests will go to yarn-scheduler and as per resource availability , they will be in running or submitted or pending state.

Now, we have two ways to submit job on yarn -> client mode and cluster mode.

If you are submitting your jobs in client mode, the driver runs in the client process, and the application master is only used for requesting resources from YARN. So, different job requests will have different drivers (and associated AMs) created on the same machine. This should concern you.

If you are submitting your jobs in cluster mode, the driver runs inside an application master process which is managed by YARN on the cluster. Thus, different job requests will have different Application Masters(drivers inside) created on different machines.

So, looking at your use-case, it is obvious to use load balancer for large no. of the incoming requests. And submit your jobs in cluster mode. Following it, you can tune your application with best practices properties.