Cannot install Ganglia on EMR 4.0.0

Question

I'm following this guide to install Spark on the latest AMI / EMR cluster:

http://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-spark-launch.html

I'd like to install Ganglia to monitor the cluster so I added Name=Ganglia to the list of applications to install:

aws emr create-cluster --name "Spark cluster" --release-label emr-4.0.0 --applications Name=Spark Name=Ganglia --ec2-attributes KeyName=myKey --instance-type m3.xlarge --instance-count 3 --use-default-roles

But I'm receiving the following error message:

A client error (ValidationException) occurred when calling the RunJobFlow operation: Specified application: Ganglia is invalid

Here are the versions that I am running:

aws --version
aws-cli/1.7.41 Python/2.7.7 Linux/2.6.32-431.29.2.el6.x86_64

Answer 1

Ganglia is not part of the EMR 4.0 release.

The official API documentation shows valid values of: "Hadoop", "Hive", "Mahout", "Pig", and "Spark."

It appears that the AWS CLI documentation is incorrect.

Answer 2

The usual Ganglia bootstrap action also doesn't work under emr-4.0.0. See this issue.

Answer 3

Amazon is probably working on creating an official release of Ganglia for EMR 4.x. Until that comes out, you can use this bootstrap action:

s3://support.elasticmapreduce/release/4.x/ganglia/install_ganglia_emr-4.0.0.rb

Answer 4

If you continue by using a bootstrap action, you should be ok.

// AWSCLI Example

aws emr create-cluster                  \
  --bootstrap-actions file://bootstrap_actions.json \
  ...

// bootstrap_actions.json

{
    "Name": "Install Ganglia",
    "Path": "s3://elasticmapreduce/bootstrap-actions/install-ganglia"
  },

Or from DataPipeline (pipeline definition file example):

   {
      "id": "EmrCluster",
      "name": "My Cluster (staging)",
      "type": "EmrCluster",
      "bootstrapAction": [
        "s3://elasticmapreduce/bootstrap-actions/install-ganglia"
      ],
      etc..
    },