How to set optional properties on Google Dataproc Cluster using Java API?

Question

I am trying to create Dataproc cluster using Java API following this documentation https://cloud.google.com/dataproc/docs/quickstarts/quickstart-lib

Sample code is as below


  public static void createCluster() throws IOException, InterruptedException {
    // TODO(developer): Replace these variables before running the sample.
    String projectId = "your-project-id";
    String region = "your-project-region";
    String clusterName = "your-cluster-name";
    createCluster(projectId, region, clusterName);
  }

  public static void createCluster(String projectId, String region, String clusterName)
      throws IOException, InterruptedException {
    String myEndpoint = String.format("%s-dataproc.googleapis.com:443", region);

    // Configure the settings for the cluster controller client.
    ClusterControllerSettings clusterControllerSettings =
        ClusterControllerSettings.newBuilder().setEndpoint(myEndpoint).build();

    // Create a cluster controller client with the configured settings. The client only needs to be
    // created once and can be reused for multiple requests. Using a try-with-resources
    // closes the client, but this can also be done manually with the .close() method.
    try (ClusterControllerClient clusterControllerClient =
        ClusterControllerClient.create(clusterControllerSettings)) {
      // Configure the settings for our cluster.
      InstanceGroupConfig masterConfig =
          InstanceGroupConfig.newBuilder()
              .setMachineTypeUri("n1-standard-1")
              .setNumInstances(1)
              .build();
      InstanceGroupConfig workerConfig =
          InstanceGroupConfig.newBuilder()
              .setMachineTypeUri("n1-standard-1")
              .setNumInstances(2)
              .build();
      ClusterConfig clusterConfig =
          ClusterConfig.newBuilder()
              .setMasterConfig(masterConfig)
              .setWorkerConfig(workerConfig)
              .build();
      // Create the cluster object with the desired cluster config.
      Cluster cluster =
          Cluster.newBuilder().setClusterName(clusterName).setConfig(clusterConfig).build();

      // Create the Cloud Dataproc cluster.
      OperationFuture<Cluster, ClusterOperationMetadata> createClusterAsyncRequest =
          clusterControllerClient.createClusterAsync(projectId, region, cluster);
      Cluster response = createClusterAsyncRequest.get();

      // Print out a success message.
      System.out.printf("Cluster created successfully: %s", response.getClusterName());

    } catch (ExecutionException e) {
      System.err.println(String.format("Error executing createCluster: %s ", e.getMessage()));
    }
  }
}

So based on the documentation i am able to create it successfully but there are few optional properties which i am not able to figure out how to set it here, for reference below screenshot of it which can be done using the Google Cloud console.

These properties can be added using Google Cloud SDK as below

gcloud dataproc clusters create my-cluster \
    --region=region \
    --properties=spark:spark.executor.memory=4g \
    ... other args ...

How to set this using Java API. Also the exact same thing for Labels as well, how can we set the labels on the cluster using Java API.

Henry Gong Henry Gong · Accepted Answer · 2020-07-14T05:06:16

You can check complete API reference for Dataproc Java client library.

Specifically, to set properties you want to look at SoftwareConfig.Builder. Similarly you can associate a label to a Cluster with Cluster.Builder.

How to set optional properties on Google Dataproc Cluster using Java API?

1 Answers