Submit Presto job on dataproc

Question

I am trying to submit a dataproc job on a cluster running Presto with the postgresql connector.

The cluster is initialized as followed:

gcloud beta dataproc clusters create ${CLUSTER_NAME} \
    --project=${PROJECT} \
    --region=${REGION} \
    --zone=${ZONE} \
    --bucket=${BUCKET_NAME} \
    --num-workers=${WORKERS} \
    --scopes=cloud-platform \
    --initialization-actions=${INIT_ACTION}

${INIT_ACTION} point to a bash file with the initialization actions for starting a presto cluster with postgresql.

I do not use --optional-components=PRESTO since I need --initialization-actions to perform non-default operations. And having both --optional-component and --initialization-actions does not work.

When I try to run a simple job:

gcloud beta dataproc jobs submit presto \
  --cluster ${CLUSTER_NAME} \
  --region ${REGION} \
      -e "SHOW TABLES"

I get the following error:

ERROR: (gcloud.beta.dataproc.jobs.submit.presto) FAILED_PRECONDITION: Cluster 
'<cluster-name>' requires optional component PRESTO to run PRESTO jobs

Is there some other way to define the optional component on the cluster?

UPDATE:

Using both --optional-component and --initialization-actions, as:

gcloud beta dataproc clusters create ${CLUSTER_NAME} \
    ...
    --scopes=cloud-platform \
    --optional-components=PRESTO \
    --image-version=1.3 \
    --initialization-actions=${INIT_ACTION} \
    --metadata ...

The ${INIT_ACTION} is copied from this repo. With a slight modification to the function configure_connectors to create a postgresql connector.

When running the create cluster the following error is given:

ERROR: (gcloud.beta.dataproc.clusters.create) Operation [projects/...] failed: Initialization action failed. Failed action 'gs://.../presto_config.sh', see output in: gs://.../dataproc-initialization-script-0_output.

The error output is logged as:

+ presto '--execute=select * from system.runtime.nodes;'
Error running command: java.net.ConnectException: Failed to connect to localhost/0:0:0:0:0:0:0:1:8080

Which leads me to believe I have to re-write the initialization script.

It would be nice to know which initialization script is running when I specify --optional-components=PRESTO.

Unfortunately Dataproc requires that the optional component is used when creating a cluster for the Presto job type to work. You should be able to use the optional component and also specify initialization actions when creating a cluster. Your init actions will run after the optional component is installed, so you can still override the default behavior of the Presto component. Can you describe the error that occurs if you use the optional component and specify an initialization action? — Jerry Ding
> "Your init actions will run after the optional component is installed, so you can still override the default behavior of the Presto component." Nice to know the order on which they are executed, thanks. I will try your suggestion, and update the question later today. I will — jean

KoopaKing KoopaKing · Accepted Answer · 2019-08-31T01:19:30

If all you want to do is setup the optional component to work with a Postgres endpoint writing an optional component to do it is pretty easy. You just have to add the catalog file and restart presto.

https://gist.github.com/KoopaKing/8e653e0c8d095323904946045c5fa4c2

Is an example init action. I have tested it successfully with the presto optional component, but it is pretty simple. Feel free to fork the example and stage it in your GCS bucket.

Submit Presto job on dataproc

1 Answers