0
votes

I am trying to deploy presto om EMR through our EMR launch config JSON. I have decided the config properties as advised in this github issue of presto. I have added the following presto properties in the launch config

{
    "Classification": "presto-connector-hive",
    "Properties": {
        "hive.metastore.glue.datacatalog.enabled": "true",
        "hive.table-statistics-enabled": "true"
    },
    "Configurations": []
},
{
    "Classification": "presto-config",
    "Properties": {
        "query.max-memory": "150G",
        "query.max-memory-per-node": "20G",
        "query.max-total-memory-per-node": "30G",
        "memory.heap-headroom-per-node": "10G",
        "query.initial-hash-partitions": "15"
    },
    "Configurations": []
}   

EMR cluster has been created but presto is failing due to the following errors

1) Explicit bindings are required and com.facebook.presto.memory.LowMemoryKiller is not explicitly bound.
  while locating com.facebook.presto.memory.LowMemoryKiller
    for parameter 7 at com.facebook.presto.memory.ClusterMemoryManager.<init>(ClusterMemoryManager.java:123)
  at com.facebook.presto.server.CoordinatorModule.setup(CoordinatorModule.java:189) (via modules: com.facebook.presto.server.ServerMainModule -> com.facebook.presto.server.CoordinatorModule)

2) Error: Could not coerce value '150G' to io.airlift.units.DataSize (property 'query.max-memory') in order to call [public com.facebook.presto.memory.MemoryManagerConfig com.facebook.presto.memory.MemoryManagerConfig.setMaxQueryMemory(io.airlift.units.DataSize)]

3) Error: Could not coerce value '20G' to io.airlift.units.DataSize (property 'query.max-memory-per-node') in order to call [public com.facebook.presto.memory.NodeMemoryConfig com.facebook.presto.memory.NodeMemoryConfig.setMaxQueryMemoryPerNode(io.airlift.units.DataSize)]

4) Configuration property 'memory.heap-headroom-per-node' was not used
  at io.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:234)

5) Configuration property 'query.max-memory' was not used
  at io.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:234)

6) Configuration property 'query.max-memory-per-node' was not used
  at io.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:234)

7) Configuration property 'query.max-total-memory-per-node' was not used
  at io.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:234)

7 errors
    at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:466)
    at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:155)
    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:107)
    at com.google.inject.Guice.createInjector(Guice.java:96)
    at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:241)
    at com.facebook.presto.server.PrestoServer.run(PrestoServer.java:114)
    at com.facebook.presto.server.PrestoServer.main(PrestoServer.java:66)

My config.properties file

coordinator=true
node-scheduler.include-coordinator=false
discovery.uri=X.X.X.X:YYYY
http-server.threads.max=500
discovery-server.enabled=true
sink.max-buffer-size=1GB
query.max-memory=150G
query.max-memory-per-node=20G
query.max-history=40
query.min-expire-age=30m
http-server.http.port=8889
http-server.log.path=/var/log/presto/http-request.log
http-server.log.max-size=67108864B
http-server.log.max-history=5
log.max-size=268435456B
log.max-history=5
query.initial-hash-partitions = 15
memory.heap-headroom-per-node = 10G
query.max-total-memory-per-node = 30G
1
"150G" is not correct. It should be "150GB" ("B" at the end). Same for other memory related properties.Piotr Findeisen
Thanks, but I am still facing the error 4 and 7Ayush Chauhan
Which Presto version is this? (memory.heap-headroom-per-node is not that old a property)Piotr Findeisen
I am using Presto 0.194Ayush Chauhan
That version didn't have the memory.heap-headroom-per-node and query.max-total-memory-per-node properties.Piotr Findeisen

1 Answers

3
votes

Setup fails because

  1. You need to use "GB" (not "G") as the unit when setting data size config properties
  2. Your version (0.194) doesn't support some properties that you're setting (memory.heap-headroom-per-node nor query.max-total-memory-per-node).