Checkpoint s3p flink on EMR

0
votes

I have problem with checkpoint by s3p in the flink of EMR.

When creating the EMR cluster, I have a tick in Presto and added jar file as instructed at https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html.

But when checking point by s3p in flink, it still reports

Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 's3p'. The scheme is directly supported by Flink through the following plugin: flink-s3-fs-presto. Please ensure that each plugin resides within its own subfolder within the plugins directory. See https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html for more information. If you want to use a Hadoop file system for that scheme, please add the scheme to the configuration fs.allowed-fallback-filesystems. For a full list of supported file systems, please see https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/.

Can you help me checkpoint s3p on the flink of EMR? Thanks.

1
amazon-s3apache-flinkamazon-emr
In core-site.xml there is Class binding for each fileSystem. When you get UnsupportedFileSystemSchemeException that means you are missing the binding for s3p. You may want to look there 1st. – Snigdhajyoti

1 Answers

0
votes

Presto in EMR has nothing to do with the flink-s3-fs-presto plugin in Flink. You can leave it unticked in the future (doesn't hurt either except blowing things up).

The most likely reason is that you forgot to create a subfolder in the plugins folder. Could you give me an ls of your Flink distribution?