Flink autoscaling and max parallelism

Question

Quote from javadoc on StreamExecutionEnvironment.setMaxParallelism: The maximum degree of parallelism specifies the upper limit for dynamic scaling.

Which exactly dynamic scaling is meant here? I couldn't find any empirical evidence of operator auto scaling: whatever number of free slots you have, and no matter how big is maxParallelism, and how many logical partitions is there, the actual parallelism (according to the web ui) is always the one that was set through a setParallelism

Also, according to this the most accepted and never challenged answer https://stackoverflow.com/a/43493109/2813148 there's no such thing as dynamic scaling in Flink.

So is there any? Or the javadoc is misleading (or what's the meaning of "dynamic" there)? If there's none, are there any plans for this feature?

Answer 1

Flink (in version 1.5.0) does not support dynamic scaling yet.

However, job can be manually scaled (or by an external service) by taking a savepoint, stopping the running job, and restarting the job with an adjusted (smaller or larger) parallelism. However, the new parallelism can be at most the previously configured max-parallelism. Once a job was started, the max-parallelism is baked into the savepoints and cannot be changed anymore.

Support for dynamic scaling is on the roadmap. Since version 1.5.0 (released in May 2018), Flink supports dynamic resource allocation from resource managers such as Yarn and Mesos. This is an important step towards dynamic scaling. In fact, an experimental version of this feature has been demonstrated at Flink Forward SF 2018 in April 2018.