Is it possible to parallelize preprocessings with tensorflow-transform on my machine?

Question

I am trying to preprocess larger amounts of data (one tfrecord file ~1Go) using tensorflow-transform v0.11.0 and beam only locally.

My code is largely inspired from https://github.com/tensorflow/transform/blob/master/examples/census_example.py

I have a beam pipeline that works on smaller datasets (<100Mo) but the processing time increases dramatically as I add more data. Being new to tf-transform and apache Beam, I have a hard time finding causes and solutions to the problem... And I would like to avoid using google DataFlow.

My pipeline runs locally using beam directRunner, if I understood correctly, but it uses only one core. Using multiple cores could be one way to improve my preprocessing time, but I do not know if that is possible with the beam directRunner. Is there a way to make a tensorflow-transform pipeline run on multiple cores on my machine ?

I looked in the options of the beam pipeline and of the directRunner, and I can't find any indication about letting a runner access multiple cores or creating multiple directRunners for a pipeline.

Thank you very much for any help I could get !

Answer 1

To add to Anton's comment, You can utilize Apache Flink to run the pipeline in parallel. More details are summarized in Tensorflow transform on beams with flink runner

You will also have to set the parallelism according to the total number of cores and start those many Flink TaskManagers. My recommendation would be to set parallelism to (total number of cores/2)

Answer 2

I don't believe that's supported. Direct runner's main purpose is to make sure the pipeline implements Beam model correctly. It is not optimized for production use, and will probably actually introduce inefficiencies: https://beam.apache.org/documentation/runners/direct/

As a workaround you can manually start multiple direct runner pipelines to process different portions of data.

Better option would be to use an actual parallel runner to run these kinds of jobs, e.g. you can spin up a Flink cluster: https://beam.apache.org/documentation/runners/flink/

Answer 3

@Ankur @Anton Thanks for your answers, I agree that this approach is not production friendly... We will try two other solutions:

• tensorflow-transform on DataFlow
• removing tensorflow-transform altogether and use presto to get vocabulary files for categorical inputs, compute means and standard deviations to scale numerical inputs, etc on the whole dataset