2
votes

What is the difference between GNU-parallel and the python multiprocessing module? And also which one would be ideal for which circumstances, if they differ in usage.

I am trying to parallelize tesseract and found someone recommending GNU parallel here on the tesseract issues.

I want to understand, which one would be ideal for which use case, before going with one over the other.

Also, I'm not getting the desired results from gnu-parallel, I can see 4 processes running in top, but they take a lot more time than python multiprocessing.

1)For gnu-parallel:

Time taken is 8 min 40s as can be seen here

I am using the following command:

ls image*.jpg | time parallel tesseract {} stdout -l hin

top output is here

2) Normal Tesseract using multipage feature.

$ time tesseract imagelist.txt stdout -l hin

Speed can be seen here

3) Multiprocessing based pytesseract

gives a greater speed increase to about 4-5 seconds.

My pdf can be found here

I am using convert_from_path from pdf2image or convert from imagemagick to convert pdf to image as either png or jpeg.

1
Given that GNU Parallel takes 8m40s and normal multipage tesseract takes 0m12s I think they are not doing the same. The difference in time is simply too big. If you ran the first run with GNU Parallel, maybe tesseract cached output somehow and the other runs then simply used the cache. - Ole Tange
@OleTange : Hi Ole thanks for the reply! I don't think, so the problem is with the caching (although I don't understand how this could happen, and would love to know more), since my friend who hasn't got gnu-parallel installed gets the same speed. I am still thinking that I maybe using gnu-parallel wrongly, or some other reason. I can link to the pdf or images here, if you'd like to test them yourself, it'd also be a huge help. - aspiring1
Please do. That way the question will be an MCVE. - Ole Tange
@OleTange : I've added the pdf link in the question at the bottom - aspiring1
Wow! I can actually reproduce your findings: tesseract becomes crazy slow. It goes faster if you run --tty -j+0, but the multipage feature is the fastest. - Ole Tange

1 Answers

0
votes

In python multiprocessing you have possibility of communication between processes. But then you have cost of synchroniazation. gnu pararell runs command multiple time with different parameters. Then You need aggregate results in other process.