Show real-time progress with GNU Parallel and Stata

1
votes

I'm using GNU parallel to run a Stata do file for many different data sets.

I have a Bash script that contains the following: 

parallel -a arguments.txt -j 3 stata -b do $dofileloc {}

Since the do file has several different parts for each dataset, I would like to have the progress shown "real-time" (e.g. display "data loaded for XYZ" after a part of the Stata do file finishes for a dataset etc.).

So I'd like to redirect messages from Stata to the command line, but I'm having trouble doing this.

If I don't run Stata in batch mode I can see everything, which is a bit messy. I have tried using the shell command in Stata but I can't seem to figure out the correct combination.

I would appreciate any tips.

1
bashstatagnu-parallel
To my knowledge you can't do that - at least not in real time. Why don't you break up your do file in smaller ones for each dataset? Then you could simply include a message in your bash script after each do file ends. - user8682794
Add a filter at the end maybe? parallel ... | grep "loaded" - Mark Setchell
Thank you both for the help and apologies for the delay, I've had a lot on my plate and was trying to make some progress with this question before responding. Unfortunately the filter didn't work. The reason why I don't want to break into do files for each dataset is because it is the same code for all of the datasets (there are 30+ of them) which is why I parallelized the process in the first place. I suspect Stata's batch mode might have to do with the problem - any maybe why it's not possible to do this. Thanks again! - Paprika

1 Answers

0
votes

Does this do what you want?

parallel --tag --linebuffer -a arguments.txt -j 3 stata -b do $dofileloc {}