2
votes

I am trying to download a large number of files from a webpage (which contains only a image, so I can use a simple wget), but want to speed it up using GNU Parallel. Can anyone please help me parallelize this for loop? Thanks.

    for i in `seq 1 1000`
    do
    wget -O "$i.jpg" www.somewebsite.com/webpage
    done
1
Append a whitespace and & at the end of your wget command. - Cyrus
A C-style for loop is better than using seq in bash: for ((i=1; i<=1000; i++)). - chepner

1 Answers

4
votes

You could do it like this:

 seq 1 1000 | parallel wget www.somewebsite.com/webpage/{}.jpg

You can also use the -P option to specify the number of jobs you want to run concurrently.

Also you may decide to use curl instead like:

parallel -P 1000 curl -o {}.jpg www.somewebsite.com/webpage/{}.jpg ::: {1..1000}