incrementing a number in the url to make simultaneous requests with curl

0
votes

this is my current curl request. i am trying to figure out how to increment the "details" value by 1 (so the next requests will be 0000002 and so on and so forth). i want it to loop through 1000 values then output the responses to each request to a text file. any help will be greatly appreciated!

curl "https://webapi.xxx.com/api/getthisdetails?details=0000001" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0" -H "Accept: application/json, text/javascript, /; q=0.01" -H "Accept-Language: en-US,en;q=0.5" -H "Content-Type: application/x-www-form-urlencoded; charset=UTF-8" -H "Authorization: Bearer xxxxxxxxxxxxxxxxx" -H "Origin: https://www.xxxxxx.com" -H "Connection: keep-alive" -H "Referer: https://www.xxxxxxx.com/" -H "Pragma: no-cache" -H "Cache-Control: no-cache"

1
curl

1 Answers

0
votes

curl can iterate over them by itself. By avoiding a shell loop you will have curl reuse the connection:

curl "https://webapi.xxx.com/api/getthisdetails?details=[0000001-0001000]"  -o file_#1 [the other headers]

Not that the #1 in the -o output will use the full number from within the brackets, starting with 0000001.

Parallel

The above command line will do all requests in a serial manner, one after the other. If you rather want to instead do them in parallel, add --parallel to the command line and curl will do up to 50 (by default, can be changed) transfers simultaneously.

--parallel was added to curl 7.66.0