How to download many PDF files using World Bank API by Python

Question

I am trying to download many pdf files (few hundreds) from the World Bank archive website using Python. The API web-link can be customized using any choice of terms (for instance, country or sector- education, health etc.).

I have tried the following code using the mentioned url to download files that are specific to the education sector in Vietnam. The URL contains the operational document with all pdf links with the specified terms. However, the files cannot be downloaded.

import requests
from urllib.parse import urljoin
from bs4 import BeautifulSoup

url = "http://search.worldbank.org/api/v2/wds?format=json&countcode=VN&majdocty_exact=Publications&teratopic_exact=Education&srt=docdt&order=desc"

#Folder to download the files
folder_location = r'J:\New Volume (B)\pdfs'

response = requests.get(url)
soup= BeautifulSoup(response.text, "html.parser")     
for link in soup.select("a[href$='.pdf']"):
    #Name the pdf files 
    filename = os.path.join(folder_location,link['href'].split('/')[-1])
    with open(filename, 'wb') as f:
        f.write(requests.get(urljoin(url,link['href'])).content)

I do not find any error after running the code, but nor can I download any file. Any help would be highly appreciated. Thank you.

Hi thanks, could you help me how the code should be written then? @SmartManoj — Ripon

Smart Manoj Smart Manoj · Accepted Answer · 2019-06-05T02:53:11

Use response.json(); bs4 not needed

import requests
from urllib.parse import urljoin
from bs4 import BeautifulSoup

url = "http://search.worldbank.org/api/v2/wds?format=json&countcode=VN&majdocty_exact=Publications&teratopic_exact=Education&srt=docdt&order=desc"

#Folder to download the files
folder_location = r'J:\New Volume (B)\pdfs'

response = requests.get(url).json()
for i in response['documents']:
    url=(response['documents'][i].get('pdfurl'))
    if url:
        filename = os.path.join(folder_location,url.split('/')[-1])
        with open(filename, 'wb') as f:
            f.write(requests.get(url).content)

How to download many PDF files using World Bank API by Python

1 Answers