Problem HTTP error 403 in Python 3 Web Scraping the publications

1
votes

This is the traceback of the error that is happening when I am trying to put the URL of the publication. It works for the regular websites such as stackoverflow or Wikipedia, but when I try it on the publications such as https://www.sciencedirect.com/science/article/pii/S1388248120302113?via%3Dihub, The error shows up. Can anyone help me??

Here is my code: req = Request(' https://www.sciencedirect.com/science/article/pii/S1388248120302113?via%3Dihub', headers={'User-Agent': 'Mozilla/5.0'}) html_plain = urlopen(req).read()

Here is the traceback of the error:

File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1776.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 214, in urlopen return opener.open(url, data, timeout) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1776.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 523, in open response = meth(req, response) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1776.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 632, in http_response
response = self.parent.error( File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1776.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 561, in error return self._call_chain(*args) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1776.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 494, in _call_chain result = func(*args) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1776.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 641, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 403: Forbidden

thankyou!

1
pythonhtmlhttpweb-scrapingnlp

1 Answers

0
votes

The HTTP 403 Forbidden client error status response code indicates that the server understood the request but refuses to authorize it. This is not an error in the code, this is the website refusing to server that page when you are doing the web scraping.