'NoneType' object has no attribute 'text' in getting google results

Question

I asked in this post How to get the number of results from google? more information on how to get the number of results from google in a date range, but it was closed as duplicate. Unfortunately the two other results that were suggested have not fixed my issues in getting this number. Using the following code:

import requests
from bs4 import BeautifulSoup
import argparse


parser = argparse.ArgumentParser(description='Get Google Count.')
parser.add_argument('word', help='word to count')
args, unknown = parser.parse_known_args()

r = requests.get('http://www.google.com/search',
                 params={'q':'"'+args.word+'"',
                         "tbs":"li:1"}
                )

soup = BeautifulSoup(r.text)
print (soup.find('div',{'id':'resultStats'}).text)

I have found this error:

--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) in 14 15 soup = BeautifulSoup(r.text) ---> 16 print (soup.find('div',{'id':'resultStats'}).text)

AttributeError: 'NoneType' object has no attribute 'text'

Also trying with this code:

import requests
from bs4 import BeautifulSoup

search = "manchester"

r = requests.get("https://www.google.com/search", params={'q':search})

soup = BeautifulSoup(r.text, "lxml")
res = soup.find("div", {"id": "resultStats"})
print (res.text)
print int(res.text.replace(",", "").split()[1])

I got the same error.

Do you have any idea on how to fix it? thanks

Sam Sam · Accepted Answer · 2020-03-08T17:12:49

The error means that your call to requests.get returns nothing.
There could be several reasons, most of them network related (no DNS, no web access...).
Can you access that URL via curl on the host where your script is running?
I tried to run this script on my system and Google was not happy. Apparently it is against their terms of services. I think their API is the way to go here.

'NoneType' object has no attribute 'text' in getting google results

1 Answers