Search the entire web using a Google API on localhost

3
votes

I would like to query Google Search on many topics (~800 queries in total), for instance "cars", to get the first 100 results on localhost using Python. How can I do it?

Google Custom Search Engine seems to work only if it's used on a website (we have to provide an address of the website and localhost is refused).

The old Google Web Search API is deprecated. And it's limited in the number of results (we can't get 100 results as I would like).

This small independent Google Search tool could also works for a request or two, but I guess I will be banned if I try ~800 requests

1
google-searchgoogle-custom-searchgoogle-search-api
My interpretation here is that Google does not want you to use their service as a revenue-free source for data-mining. Otherwise, you could simply set up a competing search engine that leverages Google results, but put your own ads on the results. You will have to choose to either violate the terms of service or get your information another way. I recommend the latter. If this is legit serious research, you might be able to contact Google about it. That's a long shot though. – Trott

1 Answers

1
votes

There is a google support page describing how to do this.

https://support.google.com/customsearch/answer/2631040?hl=en

It says:

On the Custom Search home page, click the search engine you want.
Click Setup, and then click the Basics tab.
Select Search the entire web but emphasize included sites.
In the Sites to search section, delete the site you entered during the initial setup process.

It does seem to be working for me. I imagine google have enabled this now that they charge for lookups after the first 100 per day.