21
votes

I need to do a very large search on Github for a statistic in my thesis.

For example, I need to explore a large number of Android projects on GitHub, but the site limits the search result to 1000 (ex. https://github.com/search?l=java&q=onCreate&ref=searchresults&type=Code&utf8=%E2%9C%93). Also using the Java GitHub API I tried the library org.eclipse.egit.github.core.client.GitHubClient using the method GitHubClient.searchRepositories() but even there the number of results is limited.

Does anyone know how to get all results?

2
Have you looked at the GitHub Archive? It could be a way to get your data without having to bother the live GitHub search API, which as you found out gives a limited number of results, and is also rate-limited. - Wander Nauta
Are you able to page through the results? You could get the first chunk of 1000, get the next chunk, and repeat until you have it all. - Kyle Falconer
This is not a Java question, or even a programming question. - shmosel
Correct, you're limited to 1000 results per search & 30 requests per minute: developer.github.com/v3/search/#about-the-search-api - zapl
Is your code publicly available? - Soubriquet

2 Answers

34
votes

The Search API will return up to 1000 results per query (including pagination), as documented here:

https://developer.github.com/v3/search/#about-the-search-api

However, there's a neat trick you could use to fetch more than 1000 results when executing a repository search. You could split up your search into segments, by the date when the repositories were created. For example, you could first search for repositories that were created in the first week of October 2013, then second week, then September, and so on.

Because you would be restricting search to a narrow period, you will probably get less than 1000 results, and would therefore be able to get all of them. In case you notice that more than 1000 results are returned for a period, you would have to narrow the period even more, so that you can collect all results.

https://help.github.com/articles/searching-repositories/#search-based-on-when-a-repository-was-created-or-last-updated

You should be able to automate this via the API.

5
votes

If you are searching for all files in Github with filename:your-file-name, you could also slice it with a query attribute : size.

For example, you are looking for all files named test.rb in Github, Github API may return more than 11M results, but you could only get 1000 of them because the GitHub Search API provides up to 1,000 results for each search. An url like : https://api.github.com/search/code?q=filename:test.rb+size:1000..1500 would be able to slice your search by changing size range.