Programmatically listing Bigquery datasets to which access has been granted, without knowing project ID in advance

3
votes

Is there a way (using the BigQuery API) to programmatically list all datasets to which an account has been granted access, without knowing the name of the project(s) containing these datasets ahead of time? I have a feeling that the API may not expose this information.

I am looking to obtain something similar to the list of datasets which appears on the left hand side of the BigQuery web console.

I am referring to datasets where I do not have access to the project which contains them, and have been granted access to a specific dataset only. Otherwise, we could simply use the API to list projects and then query the list of datasets for each project.

Presumably this would have to exclude public datasets (too numerous), and include only those to which my account has been explicitly been granted access.

3
javagoogle-bigquerygoogle-cloud-platform
(btw, took a second to edit out the thanks in your post. While it's definitely a nice thought to put it in, it's considered noise on Stack :) you show respect and appreciation by posting quality posts and voting according to quality :). Welcome to the site!) – Patrice
Have you tried something on your own? – ivan.mylyanyk
I have tried exploring the API, but I cannot find anything which looks like it will work. projects().list() only returns projects for which you have project-level access. And datasets().list() requires projectId as a parameter. – Jason Davey
If you already know the name of a project containing a dataset to which you have access, then you can list the datasets to which you have access using datasets().list(projectId) (even if you don't have project-level access). – Jason Davey

3 Answers

0
votes

There's currently no way to do this in the BigQuery API. As you say, you can get a list of projects that you're a member of, and then list the datasets in those projects, but there's no way to get a complete list of all datasets that have been shared with you.

We'd like to offer better search and discovery features in the future, but unfortunately it's not possible right now.

0
votes

Google has API explorers for most its tools. Whenever you have such a question, I would suggest looking up the explorer, as it gives you exactly what's mandatory, what you can and cannot do.

If you look up the API explorer here (from "dataset -> list" in the official doc), you see there is only ONE mandatory field : projectID. So the answer to your question is no, you DO need the name of the project

0
votes

Using BQ command line tool:

bq ls --format=prettyjson

this returns a JSON formatted like:

[
    {
        "datasetId": "wr_1"
    },
    {
        "datasetId": "wr_2"
    },
    {
        "datasetId": "wr_3"
    },
    {
        "datasetId": "wr_4"
    }
]

You can do this also via API but you need to do multiple calls