How do I identify the Google Cloud Storage URI from my Google Developers Console?

58
votes

When I attempt load data into BigQuery from Google Cloud Storage it asks for the Google Cloud Storage URI (gs://). I have reviewed all of your online support as well as stackoverflow and cannot find a way to identify the URL for my uploaded data via the browser based Google Developers Console. The only way I see to find the URL is via gsutil and I have not been able to get gsutil to work on my machine.

Is there a way to determine the URL via the browser based Google Developers Console?

6
google-cloud-platformgoogle-bigquerygoogle-cloud-storage
Could you post (maybe a separate question) the problems you encountered trying to set up gsutil? – jterrace
yes, how did you upload your data? If it's small enough you can upload it straight to BigQuery – Felipe Hoffa
I uploaded data via the google developers console. The suggestion to try gs://bucket/file name worked. This was very helpful. – Kelly

6 Answers

115
votes

The path should be gs://<bucket_name>/<file_path_inside_bucket>.

5
votes

To answer this question more information is needed. Did you already load your data into GCS?

If not, the easiest would be to go to the project console, click on project, and Storage -> Cloud Storage -> Storage browser.

You can create buckets there and upload files to the bucket.

Then the files will be found at gs://<bucket_name>/<file_path_inside_bucket> as @nmore says.

2
votes

Couldn't find a direct way to get the url. But found an indirect way and below are the steps:

  1. Go to GCS
  2. Go into the folder in which the file has been uploaded
  3. Click on the three dots at the right end of your file's row
  4. Click rename
  5. Click on gsutil equivalent link
  6. Copy the url alone
2
votes

Follow the following steps :
1. Go to GCS
2. Go into the folder in which the file has been uploaded
3. On the top you can see overview option
4. You can see there will be Link URL and link for GSUtil

1
votes

Retrieving the Google Cloud Storage URI To create an external table using a Google Cloud Storage data source, you must provide the Cloud Storage URI.

The Cloud Storage URI comprises your bucket name and your object (filename). For example, if the Cloud Storage bucket is named mybucket and the data file is named myfile.csv, the bucket URI would be gs://mybucket/myfile.csv. If your data is separated into multiple files you can use a wildcard in the URI. For more information, see Cloud Storage Request URIs.

BigQuery does not support source URIs that include multiple consecutive slashes after the initial double slash. Cloud Storage object names can contain multiple consecutive slash ("/") characters. However, BigQuery converts multiple consecutives slashes into a single slash. For example, the following source URI, though valid in Cloud Storage, does not work in BigQuery: gs://[BUCKET]/my//object//name.

To retrieve the Cloud Storage URI:

Open the Cloud Storage web UI.

CLOUD STORAGE WEB UI

Browse to the location of the object (file) that contains the source data.

At the top of the Cloud Storage web UI, note the path to the object. To compose the URI, replace gs://[BUCKET]/[FILE] with the appropriate path, for example, gs://mybucket/myfile.json. [BUCKET] is the Cloud Storage bucket name and [FILE] is the name of the object (file) containing the data.

0
votes

If you need help on subdirectories, check this out on https://cloud.google.com/storage/docs/gsutil/addlhelp/HowSubdirectoriesWork

And https://cloud.google.com/storage/images/gsutil-subdirectories-thumb.png, if you need to see how gsutil provides a hierarchical view of objects in a bucket.