Using GoLang SDK for google cloud storage.
Cannot find how to download files in chunks.
0
votes
Please, provide more information what you mean by downloading files in chunks. There is documentation for Google Cloud Storage: godoc.org/cloud.google.com/go/storage. As you can see it exposes a bucket objects as a regular reader interface. You could implement your own reader however you need to.
- zdebra
Downloading file in chunks means downloading file in chunks. That's plain english.
- ND003
1 Answers
3
votes
The Google Cloud documentation says to download an object from Cloud Storage, you should use the following:
rc, err := client.Bucket(bucket).Object(object).NewReader(ctx)
if err != nil {
return nil, err
}
defer rc.Close()
data, err := ioutil.ReadAll(rc)
if err != nil {
return nil, err
}
return data, nil
Source: https://cloud.google.com/storage/docs/downloading-objects#storage-download-object-code_sample
Given their SDK returns an io.Reader, you don't need to worry about the underlying method being used to be able to reference the download in chunks (although, quickly looking through their source, it just implements http.NewRequest, which does what you want, using the same logic).
The reason it doesn't seem to "chunked" from their example is because of the usage of ioutil.ReadAll, which although great for simple use cases, extracts all of the Readers data into memory (meaning it also has to wait for the data to become available).
For a better understanding of how to deal with a Reader in steps, I recommend taking a look at https://tour.golang.org/methods/21 for a tour of io.Reader and how you can use it more efficiently.
