download large file from drive to colab

0
votes

I have a link to a public Google Drive hosted file:

https://drive.google.com/uc?id=19VsarMcYRNPLTDr6b6ABJyY8JUeBueL8&export=download

Below is a .sh script that works for a different file and link:

#!/usr/bin/env bash
function gdrive_download () { # credit to https://github.com/ethanjperez/convince
  CONFIRM=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate "https://docs.google.com/uc?export=download&id=$1" -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')
  wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$CONFIRM&id=$1" -O $2
  rm -rf /tmp/cookies.txt
}

mkdir -p Models/real-fixed-cam Models/real-hand-held
gdrive_download 1yiNsSkPYoBZ55fSQ1iwb1io9QL_PcR2i Models/real-fixed-cam/netG_epoch_12.pth
gdrive_download 13HckO9fPAKYocdB_CAC5n8uyM3xQ2MpG Models/real-hand-held/netG_epoch_12.pth

The above script is called in Colab with this:

!wget https://gist.githubusercontent.com/andreyryabtsev/458f7450c630952d1e75e195f94845a0/raw/0b4336ac2a2140ac2313f9966316467e8cd3002a/download.sh
!chmod +x download.sh
!./download.sh

I have adapted it as follows to fit my needs:

#!/usr/bin/env bash
function gdrive_download () { # credit to https://github.com/ethanjperez/convince
  CONFIRM=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate "https://docs.google.com/uc?export=download&id=$1" -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')
  wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$CONFIRM&id=$1" -O $2
  rm -rf /tmp/cookies.txt
}

mkdir -p pix2pix/checkpoint
gdrive_download 19VsarMcYRNPLTDr6b6ABJyY8JUeBueL8 pix2pix/checkpoint/weights.zip

The above code is called in colab with:

!wget https://gist.githubusercontent.com/Daryl149/070397c9cb3539f5cd01173f6c44200d/raw/207a76e94e70e6c9334f48c25b4998f4fd1b95e3/download.sh
!chmod +x download.sh
!./download.sh

The folder is correctly created. But instead of downloading a 500mb+ zip file to the checkpoints folder, it actually downloads the html from the Download confirmation page. In the logging, the script does seem to pick up a fresh download confirmation string every time that should normally force a Google Drive download without virus scan:

--2020-07-27 21:55:21--  https://drive.google.com/uc?export=download&confirm=umyj&id=19VsarMcYRNPLTDr6b6ABJyY8JUeBueL8
Resolving drive.google.com (drive.google.com)... 74.125.142.138, 74.125.142.101, 74.125.142.100, ...
Connecting to drive.google.com (drive.google.com)|74.125.142.138|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘pix2pix/checkpoint/weights.zip’
2
shellgoogle-drive-apigoogle-colaboratory

2 Answers

1
votes

Try this

!gdown --id 19VsarMcYRNPLTDr6b6ABJyY8JUeBueL8

Then, you can create a new directory with !mkdir or move the weights.zip there.

1
votes

Based on @korakot's answer, the full working code to achieve the result in Colab is:

!gdown https://drive.google.com/uc?id=19VsarMcYRNPLTDr6b6ABJyY8JUeBueL8
!mkdir /content/Person_remover/pix2pix/checkpoint
import shutil
shutil.move("/content/Person_remover/weights.zip", "/content/Person_remover/pix2pix/checkpoint")