Python rclone check difference of two folders

Question

I'm trying to code an automatic script for uploading to the gdrive with rclone. I will not go through all the code only in this check statement, the rclone command checks files from the local folder and mounted folder something like this: rclone check "local folder" "mounted folder" --ignore existing --onlyoneway it returns in terminal some data that can't be stored in a text file or I don't now how.

def upload_check():
    print(" check if all files are uploaded ")
    global Error_upload
    if :#I stuck here, rclone check and return true or false if all files are uploaded by name and size
        Error_upload = True
        return Error_upload
        print("Not uploaded ")#---------------------------
    else:# all good
        Error_upload = False
        return Error_upload
        print("all files are online")#---------------------------

my question is how to properly check two directories if they are identical by all files inside and files size and returning Boolean True or False?

you probably need to use one of the Python library for rclone like python-rclone. In general Python provides a large ecosystem for a lot of functionality thourgh the PyPi package server. — tuned

Diego Martinčić Diego Martinčić · Accepted Answer · 2020-10-14T13:47:53

After a few days I come up with this complicated solution:

import shutil
import os
local = "Local/"
destination = "uploaded/"
checkfile = "logfile.txt"
def upload_check():
    print(" check if all files are uploaded ")
    global Error_upload 
    os.system("rclone check 'Local' 'gdrive' --one-way  -vv -P --combined logfile.txt")
    destination = "uploaded/"
    checkfile = "logfile.txt"
    search = "=" # move from the folder successfuly uplouded files

    list_of_files = []
    lines = []
    folders = []
    uniq_folder_list = []
    shutil_l = []
    shutil_f = []
    
    for line in open(checkfile, "r"):
        if search in line:
            list_of_files = line.split("/")[1]
            lines.append(list_of_files.rstrip())
            list_of_folders = line.split(" ")[1].split("/")[0]
            folders.append(list_of_folders.rstrip())
    [uniq_folder_list.append(n) for n in folders if n not in uniq_folder_list] 
    for new_folder in uniq_folder_list:
        if not os.path.exists(destination + new_folder):
            os.makedirs(destination + new_folder)
    for l, f in zip(lines, folders):
        l1 = (local + f + "/" + l)
        f1 = (destination + f)
        shutil_l.append(l1.rstrip())
        shutil_f.append(f1.rstrip())
    for src, dest in zip(shutil_l, shutil_f):
        shutil.move(src,dest)

    os.system("rclone check 'Local' 'gdrive' --one-way  -vv -P --combined logfile.txt")
    with open(checkfile, 'r') as read_obj:
        one_char = read_obj.read(1)
        if not one_char:
            Error_upload = False
            return Error_upload
            print("all files are online")
        else:
            Error_upload = True
            return Error_upload
            print("Not uploaded ")

First I created some files and a couple of them uploaded them to the drive, also one corrupted file. Than this scrip do the job. The file logfile.txt contains a list generated with rclone

rclone check 'Local' 'gdrive' --one-way -vv -P --combined logfile.txt

this bash command will generate a logfile:

+ 20_10_10/IMG_1301-00006.jpg
+ 20_10_10/IMG_1640-00007.jpg
+ 20_10_10/IMG_1640-00008.jpg
+ 20_10_10/IMG_1640-00009.jpg
+ 20_10_10/IMG_1640-00010.jpg
+ 20_10_10/IMG_1640-00011.jpg #missing on remote
* 20_10_10/IMG_1301-00004.jpg #corrupted file
= 20_10_10/IMG_1301-00005.jpg
= 20_10_10/IMG_1301-00003.jpg
= 20_10_10/IMG_1301-00001.jpg
= 20_10_09/IMG_2145-00028.jpg
= 20_10_10/IMG_1301-00002.jpg

more info on rclone check help on rclone. The files with "=" are identical on local and remote destination, so we want to move them from the source folder to an uploaded folder.

The script runs again and if the read function can't read anything, all files are online and the upload function does not need to run again. But since there are un uploaded files and a corrupted file (it can happened if the connection is lost while uploading) the script will run the upload function or what ever other function triggered by if function with variable "Error_upload"

just for reference:

if Error_upload == True:
   print("All files are on the cloud")
else:
   upload() #your upload function
   upload_check()

I certainly know that this code could be simpler and improved.

Python rclone check difference of two folders

1 Answers