0
votes

I'm working a program to upload (large) files to a remote SFTP-server, while also calculating the file's SHA256. The uploads are slow, and the program is supposed to open multiple SFTP-connections.

Here is the main code:

async def bwrite(fd, buf):
    log.debug('Writing %d bytes to %s', len(buf), fd)
    fd.write(buf)

async def digester(digest, buf):
    log.debug('Updating digest %s with %d more bytes', digest, len(buf))
    digest.update(buf)

async def upload(fNames, SFTP, rename):

    for fName in fNames:
        inp = open(fName, "rb", 655360)
        log.info('Opened local %s', fName)
        digest = hashlib.sha256()
        rName = rename % os.path.splitext(fName)[0]
        out = SFTP.open(rName, "w", 655360)
...
        while True:
            buf = inp.read(bsize)
            if not buf:
                break
            await bwrite(out, buf)
            await digester(digest, buf)

        inp.close()
        out.close()
...

for i in range(0, len(clients)):
    fNames = args[(i * chunk):((i + 1) * chunk)]
    log.debug('Connection %s: %d files: %s',
        clients[i], len(fNames), fNames)
    uploads.append(upload(fNames, clients[i], Rename))

log.info('%d uploads initiated, awaiting completion', len(uploads))
results = asyncio.gather(*uploads)
loop = asyncio.get_event_loop()
loop.run_until_complete(results)
loop.close()

The idea is for multiple upload coroutines to run "in parallel" -- each using its own separate SFTP-connection -- pushing out one or more files to the server.

It even works -- but only a single upload is running at any time. I expected multiple ones to get control -- while their siblings awaits the bwrite and/or the digester. What am I doing wrong?

(Using Python-3.6 on FreeBSD-11, if that matters... The program is supposed to run on RHEL6 eventually...)