How to encrypt large file using Python?

Question

I'm trying to encrypt file that is larger than 1GB. I don't want to read it all to memory. I chose Fernet (cryptography.fernet) for this task, because it was most recommended (faster than asymetric solutions).

I generated the key. Then I've created a script to encrypt:

    key = Fernet(read_key())

    with open(source, "rb") as src, open(destination, "wb") as dest:
        for chunk in iter(lambda: src.read(4096), b""):
            encrypted = key.encrypt(chunk)
            dest.write(encrypted)

and for decryption:

    key = Fernet(read_key())

    with open(source, "rb") as src, open(destination, "wb") as dest:
        for chunk in iter(lambda: src.read(4096), b""):
            decrypted = key.decrypt(chunk)
            dest.write(decrypted)

Encryption works - no surprise, but decryption is not. Firstly I thought that it might work, but it's not. I guess chunk size increases when encrypted, and then when I'm reading 4096 bytes, it's not a whole encrypted chunk. I've got an error trying to decrypt:

Traceback (most recent call last):
  File "/redacted/path/venv/lib/python3.7/site-packages/cryptography/fernet.py", line 119, in _verify_signature
    h.verify(data[-32:])
  File "/redacted/path/venv/lib/python3.7/site-packages/cryptography/hazmat/primitives/hmac.py", line 74, in verify
    ctx.verify(signature)
  File "/redacted/path/venv/lib/python3.7/site-packages/cryptography/hazmat/backends/openssl/hmac.py", line 75, in verify
    raise InvalidSignature("Signature did not match digest.")
cryptography.exceptions.InvalidSignature: Signature did not match digest.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/redacted/path/main.py", line 63, in <module>
    decrypted = key.decrypt(chunk)
  File "/redacted/path/venv/lib/python3.7/site-packages/cryptography/fernet.py", line 80, in decrypt
    return self._decrypt_data(data, timestamp, time_info)
  File "/redacted/path/venv/lib/python3.7/site-packages/cryptography/fernet.py", line 137, in _decrypt_data
    self._verify_signature(data)
  File "/redacted/path/venv/lib/python3.7/site-packages/cryptography/fernet.py", line 121, in _verify_signature
    raise InvalidToken
cryptography.fernet.InvalidToken

Is there's a way to solve this? Maybe there's a better (simpler) approach with different solution than fernet?

Does this answer your question? Cryptography token object raises exception and cannot decrypt even though the tokens are identical — buran
@buran No. I generated a key once and store it in file from which I read using read_key(). I saw that question earlier, when I was looking for an answer. I couldn't find anything helpful. — Mike

Ebbe M. Pedersen Ebbe M. Pedersen · Accepted Answer · 2021-09-24T10:34:25

Fernet is not supposed to be used in a streaming fasion. They explain that in the documentation:

From the documentation (last section):

Limitations

Fernet is ideal for encrypting data that easily fits in memory. As a design feature it does not expose unauthenticated bytes. This means that the complete message contents must be available in memory, making Fernet generally unsuitable for very large files at this time.

How to encrypt large file using Python?

1 Answers