Best practices for passing encrypted data between different programming languages

Question

I have read that if you want to encrypt a string using one programming language and decrypt that string using another programming language, then to ensure compatibility it is best to do some conversions prior to doing the encryption. I have read that it's a best practice to encrypt the byte array of a string rather than the string itself. Also, I have read that certain encryption algorithms expect each encrypted packet to be a fixed length in size. If the last packet to be encrypted isn't the required size, then encryption would fail. Therefore it seems like a good idea to encrypt data that has first been converted into a fixed length, such as hex.

I am trying to identify best practices that are generally useful regardless of the encryption algorithm being used. To maximize compatibility when encrypting and decrypting data across different languages and platforms, I would like a critique on the following steps as a process:

Encryption:

• start with a plain text string
• convert plain text string to byte array
• convert byte array to hex
• encrypt hex to encrypted string
• end with an encrypted string

Decryption:

• start with an encrypted string
• decrypt encrypted string to hex
• convert hex to byte array
• convert byte array to plain text string
• end with a plain text string

Answer 1

Really the best practice for encryption is to use a high level encryption framework, there's a lot of things you can do wrong working with the primitives. And mfanto does a good a good job of mentioning important things you need to know if you don't use a high level encryption framework. And i'm guessing that if you are trying to maximize compatibility across programming languages, it's because you need other developers to inter-operate with the encryption, and then they need to learn the low level details of working with encryption too.

So my suggestion for high level framework is to use the Google Keyczar framework, as it handles the details of, algorithm, key management, padding, iv, authentication tag, wire format all for you. And it exists for many different programming Java, Python, C++, C# and Go. Check it out.

I wrote the C# version, so I can tell you the primitives it uses behind the scenes are widely available in most other programming languages too, and it uses standards like json for key management and storage.

Answer 2

Your premise is correct, but in some ways it's a little easier than that. Modern crypto algorithms are meant to be language agnostic, and provided you have identical inputs with identical keys, you should get identical results.

It's true that for most ciphers and some modes, data needs to be a fixed length. Converting to hex won't do it, because the data needs to end on fixed boundaries. With AES for example, if you want to encrypt 4 bytes, you'll need to pad it out to 16 bytes, which a hex representation wouldn't do. Fortunately that'll most likely happen within the crypto API you end up using, with one of the standard padding schemes. Since you didn't tag a language, here's a list of padding modes that the AesManaged class in .NET supports.

On the flip side, encrypting data properly requires a lot more than just byte encoding. You need to choose the correct mode of operation (CBC or CTR is preferred), and then provide some type of message integrity. Encryption alone doesn't protect against tampering with data. If you want to simplify things a bit, then look at a mode like GCM, which handles both confidentiality, and integrity.

Your scheme should then look something like:

• Convert plain text to string to byte array. See @rossum's comment for an important note about character encoding.
• Generate a random symmetric key or use PBKDF2 to convert a passphrase to a key
• Generate a random IV/nonce for use with GCM
• Encrypt the byte array and store it, along with the Authentication Tag
• You might optionally want to store the byte array as a Base64 string.

For decryption:

• If you stored the byte array as a Base64 string, convert back to the byte array.
• Decrypt encrypted byte array to plaintext
• Verify the resulting Authentication Tag matches the stored Authentication Tag
• Convert byte array to plain text string.

Answer 3

I use HashIds for this purpose. It's simple and supports wide range of programming language. We use it to pass encrypted data between our PHP, Node.js, and Golang microservices whenever we need to decrypt data in the destination.

Answer 4

I have read that it's a best practice to encrypt the byte array of a string rather than the string itself.

Crytographic algorithms generally work on byte arrays or byte stream, so yes. You don't encrypt objects (strings) directly, you encrypt their byte representations.

Also, I have read that certain encryption algorithms expect each encrypted packet to be a fixed length in size. If the last packet to be encrypted isn't the required size, then encryption would fail.

This is an implementation detail of the particular encryption algorithm you choose. It really depends on what the API interface is to the algorithm.

Generally speaking, yes, crytographic algorithms will break input into fixed-size blocks. If the last block isn't full then they may pad the end with arbitrary bytes to get a full chunk. To distinguish between padded data and data which just happens to have what-look-like-padding bytes at the end, they'll prepend or append the length of the plain text to the byte stream.

This is the kind of detail that should not be left up to the user, and a good encryption library will take care of these details for you. Ideally you just want to feed in your plain text bytes and get encrypted bytes out on the other side.

Therefore it seems like a good idea to encrypt data that has first been converted into a fixed length, such as hex.

Converting bytes to hex doesn't make it fixed length. It doubles the size, but that's not fixed. It makes it ASCII-safe so it can be embedded into text files and e-mails easily, but that's not relevant here. (And Base64 is a better binary→ASCII encoding than hex anyways.)

In the interest of identifying best practices for ensuring compatibility with encrypting and decrypting data across different languages and platforms, I would like a critique on the following steps as a process:

Encryption:

• plain text string
• convert plain text string to byte array
• convert byte array to hex
• encrypt hex to encrypted string
• encrypted string
• plain text byte array to encrypted byte array

Decryption:

• encrypted string
• decrypt encrypted string to hex
• convert hex to byte array
• encrypted byte array
• decrypt encrypted byte array to plain text byte array
• convert byte array to plain text string
• plain text string

To encrypt, convert the plain text string into its byte representation and then encrypt these bytes. The result will be an encrypted byte array.

Transfer the byte array to the other program in the manner of your choosing.

To decrypt, decrypt the encrypted byte array into a plain text byte array. Construct your string from this byte array. Done.