bash base64 producing inconsistent output?

Question

Can anyone explain this?

[vagrant@centos ~]$ echo "10IXydrdsc4DVAgxzrXldNw5GMeVAHKG:TAO04JuWz4PBVWYm" | base64
MTBJWHlkcmRzYzREVkFneHpyWGxkTnc1R01lVkFIS0c6VEFPMDRKdVd6NFBCVldZbQo=
[vagrant@centos ~]$ echo "MTBJWHlkcmRzYzREVkFneHpyWGxkTnc1R01lVkFIS0c6VEFPMDRKdVd6NFBCVldZbQ==" | base64 -d
10IXydrdsc4DVAgxzrXldNw5GMeVAHKG:TAO04JuWz4PBVWYm

The first string encodes with o= at the end, but the encoded string with == at the end instead, decodes to the same original string...

GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)

Answer 1

Compare these

echo "10IXydrdsc4DVAgxzrXldNw5GMeVAHKG:TAO04JuWz4PBVWYm" | base64 | od -c
echo "MTBJWHlkcmRzYzREVkFneHpyWGxkTnc1R01lVkFIS0c6VEFPMDRKdVd6NFBCVldZbQ==" | base64 -D | od -c
echo "MTBJWHlkcmRzYzREVkFneHpyWGxkTnc1R01lVkFIS0c6VEFPMDRKdVd6NFBCVldZbQo=" | base64 -D | od -c

If we don't send the newline when using echo the o is missing, have a look at this...

echo -n "10IXydrdsc4DVAgxzrXldNw5GMeVAHKG:TAO04JuWz4PBVWYm" | base64

It's the newline that's being encoded that gives the o in o=

The = is padding and it might not always be there. Have a look here..

https://en.wikipedia.org/wiki/Base64#Padding

Different implementations may also use different padding characters. You can see some of the differences here

https://en.wikipedia.org/wiki/Base64#Variants_summary_table

From the RFC

3.2. Padding of Encoded Data

In some circumstances, the use of padding ("=") in base-encoded data is not required or used. In the general case, when assumptions about the size of transported data cannot be made, padding is required to yield correct decoded data.

Implementations MUST include appropriate pad characters at the end of encoded data unless the specification referring to this document explicitly states otherwise.

The base64 and base32 alphabets use padding, as described below in sections 4 and 6, but the base16 alphabet does not need it; see
section 8.

Answer 2

When you use $echo, a newline is appended to the end of the output. This newline character is part of the base64 encoding. When you change the 'o' to a '=', you're changing the encoding of the newline character. In this case, the character it decodes to is still not a printable character.

In my terminal, decoding the two string yields the same output, but the string ending in "o=" has a newline, and the string ending in "==" does not.

$> echo "MTBJWHlkcmRzYzREVkFneHpyWGxkTnc1R01lVkFIS0c6VEFPMDRKdVd6NFBCVldZbQo=" | base64 -d
10IXydrdsc4DVAgxzrXldNw5GMeVAHKG:TAO04JuWz4PBVWYm
$> echo "MTBJWHlkcmRzYzREVkFneHpyWGxkTnc1R01lVkFIS0c6VEFPMDRKdVd6NFBCVldZbQ==" | base64 -d
10IXydrdsc4DVAgxzrXldNw5GMeVAHKG:TAO04JuWz4PBVWYm $>

Using $echo -n would allow you to pass the string into base64 without the trailing newline. The string without the newline encodes to the string ending in "==".

Answer 3

PHP also encodes it properly, which leads me to believe there is some issue with the base64 program in bash, as I haven't found any mention of 'o' somehow being used as a padding character.

php > echo base64_encode("10IXydrdsc4DVAgxzrXldNw5GMeVAHKG:TAO04JuWz4PBVWYm"); MTBJWHlkcmRzYzREVkFneHpyWGxkTnc1R01lVkFIS0c6VEFPMDRKdVd6NFBCVldZbQ==