url_decode64 binary serialization to string

Question

I am currently trying to decode some base64 CDATA from Youtube Comments. It seems to decode it into a binary fine, but not sure how to convert it to a string.

<?xml version="1.0" encoding="utf-8"?><root><comments><![CDATA[EAEYACCQTg==]]></comments></root>

And in elixir

iex> Base.url_decode64!("EAEYACCQTg==")    
<<16, 1, 24, 0, 32, 144, 78>>

And if I try to toss it into a utf8 string, it doesn't match.

iex> <<x::utf8>> = Base.url_decode64!("EAEYACCQTg==")
** (MatchError) no match of right hand side value: <<16, 1, 24, 0, 32, 144, 78>>

Strings in Elixir are binaries, so there's no conversion necessary here. For example, <<65, 66, 67>> is exactly the same as "ABC", it's just a matter of Elixir pretty printing it for you. When a binary only contains valid codepoints it will be printed as string, otherwise it will be printed as a "raw" binary. The point is: the bytes you have decoded are not a valid string, for example strings cannot contain 0 bytes. Probably there's another level of encoding involved here. Do you happen to know what the result should exactly look like? — Patrick Oscity
I think that's my problem, its not really a string representation. I am trying to decode it from their site directly, and not through an API. Not sure of the actual encoding of the data is. — rockerBOO
Do you know what the expected output is? That would make it much easier to figure out the necessary steps for decoding. — Patrick Oscity
This representation is when no new comments are present. I was imagining it would be a JSON or maybe BSON representation? pastebin.com/Q7qUpM3f is a larger example showing multiple new comments. When I parse it in python, it comes out like: pastebin.com/9jhEtqpw — rockerBOO
Yeah looks like some binary serialization format. I hear Google uses Protobuf in some places, but I have no experience with that. — Patrick Oscity

amencarini amencarini · Accepted Answer · 2015-08-31T07:50:40

I think that the encoded value you're using is the problem:

iex(1)> Base.url_decode64!("EAEYACCQTg==") |> String.valid?
false
iex(2)> <<104, 101, 197, 130, 197, 130, 111>> |> String.valid?
true
iex(3)> IO.puts "The string is #{<<104, 101, 197, 130, 197, 130, 111>>}"
The string is hełło
:ok

If the decoded string was valid you'd get a string out of the box as a return value:

iex(4)> Base.encode64("foobar")
"Zm9vYmFy"
iex(5)> Base.url_decode64!("Zm9vYmFy")
"foobar"

url_decode64 binary serialization to string

2 Answers