Spark history logs decompress manually

Question

I have some problems with spark history server and therefore tried to uncompress lz4 logs manually. I installed liblz4-tool and downloaded application log application_1510049252849_0303.lz4. But when uncompressing it with lz4 ~/application_1510049252849_0303.lz4 I got the following error:

Error 44 : Unrecognized header : file cannot be decoded 

And this I have for every spark history lz4 files. What is the solution? Is there a way to open it manually?

Answer 1

I had the same error yesterday while trying to decompress the lz4 event logs. Apparently there seems to be some incompatibilities between hadoop lz4 codec and other lz4 codecs: see https://issues.apache.org/jira/browse/HADOOP-12990.

For what it's worth, spark history server can read compressed event logs in lz4, so you don't need to decompress them manually. Hope this helps.

Answer 2

lz4 no need to decompress while placing into spark history server. Just you can place your event logs into event log directory. It will automatically pick that application and show it in the Spark History Server UI.