[do not review] experimental support for lz4 compression (not working)

[andygrove]

Which issue does this PR close?

Closes #.

Rationale for this change

What changes are included in this PR?

How are these changes tested?

[Dandandan]

I am not really familiar with the code, but shouldn't StreamWriter and encoder be created only once per stream instead of per batch?

[andygrove]

Yes, that is something I have been thinking about as well. We have the cost of writing the schema for each batch currently, and the schema is guaranteed to be the same for each batch.

[andygrove]

To add some more context here, we buffer rows per partition until we reach the desired batch size and then need to serialize that batch to bytes that can be read as one block by CometBlockStoreShuffleReader.

[andygrove]

Here are my findings from hacking on this today.

LZ4 provides two compression formats: LZ4 Block Format and LZ4 Frame Format.

Spark uses the Java library https://github.com/lz4/lz4-java and specifically uses LZ4BlockOutputStream which seems to be a proprietary streaming LZ4 format, as noted in the documentation:

/**
 * Streaming LZ4 (not compatible with the LZ4 Frame format).
 * This class compresses data into fixed-size blocks of compressed data.
 * This class uses its own format and is not compatible with the LZ4 Frame format.
 * For interoperability with other LZ4 tools, use {@link LZ4FrameOutputStream},
 * which is compatible with the LZ4 Frame format. This class remains for backward compatibility.
 * @see LZ4BlockInputStream
 * @see LZ4FrameOutputStream
 */
public class LZ4BlockOutputStream extends FilterOutputStream {

edit: Apache Commons provides FramedLZ4CompressorInputStream and BlockLZ4CompressorInputStream so I am testing with using those from CometBlockStoreShuffleReader instead of using Spark's codec.