[Feature] In memory LZ4Frame decompression

[vpenades]

Could we have an API to encode/decode frame blocks directly to arrays/spans ?

something like this:

Byte[] lz4bytes = { ... };

int flattenedSize = LZ4Stream.GetFlattenedSize(lz4bytes);

bytes[] flattenedBytes = new Byte[flattenedSize];
LZ4Stream.Decode(lz4bytes,flattenedBytes);

[MiloszKrajewski]

Isn't LZ4Pickler.Pickle and LZ4Pickler.Unpickle do (almost) exactly that?

[vpenades]

@MiloszKrajewski do Pickle and Unpickle handle Frames?

What I am trying to do is to read and play ROS-Bag files... they're essentially video files, where every frame can be compressed using LZ4 or BZ2... I usually need to decompress between 60 and 90 images per second, so I'm trying to avoid any unnecesary byte copy. I am also using ArrayPool to reuse buffers.

[MiloszKrajewski]

That's fair. From what I've seen every frame is ROS bag file is a mini LZ4Stream. I guess all the magic is in https://github.com/ros/ros_comm/blob/melodic-devel/utilities/roslz4/src/lz4s.c but it also seems to me like customized lz4 implementation so I'm not exactly sure if it would be easy to implement.

I understand LZ4Stream.Decode(new MemoryStream(lz4bytes)).ReadToEnd() is what you need but with less memory allocations?

[vpenades]

I understand LZ4Stream.Decode(new MemoryStream(lz4bytes)).ReadToEnd() is what you need but with less memory allocations?

Exactly. Right now it's what I am using, and it's decoding the LZ4 miniblocks just fine.

But in order to improve performance, any memory reallocation is a bottleneck. I am using Spans, ArraySegments and ArrayPools everywhere...

I also would like to try with realtime recording/encoding; right now I am recording raw frames at 500mb/second throughput, and it's already at the limit. So my plan is that if realtime encoding is not fast enough, I would do it in a post process pass.

[MiloszKrajewski]

Currently I don't provide ContentLength as Stream.Length but this might be relatively easy thing to do (not trivial though, as it is expected to be available right after construction, so reading of inner file would need to happen before reading lz4 stream, kind of non-obvious side effect).
Once you have frame length you could use https://github.com/Microsoft/Microsoft.IO.RecyclableMemoryStream.

I know it is still not ideal, but much more plausible in short term.

[MiloszKrajewski]

In version 1.0.2 you can now read content length (if compressed with content length).

[MiloszKrajewski]

1.3.0-beta has a lot of new streaming methods and some of them are very memory friendly, let's say.