the AbstractLPCMFormat laundry list

[jrevels]

Moving these from some notes to here for visibility/archival purposes. It'd be fun to try out implementing AbstractLPCMFormat subtypes that utilize the following codecs/formats/tools, and see which ones are the most performant/ergonomic for a variety of generic synthesized LPCM data (obviously domain-specific codecs should always win out for specific modalities, which is why Onda allows this whole subsystem to be overloaded):

• Arrow (including w/ LZ4 column compression)
• Blosc (https://github.com/Blosc/c-blosc, https://github.com/Blosc/c-blosc2 ; see also https://www.blosc.org/posts/support-lossy-zfp/, https://www.blosc.org/posts/blosc2-ndim-intro/)
• zarr (https://github.com/zarr-developers/zarr-python)
• zfp (relevant ZFP Compression zarr-developers/numcodecs#117)
• lz4 frame format (https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md)
• MED format (successor to MEF) (https://medformat.org/)
• something custom 😈 could have an overloadable pipeline of variance reduction (DPCM) -> temporal decorrelation -> interchannel decorrelation -> entropy coding. maybe with the temporal/interchannel decorrelation stages as swappable. Lots of fun options to play with here; in its MOST fun incarnation, we could build a data-driven "compiler" for LPCM data compression codecs where input is sample data + domain knowledge encoded into each stage, and output is a relatively efficient codec synthesized to fit the input (EDIT: see https://www.blosc.org/pages/btune/)