Don't store blocks in database

[erikgrinaker]

The block store currently stores blocks in a database, typically LevelDB. This is suboptimal, since these databases usually aren't optimized for storage of BLOBs, operations like LSM-tree compaction becomes prohibitively expensive, and contents can't be streamed. Blocks should instead be stored in the filesystem, with metadata and a checksum in the LevelDB database.

When fixing this, we should fix #4567 as well.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[melekes]

@erikgrinaker do u remember why this wasn't accepted? I would like to tackle this if possible.

[erikgrinaker]

It's accepted, just a bit tricker than I first though. Tendermint primarily uses block parts, not whole blocks - I think it's too expensive to store block parts as separate files (e.g. due to inode usage and half-full filesystem blocks), so we'll need to maintain an index for each block's part offsets.

[tac0turtle]

I think it's too expensive to store block parts as separate files (e.g. due to inode usage and half-full filesystem blocks)

We could look into removing block parts from various parts of the codebase as it pertains to consensus primarily. According to this issue tendermint/spec#78 it is "an unfortunate artifact". It may make sense to tackle the removal in places instead of putting a bandaid on it, although this will require a greater refactor.

[melekes]

We'll still have to store erasure-coded chunks somewhere, so I wouldn't say it's a bandaid

[erikgrinaker]

So the reason why I abandoned this last time was because if we end up needing to maintain a bunch of indexes, randomly access block parts, and storing temporary data and stuff, then I'm not sure that storing blocks as files makes much sense anymore. A better approach might be to use e.g. a B-tree based store (like BoltDB) which doesn't need to do compaction all the time (which I think is what kills LevelDB performance for blocks).

Would be interesting to see some performance figures, and maybe do a proof of concept implementation to get a sense of the complexity of doing all of this stuff ourselves.