Tracking issue for introducing inverted index

[zhongzc]

What type of enhancement is this?

Performance

What does the enhancement do?

In the current system setup, in the Mito Engine, the first column of Primary Keys has a Min-Max index, which significantly optimizes the outcome. However, there are limitations when it comes to other columns, primarily tags. This RFC suggests the implementation of an inverted index to provide enhanced filtering benefits to bridge these limitations and improve overall system performance.

This issue is used to track the relevant implementation tasks of RFC:

• Read and write files in Puffin format
  • feat: introduce metadata struct of puffin #2725
  • feat(puffin): add partial reader #2741
  • feat(puffin): add file reader #2751
  • feat(puffin): add file writer #2776
• Read and write Blob in greptime-inverted-index-v1 format within Puffin file
  • feat(inverted_index): add index reader #2803
  • feat(inverted_index.format): add writer #2900
• Implementation of index searcher
  • feat(inverted_index.search): add fst applier #2851
  • feat(inverted_index.search): add fst values mapper #2862
  • feat(inverted_index.search): add index applier #2868
• Implementation of index creator
  • feat(inverted_index.create): add read/write for external intermediate files #2942
  • feat(inverted_index.create): add external sorter #2950
  • feat(inverted_index.create): add index creator #2960
• Integration with the existing Parquet SST reader
  • feat(inverted_index): Add applier builder to convert Expr to Predicates (Part 1) #3034
  • feat(inverted_index): Add applier builder to convert Expr to Predicates (Part 2) #3068
  • feat(inverted_index): implement apply for SstIndexApplier #3088
  • feat(parquet): introduce inverted index applier to reader #3130
• Integration with the existing Parquet SST writer
  • feat(inverted_index): introduce SstIndexCreator #3107
  • feat(mito): support write cache for index file #3144
  • feat(mito): enable inverted index #3158
• Add variables to control apply on query

Implementation challenges

No response

[tisonkun]

@zhongzc Is there any further remaining task here?

[killme2008]

I think we can close this issue. cc @zhongzc