Implement BloomFilter query pushdown optimization

[dai-chen]

Description

This PR introduces BloomFilter query pushdown optimization in OpenSearch by using a Painless script. Without dependency on OpenSearch SQL or another new mapper plugin, the following BloomFilter deserialization and membership check code are encoded in a Painless script https://github.com/dai-chen/opensearch-spark/blob/implement-bloom-filter-query-pushdown/flint-spark-integration/src/main/resources/bloom_filter_query.script.

  public static BloomFilter readFrom(InputStream in) {
      DataInputStream dis = new DataInputStream(in);
      int version = dis.readInt();
      int numHashFunctions = dis.readInt();
      BitArray bits = BitArray.readFrom(dis);
      ...
  }

 public boolean mightContainLong(long item) {
    int h1 = Murmur3_x86_32.hashLong(item, 0);
    int h2 = Murmur3_x86_32.hashLong(item, h1);

    long bitSize = bits.bitSize();
    for (int i = 1; i <= numHashFunctions; i++) {
      int combinedHash = h1 + (i * h2);
      // Flip all the bits if it's negative (guaranteed positive number)
      if (combinedHash < 0) {
        combinedHash = ~combinedHash;
      }
      if (!bits.get(combinedHash % bitSize)) {
        return false;
      }
    }
    return true;
  }

Because Painless language only support basic JDK API, API in Murmur3, DataInputStream and ByteArrayInputStream needs to be inlined accordingly. The Painless script length is 15.4k while OpenSearch default limit is 64k.

TODO

If any limitation or performance issue, we can either store the script by https://opensearch.org/docs/latest/api-reference/script-apis/create-stored-script/ or add dependency on SQL or a new plugin.

PR Planned

• Implement BloomFilter skipping index building logic #242
• Implement BloomFilter query rewrite (without pushdown optimization) #248
• Implement BloomFilter query pushdown optimization #271 [Current]
• Implement adaptive BloomFilter algorithm #251
• Support bloom filter type in Flint SQL

Issues Resolved

#206

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[vamsi-amazon]

We can probably minify this removing spaces and changing variable names. That reduces readability. Not sure if we can add a step in final build before bundling.

[vamsi-amazon]

Ignore if it is too much optimization and we are way below limits.

[dai-chen]

Good point. I recall in C there is tool compress/obfuscate the code. Painless is not that popular that I'm not sure if any tool rather than do this manually. Will take another look. Thanks!

[vamsi-amazon]

Since painless is mainly java, I just used this https://www.aspect-ratios.com/minify-java/

[dai-chen]

I'm thinking of something at runtime when loading or sending to OpenSearch. Otherwise like you pointed out, it will result in code hard to maintain? Will make a note and address it if any issue found later. Thanks!