Add Kafka 3.1.0 to CI matrix, migrate to bitnami kafka image

[bai]

Migrate from cp-kafka proprietary kafka docker image to bitnami one.

As you could see in the diff, we've had quite some hackery around handling the fact that Confluent Platform versioning scheme is very off with Kafka one.
Not only that, Confluent Platform lags behind quite noticeably — we add workarounds to pretend we're really testing against the latest Kafka version, while we're not.

To mitigate these annoyances, in this PR I'm migrating to a more actively maintained Bitnami Kafka docker image that seems to be pretty up to date with mainstream Kafka releases. Thanks Bitnami folks!

[caution-tape-bot]

We noticed that this PR either modifies or introduces code that pipes the output of curl or wget into a shell. This is dangerous because it grants full control over the machine that downloads and executes the script, with no way to control or monitor what is executed.
Code should be fetched in a way where its authenticity can be verified, for instance through a package manager, or by verifying a cryptographic hash or signature.
If you are unsure how to do this, please check the developer handbook or ask in #help-infrasec and we'll be happy to take a look. cc: @Shopify/infrasec

[dnwe]

@bai I’m +1 for moving away from proprietary images so we can get closer to main releases of kafka. However, should we just build our own images rather than rely on bitnami ones?

[bai]

I’m +1 for moving away from proprietary images so we can get closer to main releases of kafka. However, should we just build our own images rather than rely on bitnami ones?

I'll take a look at that; want to see if bitnami even works and improves the compatibility. To be honest I'm a bit "lazy" to maintain our own Kafka docker images since IIRC it's not trivial to build them - albeit we have to do it once.

[bai]

Oh wow, didn't think it's going to work just like that. @dnwe could you please take a look at this PR once again? 🙏🏼

(Obviously I'll squash and rename commits, but wanted to see if it looks good to you.)

[bai]

I don't expect Zookeeper version requirement to change as often as Kafka versions change, so oped to hardcode Zookeeper version instead of adding an extra configuration parameter.

We might explore moving to non-proprietary zookeeper docker image in future.

[dnwe]

@bai good work — I wonder if it's worth keeping 2.8.1 around in the CI matrix for a bit longer as well? Whilst I know our official support statement is N-1, I think as 3.1.0 has only just released we probably want a bit of coverage on the 2.x series
Unless you think 6 (3 x 1.16, 3 x 1.17) runs of the test suite is too much?

Aside: I wonder if we should split up the matrix so that we run the non-functional unittests with Go 1.16.x and 1.17.x and then run the functional tests against the Kafka versions matrix with just 1.17.x

[bai]

@dnwe — thanks for feedback. I've updated the build matrix to reflect it, I think it's a great suggestion. Let me know what you think.

[dnwe]

Looks great!