Rethink our CI to avoid building all the docker images every time

[kun-lu20]

Created from #2948 (comment)

The current situation is that our CI will build all the docker images every time. Since using docker buildx to build multi-arch images will be slower than regular build, we need to find an effective way to reduce the time penalty.

@jpkrohling suggests we should avoid building all the things all the time if we don't need them for the tests. We can then have a nightly build to publish the snapshot images.

[yurishkuro]

It would be great if GitHub supported stages of CI, so that we don't run every single thing under the sun before the basic checks like linters & unit tests are successful.

[jpkrohling]

I think it does support dependencies between jobs. We could then have smoke-tests running before, and full tests running after that. I still think that we don't need the full suite for every single PR. A nightly build, with the option of being manually triggered just before a release, would be sufficient, in my opinion.