[Bug] Docker plugin latest images should update with new dbt-core versions

[alexrosenfeld10]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

dbt core 1.0.2 has been released as well as its docker image, ghcr.io/dbt-labs/dbt-core:1.0.2. However, the latest version of the plugin specific docker images don't use the latest dbt core version (well, at least snowflake - I didn't test the others, but i'd imagine it's the same).

Expected Behavior

The "latest" plugin docker images should use the latest version of dbt-core.

Steps To Reproduce

Pretty simple:

#FROM ghcr.io/dbt-labs/dbt-snowflake:latest
RUN dbt --version

Relevant log output

Step 1/2 : FROM ghcr.io/dbt-labs/dbt-snowflake:latest
 ---> 07691378d6fd
Step 2/2 : RUN dbt --version
 ---> Running in cf649bd6b24f
installed version: 1.0.1
   latest version: 1.0.2

Your version of dbt is out of date! You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation

Plugins:
  - snowflake: 1.0.0

### Environment

```markdown
- OS: n/a
- Python: n/a
- dbt: n/a

What database are you using dbt with?

No response

Additional Context

https://getdbt.slack.com/archives/CBSQTAPLG/p1645383426339879

[alexrosenfeld10]

I realize the obvious workaround here is just RUN pip install dbt-snowflake. I do think there should be some process that triggers these releases all together though, which is more why I'm opening the issue.

[alexrosenfeld10]

I understand from this slack message that this behavior is more or less intentional, but I'm still curious for a bit more context as to why 😄

[jtcohen6]

Thanks for opening @alexrosenfeld10. The current intention is definitely to rerelease the latest + <Major>.<Minor>.latest tags of all plugins that depend on dbt-core, when there's a new version of dbt-core available. We'll be following up on this issue.

[alexrosenfeld10]

@jtcohen6 I may be misunderstanding this, but this still seems not ideal. Take the following scenario:

1. I set my dockerfile to FROM ghcr.io/dbt-labs/dbt-snowflake:1.0.latest. Currently this means I'll use dbt-core version 1.0.2
2. dbt-core version 1.0.3 comes out, and has an unfortunate and impactful bug that got released!
3. My CI job runs and pulls latest, which now means I am impacted by the bug without knowing it or upgrading to it intentionally.

I'm sure the engineering team has all sorts of automated testing and hopes to not release bugs in patch versions, but inevitably bugs slip in to any software. Might the team consider a different approach to mitigate this issue? Or do you think I'm overly concerned?

(again I know I can just install the specific core version and pip install the plugin. More curious about the intention and design choice here, so thanks in advance for entertaining my questions!)

[jtcohen6]

@alexrosenfeld10 I think your concern is well placed and reasonable! You always have the right to install the exact pinned versions of dbt-core + plugins that you'd like, and still use our official images as your jumping-off point.

I'll just say that our team is tremendously committed to limiting the scope of fixes in patch releases after v1.0, and avoiding net-new bugs in patches. When they happen (it's software, it happens), we will drop everything to release a new working patch ASAP. This just happened earlier today: a community member identified a highly specific bug (#4752, net-new in v1.0.2); we estimated that it would affect ~0.5% of projects; we've already merged the fix, and we're planning to push v1.0.3 live very soon. We also used this to prompt internal conversation and reflection on how we would patch the fix differently next time. This is added motivation for our current initiative, overhauling our test suite to better represent dbt-core behavior "in the wild," and layering in extra QA practices to close the gaps that inevitably remain.

That's the commitment we're making to you, as a potential user of the 1.0.latest Docker image tag—and it's the same commitment we're making to all the folks using dbt Cloud, who are also (starting in v1.0) automatically updated to use the latest stable patch release for each minor version. Cautious, intentional patching is a crucial capability for us to deliver high-quality software that runs in production, reliably, for thousands of projects. It's not a commitment we take lightly.

To that end, v1.0.2 fixed a separate dependency error (#4745) that affected 100% of projects. Folks installing 1.0.latest got that fix; no further action needed.

That's the rationale underlying latest and <Major>.<Minor>.latest tags. I also appreciate the desire for totally locked-down dependencies, which is what's offered by the specific version image (dbt-snowflake:1.0.0). In our current scheme, we won't be updating those images to account for a new version of dbt-core, or of any other dependency.

So, those are the options available to you:

• Specific version images, with totally locked dependencies
• latest-tagged images, which ensure you're getting the latest patch fixes
• Either + your own custom pip install logic

Does that delineation make sense? (@iknox-fa Does it makes sense to you?)

[alexrosenfeld10]

@jtcohen6 that all makes sense to me, yes. I think, given the commitments (and the fact that dbt Cloud takes patch updates OOTB) I'll probably just target major.minor.latest (well, when this issue is closed and it's actually pulling latest that is 😉).

Thanks 👍

[iknox-fa]

@alexrosenfeld10 We should be all set now. Give it a try and feel free to reach out if it gives you any more trouble.

[alexrosenfeld10]

@iknox-fa seems good to me:

Step 1/5 : FROM ghcr.io/dbt-labs/dbt-snowflake:1.0.latest
1.0.latest: Pulling from dbt-labs/dbt-snowflake
a2abf6c4d29d: Already exists 
27003db43ed4: Already exists 
eba080ca5a70: Already exists 
00ceeca3d803: Already exists 
8980b900ecd2: Already exists 
f03d51f7c312: Pull complete 
7eee275ec764: Pull complete 
fdb8914e8ad8: Pull complete 
96d54054c81e: Pull complete 
Digest: sha256:8da5a372715eb57afba20345a888411ca1885e359660050472488061f849459d
Status: Downloaded newer image for ghcr.io/dbt-labs/dbt-snowflake:1.0.latest
 ---> b6a0eb30321b
Step 2/5 : RUN dbt --version
 ---> Running in fc91a6a3a022
installed version: 1.0.3
   latest version: 1.0.3

Up to date!

Plugins:
  - snowflake: 1.0.0 - Up to date!

[alexrosenfeld10]

@iknox-fa while you're in the docker neighborhood, was wondering if you had thoughts about this: #4784 and / or if it's related to potential bugs in the recent base images? (I realize it's a bit separate, so I'll close this issue, I hope we can continue the docker discussions over there as I'm pretty stuck on this)