-
-
Notifications
You must be signed in to change notification settings - Fork 478
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: support for custom registry prefixes at the configuration level #1928
Merged
mdelapenya
merged 10 commits into
testcontainers:main
from
mdelapenya:image-substitutors-round-two
Nov 28, 2023
Merged
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
ce0d4cf
feat: support for setting the reaper image at the properties level
mdelapenya 71a97fc
chore: simplify reaper creation using the config
mdelapenya d6b8159
docs: document Ryuk properties/env vars
mdelapenya 39cb31a
chore: propose a Hub prefix instead
mdelapenya bda7f05
feat: always apply the prefix substitution to all images
mdelapenya d23feaa
chore: exclude certain conditions for the Hub prefix substitution
mdelapenya 4bf1207
docs: include docs about image substitutors and hub prefix
mdelapenya 17d6795
chore: do not break users of the constant
mdelapenya 6fbdbe2
docs: update link
mdelapenya 2dc912e
docs: remove blanks
mdelapenya File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,10 +1,95 @@ | ||
In more locked down / secured environments, it can be problematic to pull images from Docker Hub and run them without additional precautions. | ||
# Image name substitution | ||
|
||
An image name substitutor converts a Docker image name, as may be specified in code, to an alternative name. This is intended to provide a way to override image names, for example to enforce pulling of images from a private registry. | ||
_Testcontainers for Go_ supports automatic substitution of Docker image names. | ||
|
||
_Testcontainers for Go_ exposes an interface to perform this operations: `ImageSubstitutor`, and a No-operation implementation to be used as reference for custom implementations: | ||
This allows the replacement of an image name specified in test code with an alternative name - for example, to replace the | ||
name of a Docker Hub image dependency with an alternative hosted on a private image registry. | ||
|
||
This is advisable to avoid Docker Hub rate limiting, and some companies will prefer this for policy reasons. | ||
|
||
!!!info | ||
As of November 2020 Docker Hub pulls are rate limited. As Testcontainers uses Docker Hub for standard images, some users may hit these rate limits and should mitigate accordingly. Suggested mitigations are noted in [this issue in Testcontainers for Java](https://github.com/testcontainers/testcontainers-java/issues/3099) at present. | ||
|
||
This page describes two approaches for image name substitution: | ||
|
||
* [Automatically modifying Docker Hub image names](#automatically-modifying-docker-hub-image-names), prefixing them with a private registry URL. | ||
* [Using an Image Name Substitutor](#developing-a-custom-function-for-transforming-image-names-on-the-fly), developing a custom function for transforming image names on the fly. | ||
|
||
!!!warning | ||
It is assumed that you have already set up a private registry hosting [all the Docker images your build requires](../supported_docker_environment/image_registry_rate_limiting.md#which-images-are-used-by-testcontainers). | ||
|
||
## Automatically modifying Docker Hub image names | ||
|
||
_Testcontainers for Go_ can be configured to modify Docker Hub image names on the fly to apply a prefix string. | ||
|
||
Consider this if: | ||
|
||
* Developers and CI machines need to use different image names. For example, developers are able to pull images from Docker Hub, but CI machines need to pull from a private registry. | ||
* Your private registry has copies of images from Docker Hub where the names are predictable, and just adding a prefix is enough. | ||
For example, `registry.mycompany.com/mirror/mysql:8.0.24` can be derived from the original Docker Hub image name (`mysql:8.0.24`) with a consistent prefix string: `registry.mycompany.com/mirror` | ||
|
||
In this case, image name references in code are **unchanged**. | ||
i.e. you would leave as-is: | ||
|
||
<!--codeinclude--> | ||
[Unchanged direct Docker Hub image name](../../container_test.go) inside_block:directDockerHubReference | ||
<!--/codeinclude--> | ||
|
||
You can then configure _Testcontainers for Go_ to apply a given prefix (e.g. `registry.mycompany.com/mirror`) to every image that it tries to pull from Docker Hub. Important to notice that **the prefix should not include a trailing slash**. This can be done in one of two ways: | ||
|
||
* Setting the `TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX=registry.mycompany.com/mirror` environment variable. | ||
* Via config file, setting `hub.image.name.prefix` in the `~/.testcontainers.properties` file in your user home directory. | ||
|
||
_Testcontainers for Go_ will automatically apply the prefix to every image that it pulls from Docker Hub - please verify that all [the required images](#images-used-by-testcontainers) exist in your registry. | ||
|
||
_Testcontainers for Go_ will not apply the prefix to: | ||
|
||
* non-Hub image names (e.g. where another registry is set) | ||
* Docker Hub image names where the hub registry is explicitly part of the name (i.e. anything with a `docker.io` or `registry.hub.docker.com` host part) | ||
|
||
## Developing a custom function for transforming image names on the fly | ||
|
||
Consider this if: | ||
|
||
* You have complex rules about which private registry images should be used as substitutes, e.g.: | ||
* non-deterministic mapping of names meaning that a [name prefix](#automatically-modifying-docker-hub-image-names) cannot be used, or | ||
* rules depending upon developer identity or location, or | ||
* you wish to add audit logging of images used in the build, or | ||
* you wish to prevent accidental usage of images that are not on an approved list. | ||
|
||
In this case, image name references in code are **unchanged**. i.e. you would leave as-is: | ||
|
||
<!--codeinclude--> | ||
[Unchanged direct Docker Hub image name](../../container_test.go) inside_block:directDockerHubReference | ||
<!--/codeinclude--> | ||
|
||
You can implement a custom image name substitutor by: | ||
|
||
* implementing the `ImageNameSubstitutor` interface, exposed by the `testcontainers` package. | ||
* configuring _Testcontainers for Go_ to use your custom implementation, defined at the `ContainerRequest` level. | ||
|
||
The following is an example image substitutor implementation prepending the `docker.io/` prefix, used in the tests: | ||
|
||
<!--codeinclude--> | ||
[Image Substitutor Interface](../../options.go) inside_block:imageSubstitutor | ||
[Noop Image Substitutor](../../container_test.go) inside_block:noopImageSubstitutor | ||
[Docker prefix Image Substitutor](../../container_test.go) inside_block:dockerImageSubstitutor | ||
[Applying the substitutor](../../container_test.go) inside_block:applyImageSubstitutors | ||
<!--/codeinclude--> | ||
|
||
## Images used by Testcontainers | ||
|
||
As of the current version of Testcontainers ({{latest_version}}): | ||
|
||
* every image directly used by your tests | ||
* images pulled by Testcontainers itself to support functionality: | ||
* [`testcontainers/ryuk`](https://hub.docker.com/r/testcontainers/ryuk) - performs fail-safe cleanup of containers, and always required (unless [Ryuk is disabled](./configuration.md#customizing-ryuk-the-resource-reaper)). | ||
* [`alpine`](https://hub.docker.com/r/_/alpine). | ||
* [`Docker in Docker`](https://hub.docker.com/_/docker). | ||
* [`nginx`](https://hub.docker.com/r/_/nginx). | ||
* [`delayed nginx`](https://hub.docker.com/r/menedev/delayed-nginx). | ||
* [`localstack`](https://hub.docker.com/r/localstack/localstack). | ||
* [`mysql`](https://hub.docker.com/r/_/mysql). | ||
* [`postgres`](https://hub.docker.com/r/_/postgres). | ||
* [`postgis`](https://hub.docker.com/r/postgis/postgis). | ||
* [`redis`](https://hub.docker.com/r/_/redis). | ||
* [`registry`](https://hub.docker.com/r/_/registry). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I missed this in a previous PR, but wonder in which uses cases do we need more than one substitute. If a substitute is added then the new image name should be pull from the registry, in this case, what happens when more than one is added. If it is not intended then should we deprecate it in a next PR and only accept one?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mmmm I see your point, now with more perspective given the context from this PR. Yeah, makes sense. We just want one function plus the prefix prepender one, in the case it exists.