Logic to pull bundle from quay.io #3221

praveenkumar · 2022-06-16T12:19:22Z

This is initial pr to switch bundle download from mirror to quay.io
End user shouldn't affect with this change except there is no download
resume functionality.

The bundle images are pushed to quay.io/crcont with following name

ocp-<hypervisor>-bundle
podman-<hypervisor>-bundle

This PR doesn't remove existing codebase to download bundles from mirror
as of now but it should be done in a separate commit so we can easily
revert back.

$ crc setup
[...]
INFO Getting bundle for the CRC executable
INFO Downloading crc_libvirt_4.10.14_amd64.crcbundle
Getting image source signatures
Copying blob 7e0dbb47dc47 done
Copying config 4f32695e8f done
Writing manifest to image destination
Storing signatures
INFO Extracting the image bundle layer...
crc_libvirt_4.10.14_amd64.crcbundle: 3.15 GiB / 3.15 GiB [---------------------------------------------------------------------------------------------------------------------------] 100.00%
INFO Uncompressing /home/prkumar/.crc/cache/crc_libvirt_4.10.14_amd64.crcbundle
crc.qcow2: 12.61 GiB / 12.61 GiB [---------------------------------------------------------------------------------------------------------------------------------------------------] 100.00%
oc: 117.14 MiB / 117.14 MiB [--------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00%

Fixes: #3217

pkg/crc/preflight/preflight_checks_common.go

cfergeau · 2022-06-20T15:01:33Z

One initial comment, this makes the binaries 10MB/20% bigger

praveenkumar · 2022-06-21T05:22:51Z

One initial comment, this makes the binaries 10MB/20% bigger

Yes, because of c/image dependency, it is making the binary bit more bigger.

pkg/crc/image/image.go

cfergeau · 2022-06-21T12:10:11Z

One initial comment, this makes the binaries 10MB/20% bigger

Yes, because of c/image dependency, it is making the binary bit more bigger.

20% is not just a bit ;)

cfergeau

I'm find the log confusing

This is initial pr to switch bundle download from mirror to quay.io
End user shouldn't affect with this change except there is no download
resume functionality.

After reading this, I don't know if bundle download is fully moved to getting it from quay.io or not ("initial PR", "end user shouldn't affect with this change"). I would say explicitly that after this commit, bundles are downloaded from quay.io even if the old download code is still there.

Fwiw, I find the name pkg/crc/image very (too) generic, is this about picture, an ISO image, a qcow2 image, ... ? Might be worth naming this container-image, or maybe just container for the package name since the type used is then OCIImage.. ?

cfergeau · 2022-06-20T15:22:39Z

Makefile

@@ -68,6 +68,8 @@ endif

 # https://golang.org/cmd/link/
 LDFLAGS := $(VERSION_VARIABLES) ${GO_EXTRA_LDFLAGS}
+# Same build flags are used in the podman remote to cross build it https://github.com/containers/podman/blob/main/Makefile
+BUILDTAGS := remote exclude_graphdriver_btrfs btrfs_noversion exclude_graphdriver_devicemapper containers_image_openpgp


looks quite odd in our build logs :-/

Nothing much I can do :)

cfergeau · 2022-06-20T15:26:22Z

pkg/crc/image/image.go

+// CopyImage pull the image from the registry and put it to local storage
+func (img OCIImageHandler) CopyImage(destPath string) ([]byte, error) {
+	// Source Image from docker transport
+	srcImg := fmt.Sprintf("docker://%s:%s", img.uri(), img.ImageTag)


img.uri() is not a uri, it becomes (closer to) one after you prepend docker://. Since this is the only caller of uri() , I guess docker:// could be moved there?
Also feels odd that ImageTag needs to be separately added, while this is also the only location making use of it.

pkg/crc/preflight/preflight_checks_common.go

cfergeau · 2022-06-21T12:15:34Z

pkg/crc/image/image.go

+	v1 "github.com/opencontainers/image-spec/specs-go/v1"
+)
+
+var (


I think this could be const

cfergeau · 2022-06-21T12:21:33Z

pkg/crc/image/image.go

+
+type OCIImageHandler struct {
+	ImageName string
+	ImageTag  string


Not clear to me why we hardcode quay.io and crcont but we let external code specify which image/tag should be downloaded, especially with GetImageName implemented in this file.

Didn't get what you mean about external code as of now GetImageName only provide the image name for preset and then we define this struct and use the method to copy it.

preflight_checks_common.go has code to fill ImageName/ImageTag:

imgHandler := image.OCIImageHandler{ ImageName: image.GetImageName(preset), ImageTag: version.GetCRCVersion(), Progress: os.Stdout, }

but it calls into image to fill this. Then image.go hardcodes quay.io/crcont. Half of what is needed to fetch the image is set in preflight_checks_common.go and the other half in image.go. preflight_checks_common.go does not need to set this imo, ImageName/ImageTag could both be set internally in image.go

cfergeau · 2022-06-21T12:29:36Z

pkg/crc/image/image.go

+	}
+}
+
+// CopyImage pull the image from the registry and put it to local storage


local storage means destPath? It could be misread as podman local image s torage, so I'd mention destpath in the doc.

@cfergeau you mean podman load <image.tar> ?

Also I will put destpath in doc.

cfergeau · 2022-06-21T12:31:09Z

pkg/crc/image/image.go

+		})
+}
+
+func GetBundleImageLayerFromManifest(manifestData []byte) (string, error) {


If you return a v1.Manifest from CopyImage, then you don't need the very long name, just GetBundleImageLayer works, or GetLayerPath or so

As long as it is easy to read or understand I don't mind to have a very long function. if we return v1.Manifest from image we need to process the byte there instead here.

As long as it is easy to read or understand I don't mind to have a very long function.

Something shorter will be easier to read and understand most of the time :) With a longer function name with more details in higher level code, you still need to read it, parse it, and understand what you need to ignore from the long name,...

cfergeau · 2022-06-21T12:36:05Z

pkg/crc/image/image.go

+	if err := json.Unmarshal(manifestData, manifest); err != nil {
+		return "", err
+	}
+	return removeShaFromDigestString(manifest.Layers[0].Digest.String()), nil


not sure this deserves a separate helper since it's going to be a one-liner used once, and this method is also not very long.
I tried this code with multi-layer images, and it did not work at all (expected, and not saying it has to be fixed). I was wondering if there's a way of detecting this though, maybe disallow multi-layer images, and also check the MediaType to make sure it's the expected one?

I tried this code with multi-layer images, and it did not work at all (expected, and not saying it has to be fixed). I was wondering if there's a way of detecting this though, maybe disallow multi-layer images, and also check the MediaType to make sure it's the expected one?

I am not sure why it shouldn't work with multi layer, the logic here is just return the first layer digest, in your test does the first layer didn't have any digest or having a digest without sha256?

It will do "something" for multilayer images, but will it be doing the right thing, or something unexpected to the user? Imo it's not very well defined what will happen, so it can just error out so that we catch immediatly that something was off.

pkg/crc/image/image.go

cfergeau · 2022-06-21T12:46:08Z

pkg/crc/preflight/preflight_checks_common.go

+	if err != nil {
+		return err
+	}
+	return nil


Wondering if this deserves to be a helper in the image package as this is quite a few low-level steps that needs to be done, not sure preflight should have that much knowledge about how a container image is stored.

Same, not a strong opinion, can be part of image package.

I'd try to move it to image, but not 100% sure yet it will be the right place /o\ I leave that up to you :)

cfergeau · 2022-06-21T13:57:27Z

pkg/crc/image/image.go

+	orgName     = "crcont"
+)
+
+type OCIImageHandler struct {


Not sure OCI is really useful here, we hardcode docker: for fetching from the remote repository, and use dir: for local storage. Maybe this is still stored in an OCI compliant way, but this really is not the most important part of the implementation, especially as we don't support many layer formats.

I think it is good to be explicit around what kind of image format we are storing to dir and right now quay.io pulled images are in oci image format so better to have it.

The user of this API don't care about the format it's in.. If oci appears internally in the image.go implementation, why not, but I don't think it's useful in the public API. Also not clear why we are using dir: rather than oci: for example? If oci is not exposed in the podman/... API we are using, not sure we need to expose it ourselves.

pkg/crc/image/image.go

gbraad · 2022-06-27T11:20:28Z

pkg/crc/image/image.go

+)
+
+const (
+	registryURI = "quay.io"


can these be defined in a constants.go file instead?

@gbraad yes we can move it to constants.go, As of now it is here just to have all image related stuff in same package.

Maybe pkg/crc/image/constants.go ? I also prefer to keep this private to the image package as long as other code does not need this.

Only for 2-3 constants which also used for same package I don't feel create a different constants file.

pkg/crc/image/image.go

cfergeau

Still some comments/questions, but looking good overall

Makefile

cfergeau · 2022-06-28T12:26:28Z

Makefile

-	GOOS=darwin $(TOOLS_BINDIR)/golangci-lint run
-	GOOS=linux $(TOOLS_BINDIR)/golangci-lint run
-	GOOS=windows $(TOOLS_BINDIR)/golangci-lint run
+	GOOS=darwin $(TOOLS_BINDIR)/golangci-lint run --build-tags "$(BUILDTAGS)"


There are already build tags defined in .golangci.yml , they should be all set in the same place, either here or in the .yml file.

cfergeau · 2022-06-28T12:33:11Z

pkg/crc/image/image.go

+)
+
+const (
+	registryURI = "quay.io"


Maybe pkg/crc/image/constants.go ? I also prefer to keep this private to the image package as long as other code does not need this.

cfergeau · 2022-06-28T12:34:31Z

pkg/crc/image/image.go

+	policy := &signature.Policy{Default: []signature.PolicyRequirement{signature.NewPRInsecureAcceptAnything()}}
+	policyContext, err := signature.NewPolicyContext(policy)
+	if err != nil {
+		return nil, fmt.Errorf("error creating security context: %s", err.Error())


Could be "...: %w, err)`

pkg/crc/image/image.go

cfergeau · 2022-06-28T12:54:51Z

pkg/crc/preflight/preflight_checks_common.go

@@ -98,7 +99,7 @@ func fixBundleExtracted(bundlePath string, preset preset.Preset) func() error {
 		}
 		if bundlePath == constants.GetDefaultBundlePath(preset) {
 			logging.Infof("Downloading %s", constants.GetDefaultBundle(preset))
-			if err := bundle.Download(preset); err != nil {
+			if err := image.PullBundleImage(preset, os.Stdout); err != nil {


NB: This could be checking the bundle sha256sum for now.

Not sure it's really useful to pass os.Stdout as an argument, this could be hardcoded in the image module, similarly to context.Background()

This could be checking the bundle sha256sum for now.

After this initial PR next would be verify using signed bits so it will be taken care of.

But we don't want to wait for this 'signature' PR before merging this one, do we? Given our fast release cycles, I'm a bit uncomfortable merging a PR which depends on a non-existent PR which has to be in before our next release.

@cfergeau you want to check hardcoded bundle sha256sum for bundle which are part of the image right now?

pkg/crc/image/image.go

praveenkumar · 2022-07-04T08:29:52Z

/retest

pkg/crc/gpg/gpg.go

pkg/crc/config/settings.go

pkg/crc/gpg/gpg.go

pkg/crc/image/image.go

pkg/crc/gpg/gpg.go

pkg/crc/image/image.go

cfergeau · 2022-07-05T09:20:31Z

pkg/crc/image/image.go

+	}
+}
+
+// copyImage pull the image from the registry and put it to destination path


pulls ... puts

pkg/crc/preflight/preflight_checks_common.go

pkg/crc/image/image.go

cfergeau · 2022-07-05T11:56:33Z

Something which is not clear to me is how/when we want to ship this? Do we switch all bundles to quay.io? Do we want to keep shipping OpenShift bundles on mirror.openshift.com, while moving the more community-oriented ones to quay.io? Or do we ship everything on quay?
If we want to be able to have at the same time bundles in quay.io, and bundles on mirror.openshift.com, we could extend https://github.com/code-ready/crc/blob/main/pkg/crc/machine/bundle/constants.go to deal with both. This would also allow a progressive switch to using quay.io while we refine the release process (for example we could decide to only put one or 2 bundles there initially).

praveenkumar · 2022-07-05T12:12:39Z

Something which is not clear to me is how/when we want to ship this?

End of this sprint.

Do we switch all bundles to quay.io?

Yes, plan is to switch all bundles to quay.io

Do we want to keep shipping OpenShift bundles on mirror.openshift.com, while moving the more community-oriented ones to quay.io? Or do we ship everything on quay?

Ship everything on quay (also update the bundle on mirror for time being (1-2 sprint) in case outage on quay end) so user can download it manually and use --bundle option (as fallback).

gbraad · 2022-07-05T13:27:24Z

Yes, plan is to switch all bundles to quay.io

can we allow to use mirror first for this? For this we have been signed off. other locations I am not sure how they will respond.

I would almost suggest to have a config parameter to allow this to be toggled ? @cfergeau WDYT?

cfergeau · 2022-07-05T13:39:50Z

I would almost suggest to have a config parameter to allow this to be toggled ? @cfergeau WDYT?

Works for me, though UI will be a bit odd if at a later point we ship some bundles only through mirror.openshift.com or only through an image registry.

gbraad · 2022-07-05T13:41:02Z

a bit odd if at a later point we ship some bundles only through mirror.openshift.com or only through an image registry.

Therefore more of a preference option and not conclusive.

cfergeau · 2022-07-06T08:06:10Z

pkg/crc/image/image.go

+		OSChoice:           runtime.GOOS,
+		ArchitectureChoice: runtime.GOARCH,
+	}
+}


fwiw, these are the default values

// If not "", overrides the use of platform.GOARCH when choosing an image or verifying architecture match. ArchitectureChoice string // If not "", overrides the use of platform.GOOS when choosing an image or verifying OS match. OSChoice string

probably not worth the helper/..., we can just use an empty context.

cfergeau · 2022-07-06T10:06:18Z

pkg/crc/image/image.go

+	logging.Debugf("Bundle and sign path %v", fileList)
+
+	logging.Info("Verifying the bundle signature...")
+	if err := gpg.Verify(fileList[0], fileList[1]); err != nil {


I would not hardcode that first file in the archive will always be the bundle, and second file its signature.

@cfergeau this is the way our script will create image, should we compare the size or with the extension?

It is now compare with extension so should be good.

pkg/crc/gpg/gpg.go

cfergeau

Some minor comments left, but overall looks good!

cfergeau · 2022-07-11T09:01:19Z

go.mod

@@ -31,6 +32,7 @@ require (
 	github.com/mdlayher/vsock v1.1.1
 	github.com/onsi/ginkgo v1.16.5
 	github.com/onsi/gomega v1.19.0
+	github.com/opencontainers/image-spec v1.0.2-0.20190823105129-775207bd45b6


Why such an old version? podman uses go.mod: github.com/opencontainers/image-spec v1.0.3-0.20220114050600-8b9d41f48198, go get -u github.com/opencontainers/image-spec switches to the v1.0.2 tag which was created in 2021.

@cfergeau I am not sure https://github.com/opencontainers/image-spec here I don't even find v1.0.3 tag or branch.

opencontainers/image-spec#918 there is request for release.

I don't even find v1.0.3 tag or branch

$ go get -u github.com/opencontainers/image-spec@main go: downloading github.com/opencontainers/image-spec v1.0.3-0.20220512140940-7b36cea86235 go: upgraded github.com/opencontainers/image-spec v1.0.2 => v1.0.3-0.20220512140940-7b36cea86235

It's go which refers to a non existent 1.0.3 version when we ask it to use the main branch. But as I said in my earlier comment, using the latest release (1.0.2) is also fine with me.

Looking again at this, it seems the 20190823105129 is misleading, and that v1.0.2-0.20190823105129-775207bd45b6 might be newer than v1.0.2, but history is complicated with a lot of merges, so not sure :-/

EDIT: there's a v1.0 branch with not many commits, and a main branch with a lot more commits. The v1.0.2 tag was cut from the v1.0 branch, and does not have the commit corresponding to v1.0.2-0.20190823105129-775207bd45b6 for example. I still think we should either use v1.0.2 or main, but not some very arbitrary untagged commit.

cfergeau · 2022-07-11T09:29:08Z

pkg/crc/image/image.go

+		return err
+	}
+	defer os.RemoveAll(destDir)
+	imgDigest, err := imgHandler.copyImage(destDir, os.Stdout)


imgManifest

cfergeau · 2022-07-11T09:30:11Z

pkg/crc/image/image.go

+	if err != nil {
+		return err
+	}
+	return nil


This can be return err

cfergeau · 2022-07-11T09:34:35Z

pkg/crc/constants/constants.go

@@ -41,6 +41,21 @@ const (
 	RegistryURI   = "quay.io/crcont"
 	ClusterDomain = ".crc.testing"
 	AppsDomain    = ".apps-crc.testing"
+
+	GPGPublicKey = `-----BEGIN PGP PUBLIC KEY BLOCK-----


Might be worth a comment explaining the identity this key corresponds to, link to a public keyserver, ...?

@cfergeau yes I will to public keyserver and add link here.

cfergeau · 2022-07-11T09:37:02Z

pkg/crc/gpg/gpg.go

+	}
+	defer func() {
+		if err1 := signature.Close(); err1 != nil {
+			err = err1


Hmm if err is already set, won't this hide the initial error? (same question for the other occurrence in this function)

yes https://trstringer.com/golang-deferred-function-error-handling/ looks like we need multi error in this case to append the error.

Since we are open file in reading mode so in defer I think we are OK to just use f.Close() without handling the error. if we open file in write mode then it is more appropriate to handle defer function error.

Since we are open file in reading mode so in defer I think we are OK to just use f.Close()

I think I had the same thoughts in an earlier version of this PR :)
And in write mode, I'd just ignore Close errors if err is already set rather than usin multi error.

cfergeau · 2022-07-11T09:37:45Z

go.mod

@@ -192,6 +193,7 @@ require (

 replace (
 	github.com/apcera/gssapi => github.com/openshift/gssapi v0.0.0-20161010215902-5fb4217df13b
+	golang.org/x/crypto/openpgp => github.com/ProtonMail/go-crypto/openpgp v0.0.0-20220623141421-5afb4c282135


This PR directly imports github.com/ProtonMail/go-crypto/openpgp, is the replace required?

cfergeau · 2022-07-11T09:40:14Z

pkg/crc/image/image.go

+
+	logging.Info("Verifying the bundle signature...")
+	if len(fileList) != 2 {
+		return fmt.Errorf("image layer contains more files than expected %v", fileList)


Nit: : %v rather than just %v

cfergeau · 2022-07-11T09:41:05Z

pkg/crc/gpg/gpg.go

+)
+
+func Verify(filePath, signatureFilePath string) (err error) {
+	signed, err := os.Open(filePath)


Might be just me, but signed feels a bit unspecific. signedData, signedFile, ... ? Or even omit the signed part of it?

openshift-ci · 2022-07-11T09:42:10Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cfergeau

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [cfergeau]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

In this pr OKD and podman bundles are downloaded from quay.io The bundle images are pushed to quay.io/crcont with following name - okd-bundle - podman-bundle ``` $ crc setup [...] INFO Getting bundle for the CRC executable INFO Downloading crc_libvirt_4.10.14_amd64.crcbundle Getting image source signatures Copying blob 7e0dbb47dc47 done Copying config 4f32695e8f done Writing manifest to image destination Storing signatures INFO Extracting the image bundle layer... crc_libvirt_4.10.14_amd64.crcbundle: 3.15 GiB / 3.15 GiB [---------------------------------------------------------------------------------------------------------------------------] 100.00% INFO Uncompressing /home/prkumar/.crc/cache/crc_libvirt_4.10.14_amd64.crcbundle crc.qcow2: 12.61 GiB / 12.61 GiB [---------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% oc: 117.14 MiB / 117.14 MiB [--------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% ```

openshift-ci · 2022-07-12T06:32:06Z

New changes are detected. LGTM label has been removed.

As part of container image we also have detached gpg bundle signature and this pr is verify that with bundle to make sure bundle is coming from us.

openshift-ci bot requested review from anjannath and cfergeau June 16, 2022 12:19

praveenkumar force-pushed the issue_3217 branch from 9aa2a19 to c73c609 Compare June 16, 2022 12:45

anjannath reviewed Jun 20, 2022

View reviewed changes

pkg/crc/preflight/preflight_checks_common.go Outdated Show resolved Hide resolved

praveenkumar force-pushed the issue_3217 branch from c73c609 to fd4edb5 Compare June 20, 2022 14:33

anjannath reviewed Jun 21, 2022

View reviewed changes

pkg/crc/image/image.go Outdated Show resolved Hide resolved

anjannath reviewed Jun 21, 2022

View reviewed changes

pkg/crc/image/image.go Outdated Show resolved Hide resolved

praveenkumar force-pushed the issue_3217 branch from fd4edb5 to 48c6b5d Compare June 21, 2022 12:05

praveenkumar force-pushed the issue_3217 branch from 48c6b5d to 3b46f00 Compare June 21, 2022 12:24

cfergeau reviewed Jun 21, 2022

View reviewed changes

pkg/crc/image/image.go Outdated Show resolved Hide resolved

praveenkumar force-pushed the issue_3217 branch 2 times, most recently from 8e27579 to 522280a Compare June 24, 2022 09:45

gbraad reviewed Jun 27, 2022

View reviewed changes

pkg/crc/image/image.go Show resolved Hide resolved

cfergeau reviewed Jun 28, 2022

View reviewed changes

praveenkumar force-pushed the issue_3217 branch 2 times, most recently from 452b081 to 50bf396 Compare July 1, 2022 09:38

praveenkumar force-pushed the issue_3217 branch from 50bf396 to 3b8140f Compare July 5, 2022 06:09

gbraad reviewed Jul 5, 2022

View reviewed changes

pkg/crc/gpg/gpg.go Outdated Show resolved Hide resolved

gbraad reviewed Jul 5, 2022

View reviewed changes

pkg/crc/gpg/gpg.go Outdated Show resolved Hide resolved

praveenkumar force-pushed the issue_3217 branch from 3b8140f to ac1c5b6 Compare July 5, 2022 10:23

cfergeau reviewed Jul 5, 2022

View reviewed changes

praveenkumar force-pushed the issue_3217 branch from ac1c5b6 to 57188fe Compare July 6, 2022 05:16

cfergeau reviewed Jul 6, 2022

View reviewed changes

praveenkumar force-pushed the issue_3217 branch from 57188fe to 4398640 Compare July 6, 2022 09:23

cfergeau reviewed Jul 6, 2022

View reviewed changes

pkg/crc/gpg/gpg.go Outdated Show resolved Hide resolved

praveenkumar force-pushed the issue_3217 branch 2 times, most recently from 3eeb176 to 59b79c7 Compare July 6, 2022 16:01

cfergeau approved these changes Jul 11, 2022

View reviewed changes

openshift-ci bot assigned cfergeau Jul 11, 2022

openshift-ci bot added the lgtm label Jul 11, 2022

openshift-ci bot added the approved label Jul 11, 2022

praveenkumar force-pushed the issue_3217 branch from 59b79c7 to d08468a Compare July 12, 2022 06:32

openshift-ci bot removed the lgtm label Jul 12, 2022

Add bundle signature verification

faeeac0

As part of container image we also have detached gpg bundle signature and this pr is verify that with bundle to make sure bundle is coming from us.

praveenkumar force-pushed the issue_3217 branch from d08468a to faeeac0 Compare July 12, 2022 07:19

praveenkumar merged commit 93babb5 into crc-org:main Jul 12, 2022

anjannath mentioned this pull request Jul 21, 2022

[Task] Add logic to pull bundles from quay.io #3217

Closed

Logic to pull bundle from quay.io #3221

Logic to pull bundle from quay.io #3221

Conversation

praveenkumar commented Jun 16, 2022 • edited Loading

cfergeau commented Jun 20, 2022

praveenkumar commented Jun 21, 2022

cfergeau commented Jun 21, 2022

cfergeau left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

praveenkumar Jun 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

praveenkumar Jun 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

praveenkumar Jun 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cfergeau Jun 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cfergeau left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

praveenkumar commented Jul 4, 2022

Choose a reason for hiding this comment

cfergeau commented Jul 5, 2022

praveenkumar commented Jul 5, 2022

gbraad commented Jul 5, 2022 • edited Loading

cfergeau commented Jul 5, 2022

gbraad commented Jul 5, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

praveenkumar Jul 6, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cfergeau left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cfergeau Jul 12, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openshift-ci bot commented Jul 11, 2022

openshift-ci bot commented Jul 12, 2022

praveenkumar commented Jun 16, 2022 •

edited

Loading

praveenkumar Jun 23, 2022 •

edited

Loading

praveenkumar Jun 23, 2022 •

edited

Loading

praveenkumar Jun 23, 2022 •

edited

Loading

cfergeau Jun 23, 2022 •

edited

Loading

gbraad commented Jul 5, 2022 •

edited

Loading

praveenkumar Jul 6, 2022 •

edited

Loading

cfergeau Jul 12, 2022 •

edited

Loading