Fix BuildConfigInstantiateFailed warning when lastVersion == 0 #17146

coreydaley · 2017-11-02T13:18:52Z

Build config instatiate should not emit a warning when
bc.Status.LastVersion != lastVersion and lastVersion == 0
Fixes #16557

coreydaley · 2017-11-02T13:19:14Z

@openshift/sig-developer-experience ptal

bparees · 2017-11-02T13:22:26Z

I don't think this is the right fix. You've not just eliminated the warning, isn't this now going to run two builds (the configchangetriggered one and the imagechangetriggered one)?

openshift-ci-robot · 2017-11-02T21:44:09Z

@coreydaley: The following tests failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/openshift-jenkins/extended_conformance_gce	`14a0f39`	link	`/test extended_conformance_gce`
ci/openshift-jenkins/extended_conformance_install_update	`14a0f39`	link	`/test extended_conformance_install_update`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

bparees · 2018-01-17T20:27:46Z

@coreydaley did you forget about this?

coreydaley · 2018-01-17T20:50:23Z

@bparees nope, it's next on my list

bparees · 2018-02-17T06:44:34Z

Did you delete the branch? That will do it. Ben Parees | OpenShift

…

On Feb 16, 2018 7:18 PM, "Corey Daley" ***@***.***> wrote: Not sure why this got closed? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#17146 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEvl3nJcq30s-F32JJZMrqW2ePney-frks5tVkT6gaJpZM4QPqzA> .

coreydaley · 2018-02-17T15:49:45Z

I reset my branch so that I could start over on this PR and pushed it, weird that it closed, reopening with new code.

coreydaley · 2018-02-17T15:50:33Z

@openshift/sig-developer-experience ptal

bparees · 2018-02-17T18:39:23Z

pkg/build/controller/buildconfig/buildconfig_controller.go

+ // Don't emit this event if the BuildRequest.LastVersion equals zero
+ // Caused by a race condition between the ImageChangeTrigger vs the BuildConfigChangeTrigger
+ // Fixes https://github.com/openshift/origin/issues/16557
+ if *request.LastVersion != int64(0) {


request.LastVersion will always be zero in this codepath. so you're going to mask any/all errors.

I think you need to actually look at the error you got from Instantiate (and/or make Instantiate throw a more distinctive error you can look for)

coreydaley · 2018-02-19T19:40:20Z

@bparees ptal

gabemontero

One item occurred to me ... apologies if it was previously discussed and I missed it.

gabemontero · 2018-02-19T20:04:04Z

pkg/build/controller/buildconfig/buildconfig_controller.go

+ // Caused by a race condition between the ImageChangeTrigger vs the BuildConfigChangeTrigger
+ if !strings.Contains(instantiateErr.Error(), fmt.Sprintf("the LastVersion(1) on build config %s/%s does not match the build request LastVersion(0)", bc.Namespace, bc.Name)) {
+ c.recorder.Event(bc, kapi.EventTypeWarning, "BuildConfigInstantiateFailed", instantiateErr.Error())
+ }


What if a single request comes through (i.e. not a race between ICT and CCT) after the initial BC create with a bad last version? Wouldn't we be masking a legitimate error in that case?

Unfortunately, don't know if I have a super great solution ... perhaps we fetch the BC before performing this filter and see if it was recently created ... where determining what is recent is perhaps kludgy ... but maybe at least an improvement.

Dumb non-transactional databases

What if a single request comes through (i.e. not a race between ICT and CCT) after the initial BC create with a bad last version?

request comes from where? to where? this logic is all in the buildconfig controller. There are no other buildrequests its going to process.

@coreydaley your check on LastVersion(1) is too specific. I could potentially start multiple builds before the buildconfig controller puts in its request. Then the error would be "the LastVersion(5) on build config..." or whatever.

Just check for the back half: does not match the build request LastVersion(0)

coreydaley · 2018-02-19T20:36:06Z

@jim-minter any thoughts?

gabemontero · 2018-02-19T22:03:36Z

On Mon, Feb 19, 2018 at 4:59 PM, Ben Parees ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In pkg/build/controller/buildconfig/buildconfig_controller.go <#17146 (comment)>: > @@ -136,7 +137,11 @@ func (c *BuildConfigController) handleBuildConfig(bc *buildapi.BuildConfig) erro } else if buildgenerator.IsFatal(err) || kerrors.IsNotFound(err) || kerrors.IsBadRequest(err) || kerrors.IsForbidden(err) { instantiateErr = fmt.Errorf("gave up on Build for BuildConfig %s due to fatal error: %v", bcDesc(bc), err) utilruntime.HandleError(instantiateErr) - c.recorder.Event(bc, kapi.EventTypeWarning, "BuildConfigInstantiateFailed", instantiateErr.Error()) + // Fixes #16557 + // Caused by a race condition between the ImageChangeTrigger vs the BuildConfigChangeTrigger + if !strings.Contains(instantiateErr.Error(), fmt.Sprintf("the LastVersion(1) on build config %s/%s does not match the build request LastVersion(0)", bc.Namespace, bc.Name)) { + c.recorder.Event(bc, kapi.EventTypeWarning, "BuildConfigInstantiateFailed", instantiateErr.Error()) + } What if a single request comes through (i.e. not a race between ICT and CCT) after the initial BC create with a bad last version? request comes from where? to where? this logic is all in the buildconfig controller. There are no other buildrequests its going to process.

So you are saying somebody couldn't manually hit the build config rest endpoint with a HTTP req and interfere with this?

…

@coreydaley <https://github.com/coreydaley> your check on LastVersion(1) is too specific. I could potentially start multiple builds before the buildconfig controller puts in its request. Then the error would be "the LastVersion(5) on build config..." or whatever. Just check for the back half: does not match the build request LastVersion(0) — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#17146 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADbadDDqoJdDiT_-Yj_zekzhpI6791kyks5tWe63gaJpZM4QPqzA> .

bparees · 2018-02-19T22:14:19Z

So you are saying somebody couldn't manually hit the build config rest endpoint with a HTTP req and interfere with this?

interfere with this how?

Fixes #16557

coreydaley · 2018-02-19T22:16:00Z

@bparees update pushed

gabemontero · 2018-02-19T22:18:55Z

On Mon, Feb 19, 2018 at 5:14 PM Ben Parees ***@***.***> wrote: So you are saying somebody couldn't manually hit the build config rest endpoint with a HTTP req and interfere with this? interfere with this how?

create a request that is independent of the concurrent requests @coreydaley noted coming from the triggers And when the say the human manually builds ths json for the request he uses the wrong last version

…

— You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#17146 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADbadAM2ioJGQQH4joKYL9HRhO3FROtfks5tWfJGgaJpZM4QPqzA> .

bparees · 2018-02-19T22:22:21Z

And when the say the human manually builds ths json for the request he uses
the wrong last version

the code that handles that buildrequest is not the codepath that @coreydaley added his error checking/filtering to. (both codepaths call instantiate, but only the buildconfigcontroller codepath is having the filtering added to it)

bparees · 2018-02-19T22:22:50Z

/lgtm

openshift-ci-robot · 2018-02-19T22:22:54Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bparees, coreydaley

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

~~pkg/build/OWNERS~~ [bparees]
~~test/cmd/OWNERS~~ [bparees]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

openshift-merge-robot · 2018-02-20T01:03:42Z

Automatic merge from submit-queue.

openshift-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Nov 2, 2017

openshift-merge-robot assigned smarterclayton and csrwng Nov 2, 2017

openshift deleted a comment from openshift-merge-robot Feb 17, 2018

coreydaley closed this Feb 17, 2018

coreydaley reopened this Feb 17, 2018

coreydaley requested a review from bparees February 17, 2018 15:50

coreydaley unassigned smarterclayton and csrwng Feb 17, 2018

openshift-ci-robot added the sig/developer-experience label Feb 17, 2018

bparees reviewed Feb 17, 2018

View reviewed changes

gabemontero suggested changes Feb 19, 2018

View reviewed changes

Fixes BuildConfigInstantiateFailure event on race condition

8f39e27

Fixes #16557

openshift-ci-robot assigned bparees Feb 19, 2018

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 19, 2018

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 19, 2018

openshift-merge-robot merged commit 4e52d0a into openshift:master Feb 20, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix BuildConfigInstantiateFailed warning when lastVersion == 0 #17146

Fix BuildConfigInstantiateFailed warning when lastVersion == 0 #17146

coreydaley commented Nov 2, 2017

coreydaley commented Nov 2, 2017

bparees commented Nov 2, 2017

openshift-ci-robot commented Nov 2, 2017 •

edited

bparees commented Jan 17, 2018

coreydaley commented Jan 17, 2018

bparees commented Feb 17, 2018 via email

coreydaley commented Feb 17, 2018

coreydaley commented Feb 17, 2018

bparees Feb 17, 2018

coreydaley commented Feb 19, 2018

gabemontero left a comment

gabemontero Feb 19, 2018

bparees Feb 19, 2018

coreydaley commented Feb 19, 2018

gabemontero commented Feb 19, 2018 via email

bparees commented Feb 19, 2018

coreydaley commented Feb 19, 2018

gabemontero commented Feb 19, 2018 via email

bparees commented Feb 19, 2018

bparees commented Feb 19, 2018

openshift-ci-robot commented Feb 19, 2018

openshift-merge-robot commented Feb 20, 2018

Fix BuildConfigInstantiateFailed warning when lastVersion == 0 #17146

Fix BuildConfigInstantiateFailed warning when lastVersion == 0 #17146

Conversation

coreydaley commented Nov 2, 2017

coreydaley commented Nov 2, 2017

bparees commented Nov 2, 2017

openshift-ci-robot commented Nov 2, 2017 • edited

bparees commented Jan 17, 2018

coreydaley commented Jan 17, 2018

bparees commented Feb 17, 2018 via email

coreydaley commented Feb 17, 2018

coreydaley commented Feb 17, 2018

bparees Feb 17, 2018

Choose a reason for hiding this comment

coreydaley commented Feb 19, 2018

gabemontero left a comment

Choose a reason for hiding this comment

gabemontero Feb 19, 2018

Choose a reason for hiding this comment

bparees Feb 19, 2018

Choose a reason for hiding this comment

coreydaley commented Feb 19, 2018

gabemontero commented Feb 19, 2018 via email

bparees commented Feb 19, 2018

coreydaley commented Feb 19, 2018

gabemontero commented Feb 19, 2018 via email

bparees commented Feb 19, 2018

bparees commented Feb 19, 2018

openshift-ci-robot commented Feb 19, 2018

openshift-merge-robot commented Feb 20, 2018

openshift-ci-robot commented Nov 2, 2017 •

edited