Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Build Cop: Flaky Tests are hard to track #281

Closed
nnegrey opened this issue Feb 14, 2020 · 5 comments · Fixed by #285 or #296
Closed

Build Cop: Flaky Tests are hard to track #281

nnegrey opened this issue Feb 14, 2020 · 5 comments · Fixed by #285 or #296
Assignees
@tbpg
Labels
bot: flakybot priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@nnegrey
Copy link

nnegrey commented Feb 14, 2020

Flaky tests seem to get opened, closed, opened, closed, opened, closed by the bot, which makes it relatively difficult to properly track what I'm trying to fix.

One example: GoogleCloudPlatform/python-docs-samples#2910
@tbpg tbpg self-assigned this Feb 14, 2020
@tbpg tbpg changed the title Flaky Tests are hard to track Build Cop: Flaky Tests are hard to track Feb 14, 2020
@tbpg tbpg added priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Feb 14, 2020
@tbpg
Copy link
Contributor

tbpg commented Feb 14, 2020

Idea:

If the bot sees that the issue has been opened and closed 2 or more times, add a buildcop: flaky label and comment on it saying it's up to you to close this issue.

Then, if the test fails after someone other than the bot closed it, reopen it and say it's up to you to close this.

@nnegrey
Copy link
Author

nnegrey commented Feb 14, 2020

Yea, something that reverts to a human having to come in and fix would be nifty.

I had 3 the other day that I created a PR to fix and before it was merged 2 of them automatically closed.

@tbpg
Copy link
Contributor

tbpg commented Feb 14, 2020

Sorry about that! I'll move forward with the idea.

tbpg added a commit that referenced this issue Feb 14, 2020
@tbpg
feat(Build Cop): handle flaky tests
0da1da5 
If the bot reopens an issue, it marks it as flaky then stops commenting
on it and will not close it.

If a flaky test is closed (by a human) then fails again, it will be reopened.

Fixes #281.
tbpg added a commit that referenced this issue Feb 14, 2020
@tbpg
feat(Build Cop): handle flaky tests
7802d30 
If the bot reopens an issue, it marks it as flaky then stops commenting
on it and will not close it.

If a flaky test is closed (by a human) then fails again, it will be reopened.

Fixes #281.
tbpg added a commit that referenced this issue Feb 14, 2020
@tbpg
feat(Build Cop): handle flaky tests
05551ca 
If the bot reopens an issue, it marks it as flaky then stops commenting
on it and will not close it.

If a flaky test is closed (by a human) then fails again, it will be reopened.

Fixes #281.
@tbpg tbpg mentioned this issue Feb 14, 2020
Merged
@tbpg tbpg closed this as completed in #285 Feb 15, 2020
tbpg added a commit that referenced this issue Feb 15, 2020
@tbpg
feat(Build Cop): handle flaky tests (#285)
4dd6255 
If the bot reopens an issue, it marks it as flaky then stops commenting
on it and will not close it.

If a flaky test is closed (by a human) then fails again, it will be reopened.

Fixes #281.
@tbpg tbpg reopened this Feb 18, 2020
@tbpg
Copy link
Contributor

tbpg commented Feb 18, 2020

This didn't work for GoogleCloudPlatform/python-docs-samples#2898.

@tbpg
Copy link
Contributor

tbpg commented Feb 19, 2020

Another example of this not working: GoogleCloudPlatform/python-docs-samples#2875

@tbpg tbpg mentioned this issue Feb 19, 2020
Merged
@tbpg tbpg closed this as completed in #296 Feb 19, 2020
yoshi-automation added a commit that referenced this issue Apr 1, 2020
@yoshi-automation
feat(Build Cop): handle flaky tests (#285)
fe5c3bf 
4dd6255
commit 4dd6255
Author: Tyler Bui-Palsulich <26876514+tbpg@users.noreply.github.com>
Date:   Fri Feb 14 19:37:55 2020 -0500

    feat(Build Cop): handle flaky tests (#285)

    If the bot reopens an issue, it marks it as flaky then stops commenting
    on it and will not close it.

    If a flaky test is closed (by a human) then fails again, it will be reopened.

    Fixes #281.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tbpg tbpg

Labels
bot: flakybot priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Milestone
No milestone
2 participants
@nnegrey @tbpg