incident: git binary repo is too big (78GB) - node-test-binary-windows timing out

[refack]

I've run a git-delete-branch job, but the local repos are too "fat" so the auto GC is timing out.

[Trott]

Looks to me like it might only be failing on test-rackspace-win2008r2-x64-4? Going to try taking that one offline and running again...

[Trott]

Oh, nope, that's totally wrong, it's a bunch of hosts, but it's strange (to me) that some fail consistently and others don't.

[Trott]

Seeing what happens if I take all the failing nodes offline because why not? It's not as if anyone can get a CI run out of them right now.

Took these offline:

• test-rackspace-win2008r2-x64-4
• test-azure_msft-win10-x64-3
• test-azure_msft-win10-x64-1
• test-azure_msft-win2012r2-x64-3
• test-azure_msft-win2012r2-x64-2

And here's a run to see if it fixes things or not: https://ci.nodejs.org/job/node-test-commit-windows-fanned/12947/

[refack]

Cleaned and brought online:

• test-rackspace-win2008r2-x64-4
• test-azure_msft-win10-x64-3
• test-azure_msft-win10-x64-1
• test-azure_msft-win2012r2-x64-3
• test-azure_msft-win2012r2-x64-2

@Trott if you see a stalled job you can go to it's job page (for example
https://ci.nodejs.org/job/node-test-binary-windows/COMPILED_BY=vs2015-x86,RUNNER=win2012r2,RUN_SUBSET=3/ )
if it's the top job you can click workspace and wipe it.

As for intermittent failures, I think it's because this is triggered by git's auto-GC logic...

[Trott]

Results are better, but still some failures...took this one offline:

• test-azure_msft-win10-x64-5

There are two more with consistent build failures but they're in the middle of doing something right now so I don't want to take offline until they really fail again.

https://ci.nodejs.org/computer/test-azure_msft-win10-x64-1/builds
https://ci.nodejs.org/computer/test-rackspace-win2008r2-x64-5/builds

[Trott]

Build history is too convincing. Took these offline too:

• test-azure_msft-win10-x64-1
• test-rackspace-win2008r2-x64-5

[Trott]

Trying again to see if we can get a green Windows build now: https://ci.nodejs.org/job/node-test-commit-windows-fanned/12952/

[Trott]

Looks like it's gonna be green this time. Not sure how to fix the offline hosts but at least CI isn't perma-red.

[Trott]

Took test-azure_msft-win10-x64-5 offline too. Build failures, obviously.

• test-azure_msft-win10-x64-5

[refack]

• test-requireio_davglass-debian7-arm_pi1p-1 back (was errcode failed-to-remove-.nfs ARM failures in CI #951)

[refack]

All windows machines are back.
3 windows-fanned CI:
https://ci.nodejs.org/job/node-test-commit-windows-fanned/12986/
https://ci.nodejs.org/job/node-test-commit-windows-fanned/12987/
https://ci.nodejs.org/job/node-test-commit-windows-fanned/12988/

[refack]

test-azure_msft-win10-x64-5 was a little stubborn, but should now be Ok.
Also all PIs are back online.

[joaocgreis]

Some workers were still failing. The node-test-binary-windows job is set up to only fetch the git branch with the binaries, so the problem is not the size of the binary repo but the local git repository. The automatic git gc would take too long.

I created a new job git-clean-windows similar to git-clean-rpi set to run evey week, should prevent this from happening again. I'll close this issue after I confirm it is working as expected.

[refack]

@joaocgreis it seems like the local git repo does accumulate binaries even the branch was deleted from the remote. I rerun a node-test-binary-windows job and some workers were able to find the revision:
https://drive.google.com/file/d/0Bz0LZMH4OpErbVZkSm9YSFhCTVU/view?usp=sharing

[gibfahn]

@joaocgreis it seems like the local git repo does accumulate binaries even the branch was deleted from the remote. I rerun a node-test-binary-windows job and some workers were able to find the revision:

Does git fetch --prune help? That removes the tracking branches from remote repos (which are not otherwise removed).

I also run this locally to delete local branches which had an upstream, but the upstream was deleted:

# Delete orphaned local branches.
git fetch -p && git branch -vv | awk '/: gone]/{print $1}' | xargs git branch -D