node-www out of space?

[BethGriggs]

Lots of the release builds failed overnight

e.g. https://ci-release.nodejs.org/job/iojs+release/nodes=centos7-ppcle-release-64/5054/console

10:56:07 ssh node-www "mkdir -p nodejs/release/v13.8.0"
10:56:08 mkdir: cannot create directory ‘nodejs/release/v13.8.0’: No space left on device
10:56:08 Makefile:1139: recipe for target 'binary-upload' failed

[targos]

Ping @nodejs/build-infra

[targos]

@mhdawson @jbergstroem @joaocgreis @rvagg

[sam-github]

For the future: maybe someone on the release team should have build-infra access? Or at least node-www?

[targos]

We do have limited access to the machine, but not root.
I found that the releases (mostly nightlies) take up about all the available space (2 TB).

[sam-github]

We have nightlies going back to 2016

https://nodejs.org/download/nightly/

I might misunderstand their value, but I don't see why keeping more than a few weeks is useful.

[addaleax]

I might misunderstand their value, but I don't see why keeping more than a few weeks is useful.

I’ve occasionally used them for bisecting Node.js – it’s definitely faster than building Node.js from scratch multiple times.

But yeah, they should be low-priority and deleting one or two years’ worth of nightlies should be just fine, imo.

[joyeecheung]

Do we have backup for these? I guess we could save them somewhere else and bring them back when we have more space? (Or ideally put these nightlies in some cloud storage?)

EDIT: oh but if they are all reproducible I guess we just need to save the commit hashes and could always rebuild them all later.

[rvagg]

Gee, that filled up much quicker than anticipated. Doing an analysis now of what we have but in the meantime I've freed up ~3G of stuff littering staging so there should be space to publish some releases if needed now @BethGriggs.

[rvagg]

OK, I have 38G of space freed up for breathing room while we decide next steps.

Here's a breakdown of dist/nodejs which takes up the majority of space on that device:

1,013G	nightly
239G	v8-canary
150G	chakracore-nightly
147G	release
78G	rc
48G	test
4.6G	chakracore-release
823M	chakracore-rc
34M	docs
4.0K	next-nightly

Some options:

• A removal policy for v8-canary builds - someone want to propose one?
• A removal policy for nightly builds - someone want to propose one?
• Offload some or all of these to an object store. Maybe Google Cloud Storage since we have some credits now. We could add a redirect or just proxy from our servers to make it transparent (proxy might be nice since it would be transparent to Cloudflare too then). There's been a long-standing goal of architecture our web presence which includes getting these files off servers and into a better long-term storage system.
• Double the storage again, this is a reasonable path and doable, but maybe not necessary with one or more of the above options

[BethGriggs]

Thanks, @rvagg. I'll start kicking off the new release builds now - https://ci-release.nodejs.org/job/iojs+release/5055 (v13.x)

[rvagg]

I'm expanding on-server storage for now, at least as an interim measure since I doubt we'll make these decisions quickly.

[targos]

I think we should have an incremental removal policy for both v8-canary and nightly builds.
Something like:

• keep all builds for N weeks
• then keep 1 build per week for M months
• then keep 1 build per month forever

[rvagg]

^ that would be a good basis for a PR for a new policy doc so we can nail it down and agree on what we're going to do (if anything).

For now I've finalised an upgrade which doubles the size of that disk, so we have 2Tb of headroom now before needing to take more action. Old disk still mounted for reference but I'll discard it at some point in the near future.