Disk full on downloads server

[targos]

Found out because release builds fail to push the binaries:

07:13:16 mkdir: cannot create directory ‘nodejs/nightly/v20.0.0-nightly202212191d6fda0856’: No space left on device

[targos]

3.5T    /home/dist/nodejs

[targos]

150G    ./chakracore-nightly
823M    ./chakracore-rc
4.6G    ./chakracore-release
34M     ./docs
4.0K    ./next-nightly
2.0T    ./nightly
144G    ./rc
309G    ./release
78G     ./test
822G    ./v8-canary

[targos]

@nodejs/build-infra

What should we do? Has this already happened before?

[richardlau]

@nodejs/build-infra

What should we do? Has this already happened before?

Yes, for example:

• node-www out of space? #2162
• Primary nodejs.org server out of space #1202

[sxa]

I think what @targos suggested in #2162 (comment) is reasonable and I've got some scripts that I use elsewhere which can likely enforce that if we're happy with not retaining all 2Tb of nightlies that we're storing at present.

As a "quick fix" I've moved some the v9 nightlies to /root/nightly rather than deleting them which will give us a bit of grace pending any objections to implementing something similar to the above.

The obvious concern would be that we might want to do it separately for each version since there were overlapping major versions in the nightlies.

Do we want to run the deletion of old stuff past the TSC for approval before implementing anything here?

[sxa]

@rvagg Do you know the history of the /mnt/nodejs_org_home_old file system? As the old would imply it doesn't seem to have been touched in almost three years so I guess it's the one referenced in #2162 (comment) which you hadn't got round to discarding? I could clear it and use that as a temporary area while testing out a replacement with mv commands rather than rm`

[mhdawson]

I agree as well that what @targos suggested in #2162 (comment) makes sense.

I think if we reach consensus an FYI to the TSC in advance of implementing would make sense in case there are any concerns.

[sxa]

I've got a prototype script that I've tested which will:

• Retain everything for the last two months (So November and December at the time of writing)
• Remove everything with a date ending in 1 from the previous two calendar years (So 2021/2022 just now) so roughly one every 10 days
• Removing everything except for the nightlies on the first of the month earlier than that

The above will delete 2708 of the 2913 Node.js nightlies given the status at the time of posting.

[rvagg]

if something is still mounted with "old" in its name it can probably be deleted - that smells like me doing a disk resize upgrade and leaving the old one around for "some period of time" to be sure that the sync properly happened; it obviously did because I don't remember when I last did this

+1 to pruning nightlies, I think the only reason we've baulked on scripting it is that people do rely on them for benchmarking and other hackery -- "here's my cool thing but you need to be running Node.js nightly .... which you can download from here". Question for the TSC with input from other folks that might be prone to do this kind of thing.

[rvagg]

or .. we could just up the disk yet again .. or even better, move to an object store model like we've been talking about since almost the beginning of this current layout.

[targos]

The disk is full again.

[mhdawson]

From chat in slack for record:

Michael Dawson (New)
< 1 minute ago
I propose we just delete the
150G ./chakracore-nightly
directory

Michael Dawson (New)
< 1 minute ago
chakracore was abandoned by MS a long time ago and I don't think anybody should be using the old nightlies

Michael Dawson (New)
< 1 minute ago
not sure how long that would give us but it would unblock for now I assume

[mhdawson]

I also wonder about the canaries? Do we need to keep those beyond something like 1 month?

[mhdawson]

On the chakracore side the last one in nightlies was back in 2019.

[targos]

I just deleted the contents of chakracore-nightly. It's not lost anyway because part of the backup in /mnt/nodejs_org_home_old

[richardlau]

It's not lost anyway because part of the backup in /mnt/nodejs_org_home_old

FWIW, all of our releases/nightlies are backed up on the backup server (which is moved, I'll update the inventory shortly).

[sxa]

Good shout on the chakracore nightlies given that we do still have the releases if people want to play with that version.

However I've reinstated the 'last' one for a few of the versions back across in case anyone wants them since it includes a Node v11 and a v12 which isn't in the chakracore-release directory

[sxa]

I also wonder about the canaries? Do we need to keep those beyond something like 1 month?

I'm not too familiar with the canaries - how are they identified and what would the requirements be for those? I was planning to implement the above later today but I can hold off if you think there's a concern here.

[targos]

I think we can treat canaries the same as nightlies.

[sxa]

Pruning now ... This is the script I'm using for it if anyone wishes to enhance it (Stored in /home/dist). In addition to what I said previously it should try to retain the latest one (by the sorting done by ls) . We can set this to run as a cronjob (Needs to run as root since that's what the other of the files are)

prune.sh.gz

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda        3.9T  1.7T  2.0T  46% /home

[sxa]

I'll leave this for any further comments before proceeding. I have NOT updated the index.tab/index.json in those directories (this hasn't been done fo the chakracore-nightlies either) but I goess we should clean them up too with the deletions so I believe the remaining actions are:

• Run this regularly (at cron job time, or at publish time)
• Ensure we clean up the index files

[sxa]

For reference, this is the current size of the different nodejs artifact directories - there are 1385 v8-canary builds in here totalling 829Gb

root@infra-digitalocean-ubuntu1604-x64-1:~# du -sk /home/dist/nodejs/*
2117096	/home/dist/nodejs/chakracore-nightly
842428	/home/dist/nodejs/chakracore-rc
4769080	/home/dist/nodejs/chakracore-release
34484	/home/dist/nodejs/docs
4	/home/dist/nodejs/next-nightly
148733800	/home/dist/nodejs/nightly
150655780	/home/dist/nodejs/rc
326267556	/home/dist/nodejs/release
81597812	/home/dist/nodejs/test
869144472	/home/dist/nodejs/v8-canary
root@infra-digitalocean-ubuntu1604-x64-1:~# 

[mhdawson]

@sxa I assume that for the v8-canary that is before you have pruned them similarly to what you did for nightlies?

[sxa]

@sxa I assume that for the v8-canary that is before you have pruned them similarly to what you did for nightlies?

Correct

[mhdawson]

@sxa then I figure we should just prune the canaries in the same was as for nightlies and see if there is an issue

[mhdawson]

@targos any concerns if we apply the same scripts that we used/are planning to run regularly on the nightlies to prune older builds.

[sxa]

ping @targos - if no objections in the next day or so I'll plan to implement this automation on canaries and nightlies at the start of next week.

[targos]

I have no objections, but note that while I maintain canary builds, I personally almost never use them.

[sxa]

crontab enabled for both nightlies and v8 canaries, so closing this. Scheduled once a month and the logs are written to /home/dist if anyone wants to analyse them in the case of any problems (Since it's only being run once a month they will be available for a while for analysis)