Node >= 16.18.0 breaks npm ci

[ccl2of4]

Version

16.18.0, 16.19.0, 18.3.0

Platform

Linux ip-10-94-32-11 5.15.0-1026-aws #30-Ubuntu SMP Wed Nov 23 14:15:21 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Subsystem

http

What steps will reproduce the bug?

Execute npm ci

How often does it reproduce? Is there a required condition?

I am able to reproduce the issue every single time.

What is the expected behavior?

npm ci should execute successfully.

What do you see instead?

$ npm ci
npm ERR! code EPIPE
npm ERR! syscall write
npm ERR! errno -32
npm ERR! write EPIPE

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/ubuntu/.npm/_logs/2023-01-12T18_17_28_239Z-debug-0.log

Additional information

I originally left a comment on closed issue #44998. In fear that it would never be seen, I've opened a new issue here.

I am currently experiencing this issue on any version of node after 16.18.0. I have tried the latest versions from both the 16 and 18 release channels.

Using git bisect and building node from source, I have narrowed down the problematic commit to

5f406a3a53: http: reuse socket only when it is drained (3 months ago) <ywave620>. (associated pull request #43902)

When I run npm ci on any branch which includes this commit, I encounter this EPIPE error. If I revert the commit and rebuild node, I can see from the logs that the EPIPE error is still emitted, but it is handled properly.

Unfortunately, I'm really not sure what exactly it is about my environment that is causing this EPIPE error, as it doesn't seem like it's affecting very many users at all. I experience the issue running from AWS EC2 or ECS connected to my company's private network using a Transit Gateway. So, there must be something special about my network which triggers this issue.

In the meantime, the best I can currently do is attach a set of detailed logs.

For each of these sets of logs, I ran DEBUG='*' NODE_DEBUG='*' npm ci:

• v16.17.1 (Working version)
• v16.18.0 (First released broken version)
• v16.19.0 (Latest release on v16 LTS branch)

To quickly find the error in these logs, just search for the string 'EPIPE'. In summary, it looks like in v16.17.1, EPIPE is emitted but handled; in later versions EPIPE is emitted but is unhandled.

Furthermore, in reading through the code on my own, I am seeing a significant quantity of comments labeled TODO(ronag): that mention edge cases for error handling. I think may be the unlucky soul who is hitting these edge cases. I think, in particular, this TODO is the one that is doing me in (judging from my log output, as well as some debug statements I've added during my own test runs).

Please let me know if there's any further information I can provide for troubleshooting. As it seems like it won't be possible for any Node developers to reproduce this issue, I am more than happy to rebuild and test Node myself with any recommended changes to see if they fix my issue (I can even add additional debug statements and rerun npm ci if that helps in any way).

Thank you so much for any help!

[Trott]

@mcollina @ronag @ShogunPanda

[mcollina]

Thanks for reporting!

Can you provide steps to reproduce? We often need a reproducible example, e.g. some code that allows someone else to recreate your problem by just copying and pasting it. If it involves more than a couple of different file, create a new repository on GitHub and add a link to that.

[doteric]

Hello @mcollina
I got the same mentioned above error when installing multiple different packages at once. For example:

npm install --global semantic-release@21.0.2 @semantic-release/changelog@6.0.3 @semantic-release/commit-analyzer@9.0.2 @semantic-release/gitlab@12.0.1

Keep in mind that the error does not happen always, but from time to time.
We use a custom NPM repository with an url like so:

# "xxx" as placeholders to hidden string content"
https://nexus.xxx.xxx.xxx.xxx:8443/repository/npm_pull/

Additionally when disabling audits the issue did not appear.
That concludes to the assumption that NodeNPM might be receiving a too heavy payload from our nexus that it can't handle it and simply crashes with a EPIPE code. Or maybe the response is somehow invalid.

BUT I've tried updating the npm version from npm v8 to v9 and it fixed the issue.
I was using node 18.12.1 which contains v8 by default so bumping to 18.16.0 that contains npm@9 by default, fixes the issue. The fix is probably somewhere in NPM between the v8.x and v9.x, but I didn't bother to check where exactly. Hope it helps everyone else.

[mcollina]

@nodejs/releasers @nodejs/lts I think we should rever #43902 in v16.

[solidaris-antoinejacquesdedixmude]

Hi,
Since I upgraded my nodejs docker image from 18.7.0 to 18.18.2, I experience the same issue as @ccl2of4. Here is the output of DEBUG='*' NODE_DEBUG='*' npm ci:
npm_epipe_error_debug_cli.log

Unfortunately, I can't provide steps to reproduce this problem as it randomly occurs. It could be that this issue is triggered by something in our local network environment. Like, I notice that @ccl2of4, @doteric and I use Sonatype Nexus Repository.

[nicdun]

same here on updating NX workspace angular from 15 to 17 in jenkins pipeline - problematic dependency is upgrade @angular-devkit/build-angular to 17.0.0. Install works with Version 15.1.5 and 16.2.10.

2023-11-25T07:17:09.012Z] npm timing auditReport:getReport Completed in 6231ms
[2023-11-25T07:17:09.012Z] npm timing reify:audit Completed in 6232ms
[2023-11-25T07:17:10.623Z] npm verb stack Error: write EPIPE
[2023-11-25T07:17:10.623Z] npm verb stack     at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)
[2023-11-25T07:17:10.623Z] npm verb cwd /misc/jenkins/workspace/<branch-name>
[2023-11-25T07:17:10.623Z] npm verb Linux 4.18.0-477.27.1.el8_8.x86_64
[2023-11-25T07:17:10.623Z] npm verb node v18.12.0
[2023-11-25T07:17:10.623Z] npm verb npm  v8.19.2
[2023-11-25T07:17:10.624Z] npm ERR! code EPIPE
[2023-11-25T07:17:10.624Z] npm ERR! syscall write
[2023-11-25T07:17:10.624Z] npm ERR! errno -32
[2023-11-25T07:17:10.624Z] npm ERR! write EPIPE
[2023-11-25T07:17:10.624Z] npm verb exit -32
[2023-11-25T07:17:10.624Z] npm timing npm Completed in 12273ms