[BUG] NPM 'installing' devDependencies of my dependency when there is script "build"

[dhpagani]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

NPM is trying to install the devDependencies from my dependency just because there is a script in package.json named "build".
It didn't put this devDependency in node_modules and I only notice that because my scenario node-sass is present and fail to install.

In a scenario of pkgA and pkgB
pkgA: https://github.com/dhpagani/npm-bug-pkga/
Create a new project pkgB and add pkgA as a git dependency. Try to run npm install on pkgb and, (if your env is not prepared to install node-sass) you will see an error of node-sass postinstall script.

Expected Behavior

Running npm install in pkgb, it didn't try to install node-sass (as a transitive devDependencies) of my library.

Steps To Reproduce

Create a git dependency:

{
  "name": "pkga",
  "version": "1.0.0",
  "private": true,
  "scripts": {
    "build": "echo 'Example'"
  },
  "devDependencies": {
    "grunt-sass": "^1.1.0"
  }
}

If I use this dependency in any project, like:

{
  "name": "pkgb",
  "version": "1.0.0",
  "private": true,
  "dependencies": {
    "pkga": "git+ssh://git@github.com/dhpagani/npm-bug-pkga#main"
  }
}

and run npm install on pkgb, I received a log of postinstall script from node-sass failing (which is a transitive dependency of grunt-sass). If it didn't fail, I wouldn't notice this behavior.

If I change the name of the "build" script (to something like build2), run install again, it won't try to install node-sass.

In any scenario, like removing grunt-sass and put other dependency, the script "build" didn't run.

Environment

• OS: Ubuntu on WSL2 or macOs Catalina 10.15.7
• Node: 16.11.1 (16.10 also happens)
• npm: 8.0.0 (but on 7 also happens)

Maybe related to #3800

[leobel]

Hi @dhpagani any luck with that? I've the same issue

[dhpagani]

@leobel about this, not yet... I've not updated my projects to use the newest version yet, but I'm considering to change to another package manager, like yarn ou pnpm.

[paulbrimicombe]

We also have this issue. We publish our private packages to GitHub to make dev setup easier (no need to set up auth for both GitHub and some private NPM registry). We publish the transpiled code to a different branch in our repos and tag with the appropriate semver tag. We deliberately don't have a prepare script to prevent npm trying to install of the dev dependencies and repeat the transpilation (of code that's not even in that branch of the repository). Unfortunately with NPM 8 we see our dev dependencies getting installed if we have a build script in our package.json.

The problem appears to be in pacote. This is the line that causes the problem:

https://github.com/npm/pacote/blob/2ddf67f7c4e084ffec315f94e30bb24f944403e3/lib/git.js#L153

It look as if any of the following scripts will cause installation of dev dependencies:

postinstall
build
preinstall
install
prepare

The documentation only seems to mentions the prepare script: https://docs.npmjs.com/cli/v8/using-npm/scripts#life-cycle-scripts

Can anyone from NPM please tell us whether this is a deliberate change in behaviour (and should be documented) or whether this is a regression?

[9point6]

+1

[scooper91]

We've just come across this when upgrading from Node.js 14 to 16. For now, we've worked around it by downgrading npm to version 6.

Our experience is pretty similar to @dhpagani - we're installing a dependency which has a build script. This package has a dev dependency on a package which fails to install.

Are there any updates on this?

Thanks!

[nlf]

this is expected behavior, if you have a git dependency with a build script we operate on the assumption that we need to build the dependency for you. this causes npm to clone the repository to a temp directory, install all of its dependencies, run the build script, and then package and install the result.

if you've already pre-built your package and the build artifacts are stored in the git repository, the correct thing to do here is rename your build script to something else.

[dhpagani]

I got your point, but I didn't find in the docs about that assumption. I've check in the scripts lifecycle (https://docs.npmjs.com/cli/v8/using-npm/scripts) and in the reference of git dependencies(https://docs.npmjs.com/cli/v8/configuring-npm/package-json#git-urls-as-dependencies)

In my opinion, this assumptions (and others) must be better explained or referenced. What do you think?

[nlf]

i do agree it needs more documentation. i'm making some other edits today and i'll clarify that while i'm in there. good suggestion!

[paulbrimicombe]

this is expected behavior, if you have a git dependency with a build script we operate on the assumption that we need to build the dependency for you. this causes npm to clone the repository to a temp directory, install all of its dependencies, run the build script, and then package and install the result.

@nlf this opinion would carry much more weight if npm actually ran the build script. You can test this by running:

npm install --verbose paulbrimicombe/npm-git-deps-testing#semver:^4.0.0

This package has exit 1 in its build script (See https://github.com/paulbrimicombe/npm-git-deps-testing/blob/7f9fdcfea16351aa7d13f7f1e7691f4f4f0e6e52/package.json#L8). This script is not run by npm during installation. Putting the same thing in a prepare script does cause an error in the installation (try installing version 5.0.0 of that package).

Do I take it from your comment that the build script might be run in the future during installation for git dependencies? This is likely to be a breaking change for some package maintainers if build is only supposed to be run during development and not during distribution.

if you've already pre-built your package and the build artifacts are stored in the git repository, the correct thing to do here is rename your build script to something else.

How do package owners know that they have to do this if there is absolutely nothing in the documentation to let them know of this change? The build script is not mentioned anywhere in the lifecycle scripts documentation for npm in any version that I can find.

[do0g]

if you've already pre-built your package and the build artifacts are stored in the git repository, the correct thing to do here is rename your build script to something else.

There are projects such as ramda that publish built and minified artefacts alongside their source code. Executing the project's build script or installing development dependencies automatically could break on install.

It seems to me that NPM has always been explicit about the purpose and position in lifecycles of script hooks, and the installation of development dependencies due to any number of pre or post install, build and even prepare scripts will be unexpected behaviour to many. For this reason it should be worth considering whether the current behaviour is desired as it is both implicit and default.

Could you consider providing an opt-in explicit script hook as you do for all other behaviour? This would allow authors who did want the consumer of a package to build their project and would not break the projects of those who do not.

[ddfridley]

I see this was closed recently but I want to contribute. I have a project (github.com/EnCiv/undebate-ssp) that includes other repos (github.com/EnCiv/undebate, github.com/EnCiv/civil-server, github.com/EnCiv/civil-client). Each project includes devDependencies like storybook, jest, lint. These cause a tremendous load on the build process and time, When I set NPM_CONFIG_PRODUCTION=true then it doesn't try to load the devDependencies of .undebate-ssp but it seems to still load and build the devDependencies of the child repos. Beside the tremendious amount of time this takes, when I try to push the project to heroku - it stalls. I think that npm has fired off so many things in parallel that it just locks up.

I would like to point out that there are buildDependencies (like babel) and there are testDependencies (jest) the testDependencies of repos don't need to be loaded or built, expcept in the root - top level build (undebate-ssp).

I've seen a lot of discussion about how devDependencies should be enough. But I want to point out that all the machine cycles and network bandwidth spent on testDependencies is wasted electricity and people's time. And in some cases the project won't build.

I wish that:

1. there was a way to limit the number of process NPM can fork in total on a machine(crude work around)
2. I wish there could be a testDependencies property (or call it what you want) that would not be loaded or built for dependent repos.
3. I wish that npm audit would identify if a security issue applies to a dependency, a devDependency, or a testDependency. of the root. Some issues are more urgent than others.

Please let me know if I should start a new issue for this, or post it somewhere else. Thanks.