[BUG] npm install --production fails to install dependencies overlapping with devDependencies

[fertolg]

Is there an existing issue for this?

• I have searched the existing issues

This issue exists in the latest npm version

• I am using the latest npm

Current Behavior

Create this package in an empty directory:

{
  "name": "missing-fs-extra",
  "private": true,
  "description": "A simple package to reproduce missing dependencies when installing with the --production flag",
  "version": "0.1.0",
  "dependencies": {
    "gatsby": "^4.4.0"
  },
  "devDependencies": {
    "@storybook/react": "^6.4.9"
  },
  "license": "MIT",
  "scripts": {
    "build": "gatsby build"
  }
}

Then attempt to run:

npm install --production && npm run build

Notice how fs-extra which is a direct dependency of gatsby is missing from the installed node-modules. fs-extra is also a direct dependency (with a mismatched version) of @storybook/react, however @storybook/react is a devDependency so my understanding is that it should be ignored completely when installing with the --production flag.

Expected Behavior

fs-extra should be installed in node_modules because it is a direct dependency of gatsby which is a direct dependency of the root package.

Steps To Reproduce

1. Using npm version 8.3.0
2. With this package:

{
  "name": "missing-fs-extra",
  "private": true,
  "description": "A simple package to reproduce missing dependencies when installing with the --production flag",
  "version": "0.1.0",
  "dependencies": {
    "gatsby": "^4.4.0"
  },
  "devDependencies": {
    "@storybook/react": "^6.4.9"
  },
  "license": "MIT",
  "scripts": {
    "build": "gatsby build"
  }
}

3. Run npm install --production
4. Notice the fs-extra module is missing from the installed modules, and when gatsby runs (npm run build) it fails to resolve the dependency.

Environment

• npm: 8.3.0
• Node: v14.18.2
• OS: MacOS Catalina 10.15.7 (19H1615)
• platform: MacbookPro
• npm config:

; node bin location = /Users/fernandotoledo/.nvm/versions/node/v14.18.2/bin/node
; cwd = /Users/fernandotoledo/Desktop/test
; HOME = /Users/fernandotoledo
; Run `npm config ls -l` to show all defaults.

[wraithgar]

I can not duplicate this in npm@8.5.4

~/D/n/scratch $ npm pkg get dependencies devDependencies
{
  "dependencies": {
    "gatsby": "^4.10.0"
  },
  "devDependencies": {
    "@storybook/react": "^6.4.19"
  }
}
~/D/n/scratch $ npm i --production

changed 2 packages, and audited 1516 packages in 4s

19 vulnerabilities (6 moderate, 13 high)

To address issues that do not require attention, run:
  npm audit fix

To address all issues (including breaking changes), run:
  npm audit fix --force

Run `npm audit` for details.
~/D/n/scratch $ npm ls fs-extra
scratch@1.0.0 /Users/wraithgar/Development/npm/scratch
└─┬ gatsby@4.10.0
  ├── fs-extra@10.0.1
  ├─┬ gatsby-cli@4.10.0
  │ └── fs-extra@10.0.1 deduped
  ├─┬ gatsby-core-utils@3.10.0
  │ └── fs-extra@10.0.1 deduped
  └─┬ gatsby-telemetry@3.10.0
    └── fs-extra@10.0.1 deduped

~/D/n/scratch $ grep version node_modules/fs-extra/package.json 
  "version": "10.0.1",

[wraithgar]

After clearing my node_modules and package-lock and trying again I was able to get into that state.

Note that it does exist on disk, but gatsby can not require it for some reason

~/D/n/scratch $ npm ls fs-extra
scratch@1.0.0 /Users/wraithgar/Development/npm/scratch
└─┬ gatsby@4.10.0
  ├── fs-extra@10.0.1
  ├─┬ gatsby-cli@4.10.0
  │ └── fs-extra@10.0.1
  ├─┬ gatsby-core-utils@3.10.0
  │ └── fs-extra@10.0.1
  └─┬ gatsby-telemetry@3.10.0
    └── fs-extra@10.0.1

[wraithgar]

Found it. This is a bug in gatsby-plugin-utils. It is requiring a package that it hasn't declared in its manifest.

~/D/n/scratch $ cat node_modules/gatsby-plugin-utils/package.json 
{
  "name": "gatsby-plugin-utils",
  "version": "3.4.0",
  "description": "Gatsby utils that help creating plugins",
[...]
  "dependencies": {
    "@babel/runtime": "^7.15.4",
    "gatsby-core-utils": "^3.10.0",
    "gatsby-sharp": "^0.4.0",
    "graphql-compose": "^9.0.7",
    "import-from": "^4.0.0",
    "joi": "^17.4.2",
    "mime": "^3.0.0"
  }
[...]
}

It was only working by coincidence before, relying on fs-extra to be a hoisted dep, which is not guaranteed.

[fertolg]

Thank you for digging into this @wraithgar, apologies if I wasted your time. I'll open an issue on gatsby-plugin-utils.

[wraithgar]

No worries, we'd prefer folks open a ticket just in cast it is a bug w/ npm.