[BUG] npm ignores package.json if package-lock.json or node_modules folder exists, dependency resolution terribly broken

[sseide]

Is there an existing issue for this?

• I have searched the existing issues

This issue exists in the latest npm version

• I am using the latest npm

Current Behavior

"npm i" ignores the package.json file as soon as it finds either a package-lock.json file or a node_modules/ folder

The simplified example below clearly demonstrates how current npm (tested with 8.5.0 but the same with latest 7.x) fails to install all packages needed and nodejs crashing the app eventually with missing dependencies...

When the lock file is not in sync with the requirements defined by the package.json npm does not recognize it and does not install missing requirements. Same is true if no package-lock.json is available at all but an incomplete node_modules/ folder exists - npm just uses the current node_modules/ folder as single point of truth regenerating the lock file from this folders content without checking the package.json wether all dependencies are there.

Current versions are broken in the context of recognizing unmet dependencies. Latest working version i know is latest 6.x. Some earlier 7.x versions worked as well but not the current 7.x and 8.x.

Expected Behavior

"npm i" must check the package.json as the most important source of truth and fullfill all requirements written there (dependencies and sub-dependencies). It MUST not ignore the package json when it finds a package-lock.json or existing node_modules/ folder.

After resolving all dependencies from the package.json file it must check whether the existing lock file fullfills these requirements and installs/updates/downgrades everything else inside the lock file. Only resolutions fitting into the requirements from the package.json shall be used.
Everything else has to be newly resolved from the package versions available.

This was the normal behaviour for all npm versions up to 6.x latest since years (ever). The package.json is the final truth for all versions and dependencies needed and the package-lock.json as well as the node_modules/ folder are updated if they do not met the requirements defined by the package.json.

The current behaviour is just plain wrong and leads to tons of problems whenever someone might fail to commit the updated lock file into the repository or the file is modified for whatever reason...

Steps To Reproduce

1.I created a minimal test project with a dependency having only some sub-dependencies. The packages used do not matter, i used "maskdata" as a smallest possible working example. No JS file needed here. But smae problem exists on bigger projects with hundred s of dependencies:

package.json:

{
  "name": "npm-test",
  "version": "1.0.0",
  "description": "show broken npm dependency resolver",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "me",
  "license": "ISC",
  "dependencies": {
	"maskdata": "1.1.6"
  }
}

2. Run npm i to inilialize lock file and node_modules/ folder. Afterwards modify lock file and remove all references to the "lodash.get" package. JSON must be valid afterwards. This may happen with some files not checked into git repo or something else, why this happens does not matter.

New (modified) package-lock.json is:

{
  "name": "npm-test",
  "version": "1.0.0",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "name": "npm-test",
      "version": "1.0.0",
      "license": "ISC",
      "dependencies": {
        "maskdata": "1.1.6"
      }
    },
    "node_modules/lodash.set": {
      "version": "4.3.2",
      "resolved": "https://registry.npmjs.org/lodash.set/-/lodash.set-4.3.2.tgz",
      "integrity": "sha1-2HV7HagH3eJIFrDWqEvqGnYjCyM="
    },
    "node_modules/maskdata": {
      "version": "1.1.6",
      "resolved": "https://registry.npmjs.org/maskdata/-/maskdata-1.1.6.tgz",
      "integrity": "sha512-37GD3UH4jJxOjVLlX6isbnvpVjSsyz2RwLqcq3uBQ8lZFJw4hn3nnN8QLELjsR1m+msrNWqv9YoV7iOTEUwkpA==",
      "dependencies": {
        "lodash.get": "^4.4.2",
        "lodash.set": "^4.3.2"
      }
    }
  },
  "dependencies": {
    "lodash.set": {
      "version": "4.3.2",
      "resolved": "https://registry.npmjs.org/lodash.set/-/lodash.set-4.3.2.tgz",
      "integrity": "sha1-2HV7HagH3eJIFrDWqEvqGnYjCyM="
    },
    "maskdata": {
      "version": "1.1.6",
      "resolved": "https://registry.npmjs.org/maskdata/-/maskdata-1.1.6.tgz",
      "integrity": "sha512-37GD3UH4jJxOjVLlX6isbnvpVjSsyz2RwLqcq3uBQ8lZFJw4hn3nnN8QLELjsR1m+msrNWqv9YoV7iOTEUwkpA==",
      "requires": {
        "lodash.get": "^4.4.2",
        "lodash.set": "^4.3.2"
      }
    }
  }
}

3. Re-run npm i - the lock file is NOT updated to contain the missing dependency of "lodash.get" and the existing "node_modules/lodash.get/" folder is deleted - npm does not validate if this package-lock.json really fulfilles all requirements as stated in package.json! It does not reinstall needed "lodash.get". Executing my "index.js" app crashes with missing dependencies.

4. trying to recreate lock file does not work if "node_modules/" folder is present.
   After step 3 the node_modules folder does not contain the needed dependency of "lodash.get" anymore. Now delete the "package-lock.json" file and rerun "npm i" - NPM does not recreate the dependency list from the current package.json but only writes the wrong(!) content of the existing "node_modules/" folder into a new package-lock.json file. It does not reinstall needed "lodash.get". Executing my "index.js" app crashes with missing dependencies.

Environment

• npm: 8.5.0
• Node.js: 14.19.0
• OS Name: Debian 11 Bullseye
• System Model Name: amd64
• npm config:

; "user" config from /home/<user>/.npmrc

; node bin location = /usr/bin/node
; cwd = /home/<user>/<dir>/npm-test
; HOME = /home/<user>
; Run `npm config ls -l` to show all defaults.

Nodejs installed from official nodesource deb package and included npm@6 updated via npm i -g npm@latest to npm@8.5.0.

[sseide]

This issue might be related to #3376, but it has definitly another much broader scope

[ljharb]

Why would you expect to manually modify the lockfile at all, and then have things be in a reasonable state? That file’s only for npm to modify.

[ljharb]

(separately; I’d try installing node properly, meaning, not from apt - just to rule out that’s what’s causing you problems)

[sseide]

as i wrote clearly - there are dozends of ways why the package.json and the lock file might get out of sync - the example to modify it by hand was just to make this problem 100% reproducible and understandable.

And even deleting the lock file when its borked/out of sync to generate a complete new one does not work when there is an incomplete node_modules/ folder!

And all npm 6.x can properly handle these cases and do not break an entire installation. They "just work" (tm).

[fritzy]

The package-lock is not supposed to be modified by users. The only way for the two files to be out of sync would be to modify the package.json, in which case, it would be evaluating different packages (or versions). Please re-open this issue if it can be reproduced without directly editing the package-lock or node_modules.

[sseide]

@fritzy this is only an answer to point 3

But point 4 is ignored here - when there is no package-lock.json at all "npm" creates a wrong new lock file when a node_modules/ folder exists.

[jakobsa]

Did I get that right, package-lock.json is generated from the contents of /node_modules without regard to package.json? @fritzy can you confirm this?

[darcyclarke]

Reopening based on the last point (ie. #4) which seems to be still an issue