Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

src: move package_json_reader cache to c++ #50322

Closed
wants to merge 4 commits into from
Closed

src: move package_json_reader cache to c++ #50322

wants to merge 4 commits into from

Conversation

anonrig
Copy link
Member

@anonrig anonrig commented Oct 21, 2023 •
edited
Loading

Co-authored by @lemire

Local Benchmark

My local benchmarks says that running vite --version is 5% faster

❯ hyperfine 'node ../sveltejs-realworld/node_modules/vite/dist/node/cli.js --version' 'out/Release/node ../sveltejs-realworld/node_modules/vite/dist/node/cli.js --version' -w 10
Benchmark 1: node ../sveltejs-realworld/node_modules/vite/dist/node/cli.js --version
  Time (mean ± σ):     101.4 ms ±   0.6 ms    [User: 96.6 ms, System: 10.8 ms]
  Range (min … max):   100.3 ms … 102.5 ms    28 runs

Benchmark 2: out/Release/node ../sveltejs-realworld/node_modules/vite/dist/node/cli.js --version
  Time (mean ± σ):      96.3 ms ±   0.5 ms    [User: 90.9 ms, System: 10.1 ms]
  Range (min … max):    95.6 ms …  98.1 ms    30 runs

Summary
  out/Release/node ../sveltejs-realworld/node_modules/vite/dist/node/cli.js --version ran
    1.05 ± 0.01 times faster than node ../sveltejs-realworld/node_modules/vite/dist/node/cli.js --version

TODOs

  • Move modules functionality to node_modules.cc
    • internalModuleReadJSON
  • Move caching of package_json_reader to C++
    • Make node_modules.cc snapshottable
  • Update run_main.js to just avoid constructing the whole object for the entry point.
    • We could even call containsModuleSyntax from C++
  • Move getPackageScopeConfig from package_config.js to C++ side, and cache it on node_modules.cc
  • Replace all packageJSONReader calls that looks for specific attribute to have equivalent C++ functions
  • Investigate if imports & exports JSONParse needs to be cached on JS

Caveats

  • ERR_INVALID_PACKAGE_CONFIG does not include JSON.Parse() error message anymore. This mainly because we're using simdjson to avoid JS allocation while parsing for performance reasons.
  • exports/imports without string, array or object value will now throw ERR_INVALID_PACKAGE_CONFIG instead of returning an invalid value.

Benchmarks

@nodejs-github-bot
Copy link
Collaborator

Review requested:

  • @nodejs/gyp
  • @nodejs/loaders
  • @nodejs/startup

@nodejs-github-bot nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Oct 21, 2023
@anonrig anonrig force-pushed the package-json-cache branch 8 times, most recently from e462554 to ba0be9b Compare October 22, 2023 00:34
@GeoffreyBooth GeoffreyBooth mentioned this pull request Oct 22, 2023
Draft
@anonrig anonrig force-pushed the package-json-cache branch 4 times, most recently from 5158e35 to 2690c64 Compare October 23, 2023 19:16
joyeecheung
joyeecheung reviewed Oct 24, 2023
View reviewed changes
src/node_modules.cc Outdated Show resolved Hide resolved
@anonrig anonrig force-pushed the package-json-cache branch 12 times, most recently from 2bfbcdb to 8309e95 Compare October 25, 2023 19:31
@RafaelGSS RafaelGSS added the semver-minor PRs that contain new features and should be released in the next minor version. label Dec 15, 2023
@RafaelGSS
Copy link
Member

Applying semver-minor PRs that contain new features and should be released in the next minor version. because we're adding a new dependency to Node.js (0dd53da)

RafaelGSS added a commit that referenced this pull request Dec 15, 2023
@RafaelGSS
2023-12-19, Version 21.5.0 (Current)
a8f84e3 
Notable changes:

deps:
  * (SEMVER-MINOR) add simdjson (Yagiz Nizipli) #50322
doc:
  * deprecate hash constructor (Marco Ippolito) #51077
  * deprecate `dirent.path` (Antoine du Hamel) #51020
module:
  * merge config with `package_json_reader` (Yagiz Nizipli) #50322
src:
  * move package resolver to c++ (Yagiz Nizipli) #50322

PR-URL: TBD
@RafaelGSS RafaelGSS mentioned this pull request Dec 15, 2023
Merged
RafaelGSS added a commit that referenced this pull request Dec 15, 2023
@RafaelGSS
2023-12-19, Version 21.5.0 (Current)
8654a1b 
Notable changes:

deps:
  * (SEMVER-MINOR) add simdjson (Yagiz Nizipli) #50322
doc:
  * deprecate hash constructor (Marco Ippolito) #51077
  * deprecate `dirent.path` (Antoine du Hamel) #51020
module:
  * merge config with `package_json_reader` (Yagiz Nizipli) #50322
src:
  * move package resolver to c++ (Yagiz Nizipli) #50322

PR-URL: #51166
RafaelGSS added a commit that referenced this pull request Dec 15, 2023
@RafaelGSS
2023-12-19, Version 21.5.0 (Current)
34ddc71 
Notable changes:

deps:
  * (SEMVER-MINOR) add simdjson (Yagiz Nizipli) #50322
doc:
  * deprecate hash constructor (Marco Ippolito) #51077
  * deprecate `dirent.path` (Antoine du Hamel) #51020
module:
  * merge config with `package_json_reader` (Yagiz Nizipli) #50322
src:
  * move package resolver to c++ (Yagiz Nizipli) #50322

PR-URL: #51166
@joyeecheung joyeecheung mentioned this pull request Dec 18, 2023
Merged
RafaelGSS added a commit that referenced this pull request Dec 19, 2023
@RafaelGSS
2023-12-19, Version 21.5.0 (Current)
8b690a1 
Notable changes:

deps:
  * (SEMVER-MINOR) add simdjson (Yagiz Nizipli) #50322
doc:
  * deprecate hash constructor (Marco Ippolito) #51077
  * deprecate `dirent.path` (Antoine du Hamel) #51020
module:
  * merge config with `package_json_reader` (Yagiz Nizipli) #50322
src:
  * move package resolver to c++ (Yagiz Nizipli) #50322

PR-URL: #51166
@marco-ippolito marco-ippolito added the backport-requested-v20.x PRs awaiting manual backport to the v20.x-staging branch. label Apr 29, 2024
@marco-ippolito
Copy link
Member

marco-ippolito commented Apr 29, 2024 •
edited
Loading

@anonrig I would like to land this on v20 but its not landing cleanly, can you backport?

This was referenced May 1, 2024
Merged
Merged
@marco-ippolito marco-ippolito added backport-requested-v20.x PRs awaiting manual backport to the v20.x-staging branch. and removed backport-requested-v20.x PRs awaiting manual backport to the v20.x-staging branch. labels May 1, 2024
@marco-ippolito marco-ippolito removed the baking-for-lts PRs that need to wait before landing in a LTS release. label Jun 17, 2024
@marco-ippolito marco-ippolito mentioned this pull request Jun 17, 2024
Merged
4 tasks
This was referenced Jun 17, 2024
Closed
Closed
@anonrig anonrig mentioned this pull request Jun 18, 2024
Open
anonrig added a commit to anonrig/node that referenced this pull request Jun 18, 2024
@anonrig
deps: add simdjson
1ec9b3a 
Backport-PR-URL: nodejs#50322
PR-URL: nodejs#50322
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
anonrig added a commit to anonrig/node that referenced this pull request Jun 18, 2024
@anonrig
src: move package resolver to c++
803006e 
Co-authored-by: Daniel Lemire <daniel@lemire.me>
Backport-PR-URL: nodejs#50322
PR-URL: nodejs#50322
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>

# Conflicts:
#	lib/internal/modules/package_json_reader.js
#	src/node_file.cc
#	test/parallel/test-module-binding.js
anonrig added a commit to anonrig/node that referenced this pull request Jun 18, 2024
@anonrig
module: merge config with package_json_reader
ff40d15 
PR-URL: nodejs#50322
Backport-PR-URL: nodejs#50322
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
@marco-ippolito marco-ippolito added backport-open-v20.x Indicate that the PR has an open backport and removed backport-requested-v20.x PRs awaiting manual backport to the v20.x-staging branch. labels Jun 19, 2024
anonrig added a commit to anonrig/node that referenced this pull request Jun 27, 2024
@anonrig
deps: add simdjson
06bbabc 
Backport-PR-URL: nodejs#50322
PR-URL: nodejs#50322
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
anonrig added a commit to anonrig/node that referenced this pull request Jun 27, 2024
@anonrig
src: move package resolver to c++
0cfb97c 
Co-authored-by: Daniel Lemire <daniel@lemire.me>
Backport-PR-URL: nodejs#50322
PR-URL: nodejs#50322
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>

# Conflicts:
#	lib/internal/modules/package_json_reader.js
#	src/node_file.cc
#	test/parallel/test-module-binding.js
anonrig added a commit to anonrig/node that referenced this pull request Jun 27, 2024
@anonrig
module: merge config with package_json_reader
b447a68 
PR-URL: nodejs#50322
Backport-PR-URL: nodejs#50322
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
anonrig added a commit to anonrig/node that referenced this pull request Jun 27, 2024
@anonrig
deps: add simdjson
7f4d716 
PR-URL: nodejs#50322
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
anonrig added a commit to anonrig/node that referenced this pull request Jun 27, 2024
@anonrig
src: move package resolver to c++
19640ce 
Co-authored-by: Daniel Lemire <daniel@lemire.me>
PR-URL: nodejs#50322
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>

# Conflicts:
#	lib/internal/modules/package_json_reader.js
#	src/node_file.cc
#	test/parallel/test-module-binding.js
anonrig added a commit to anonrig/node that referenced this pull request Jun 27, 2024
@anonrig
module: merge config with package_json_reader
af801aa 
PR-URL: nodejs#50322
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lemire lemire lemire left review comments

@JonasBa JonasBa JonasBa left review comments

@ljharb ljharb ljharb approved these changes

@mcollina mcollina mcollina approved these changes

@jasnell jasnell jasnell approved these changes

@GeoffreyBooth GeoffreyBooth GeoffreyBooth approved these changes

@JakobJingleheimer JakobJingleheimer JakobJingleheimer approved these changes

@aduh95 aduh95 aduh95 approved these changes

@joyeecheung joyeecheung Awaiting requested review from joyeecheung

Assignees
No one assigned
Labels
backport-open-v20.x Indicate that the PR has an open backport c++ Issues and PRs that require attention from people who are familiar with C++. commit-queue-rebase Add this label to allow the Commit Queue to land a PR in several commits. lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. notable-change PRs with changes that should be highlighted in changelogs. semver-minor PRs that contain new features and should be released in the next minor version.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.