Fix out-of-memory error when building hundreds of pages #2969

szimek · 2017-11-19T22:22:17Z

The latest version of Gatsby crashes when trying to build ~500 pages using filesystem source plugin and json transformer plugin. The same error has been reported in #2796. The issue was most likely introduced in 7dd5f3b, though I haven't been able to figure out where exactly. However, the time to finish run graphql queries step is significantly longer in 1.9.33 than in 1.9.32 (for a number of pages that doesn't cause 1.9.33 to crash). It looks like it's caused by queryRunner function, which is called for every page, but if there are too many pages, it never gets resolved.

This is probably more of a workaround rather than a real fix, because whatever Gatsby is doing there, it probably shouldn't use up to 1.5GB memory to process 500 JSON files, even if it's processing them all in parallel. At least now the build finishes! :) Also, it takes ~16-20s to run the run graphql queries for these 500 pages in 1.9.118 with this fix, while in 1.9.32 it takes ~2s. However, as I mentioned earlier, it's (mostly) caused by changes introduced in 1.9.33, not by this fix.

gatsbybot · 2017-11-19T22:26:37Z

Deploy preview ready!

Built with commit e51c389

https://deploy-preview-2969--gatsbygram.netlify.com

gatsbybot · 2017-11-19T22:29:59Z

Deploy preview ready!

Built with commit e51c389

https://deploy-preview-2969--using-drupal.netlify.com

KyleAMathews · 2017-11-21T16:58:09Z

So why would 7dd5f3b slow down running queries? Seems like that'd just slow down inferring the schema.

This change seems pretty reasonable except I'd prefer a change that limited the number of queries we're running in parallel to something like 10-15 (testing would help) as that would still reduce peak memory usage a lot while not slowing things down as much.

KyleAMathews · 2017-11-21T17:00:09Z

I'm working on adding perf testing fairly soon. It'll measure peak & on-going memory usage + build times. That'll make it easier to work on PRs like this and ensure we're actually improving things.

KyleAMathews · 2017-11-21T17:01:32Z

Perhaps try Bluebird's concurrency limit http://bluebirdjs.com/docs/api/promise.map.html#map-option-concurrency

You can also record peak memory using https://stackoverflow.com/questions/774556/peak-memory-usage-of-a-linux-unix-process

szimek · 2017-11-21T17:12:21Z

Without this fix, the peak is 1.5GB, which is the node.js memory limit :) I'll try out different concurrency settings and report back the times and memory usage.

szimek · 2017-11-21T18:41:18Z

@KyleAMathews Here are some results from my Macbook Pro with 2.6GHz dual-core i5 CPU for ~460 pages with json and remark plugins. Obviously, these results are far from "scientific" and could be completely different if someone has e.g. 10 pages, or different queries.

Promise.mapSeries: 450MB and 55s


$ rm -rf .cache/ public && /usr/bin/time -l ../../node_modules/.bin/gatsby build
success delete html files from previous builds — 0.064 s
success open and validate gatsby-config.js — 0.009 s
success copy gatsby files — 0.032 s
success onPreBootstrap — 0.025 s
success source and transform nodes — 2.393 s
success building schema — 1.034 s
success createLayouts — 0.011 s
success createPages — 1.023 s
success createPagesStatefully — 0.122 s
success onPreExtractQueries — 0.002 s
success update schema — 0.872 s
success extract queries from components — 0.071 s
success run graphql queries — 21.027 s
success write out page data — 0.021 s
success write out redirect data — 0.001 s
success onPostBootstrap — 0.000 s
info bootstrap finished - 28.866 s
success Building CSS — 4.601 s

success Building production JavaScript bundles — 14.138 s

success Building static HTML for pages — 7.865 s

info Done building in 55.482 sec

55.54 real        53.44 user         2.62 sys

457236480  maximum resident set size

Promise.map with concurrency 2: 446MB and 51s


$ rm -rf .cache/ public && /usr/bin/time -l ../../node_modules/.bin/gatsby build
success delete html files from previous builds — 0.072 s
success open and validate gatsby-config.js — 0.006 s
success copy gatsby files — 0.032 s
success onPreBootstrap — 0.016 s
success source and transform nodes — 2.409 s
success building schema — 0.984 s
success createLayouts — 0.010 s
success createPages — 1.114 s
success createPagesStatefully — 0.024 s
success onPreExtractQueries — 0.001 s
success update schema — 0.908 s
success extract queries from components — 0.128 s
success run graphql queries — 18.617 s
success write out page data — 0.026 s
success write out redirect data — 0.001 s
success onPostBootstrap — 0.001 s
info bootstrap finished - 26.514 s
success Building CSS — 4.156 s

success Building production JavaScript bundles — 14.308 s

success Building static HTML for pages — 5.937 s

info Done building in 50.93 sec

50.99 real        53.47 user         2.53 sys

446201856  maximum resident set size

Promise.map with concurrency 3: 465MB and 52s


$ rm -rf .cache/ public && /usr/bin/time -l ../../node_modules/.bin/gatsby build
success delete html files from previous builds — 0.069 s
success open and validate gatsby-config.js — 0.008 s
success copy gatsby files — 0.042 s
success onPreBootstrap — 0.016 s
success source and transform nodes — 2.269 s
success building schema — 0.919 s
success createLayouts — 0.021 s
success createPages — 1.059 s
success createPagesStatefully — 0.075 s
success onPreExtractQueries — 0.001 s
success update schema — 0.976 s
success extract queries from components — 0.104 s
success run graphql queries — 18.828 s
success write out page data — 0.016 s
success write out redirect data — 0.001 s
success onPostBootstrap — 0.003 s
info bootstrap finished - 26.569 s
success Building CSS — 4.296 s

success Building production JavaScript bundles — 13.800 s

success Building static HTML for pages — 7.057 s

info Done building in 51.759 sec

51.86 real        52.82 user         2.64 sys

465108992  maximum resident set size

Promise.map with concurrency 5: 440MB and 54s


$ rm -rf .cache/ public && /usr/bin/time -l ../../node_modules/.bin/gatsby build
success delete html files from previous builds — 0.065 s
success open and validate gatsby-config.js — 0.010 s
success copy gatsby files — 0.033 s
success onPreBootstrap — 0.014 s
success source and transform nodes — 2.643 s
success building schema — 0.973 s
success createLayouts — 0.009 s
success createPages — 1.117 s
success createPagesStatefully — 0.022 s
success onPreExtractQueries — 0.028 s
success update schema — 0.869 s
success extract queries from components — 0.123 s
success run graphql queries — 20.753 s
success write out page data — 0.028 s
success write out redirect data — 0.001 s
success onPostBootstrap — 0.001 s
info bootstrap finished - 28.863 s
success Building CSS — 5.334 s

success Building production JavaScript bundles — 14.433 s

success Building static HTML for pages — 6.644 s

info Done building in 55.285 sec

55.37 real        57.45 user         2.68 sys

440877056  maximum resident set size

Promise.map with concurrency 10: 465MB and 58s


$ rm -rf .cache/ public && /usr/bin/time -l ../../node_modules/.bin/gatsby build
success delete html files from previous builds — 0.070 s
success open and validate gatsby-config.js — 0.012 s
success copy gatsby files — 0.029 s
success onPreBootstrap — 0.021 s
success source and transform nodes — 2.520 s
success building schema — 0.900 s
success createLayouts — 0.009 s
success createPages — 1.082 s
success createPagesStatefully — 0.060 s
success onPreExtractQueries — 0.001 s
success update schema — 0.920 s
success extract queries from components — 0.058 s
success run graphql queries — 24.362 s
success write out page data — 0.029 s
success write out redirect data — 0.001 s
success onPostBootstrap — 0.000 s
info bootstrap finished - 32.341 s
success Building CSS — 4.555 s

success Building production JavaScript bundles — 14.079 s

success Building static HTML for pages — 6.982 s

info Done building in 57.968 sec

58.04 real        59.13 user         2.67 sys

464437248  maximum resident set size

Promise.map with concurrency 100: 870MB and 88s


$ rm -rf .cache/ public && /usr/bin/time -l ../../node_modules/.bin/gatsby build
success delete html files from previous builds — 0.084 s
success open and validate gatsby-config.js — 0.006 s
success copy gatsby files — 0.045 s
success onPreBootstrap — 0.018 s
success source and transform nodes — 2.718 s
success building schema — 0.893 s
success createLayouts — 0.015 s
success createPages — 1.079 s
success createPagesStatefully — 0.019 s
success onPreExtractQueries — 0.001 s
success update schema — 0.898 s
success extract queries from components — 0.072 s
success run graphql queries — 51.573 s
success write out page data — 0.204 s
success write out redirect data — 0.001 s
success onPostBootstrap — 0.001 s
info bootstrap finished - 59.781 s
success Building CSS — 6.609 s

success Building production JavaScript bundles — 13.992 s

success Building static HTML for pages — 7.681 s

info Done building in 88.076 sec

88.14 real        89.46 user         3.64 sys

870801408  maximum resident set size

Promise.map with concurrency 250: 1526MB and 101s


$ rm -rf .cache/ public && /usr/bin/time -l ../../node_modules/.bin/gatsby build
success delete html files from previous builds — 0.077 s
success open and validate gatsby-config.js — 0.007 s
success copy gatsby files — 0.027 s
success onPreBootstrap — 0.014 s
success source and transform nodes — 2.492 s
success building schema — 0.889 s
success createLayouts — 0.008 s
success createPages — 1.079 s
success createPagesStatefully — 0.051 s
success onPreExtractQueries — 0.003 s
success update schema — 1.010 s
success extract queries from components — 0.120 s
success run graphql queries — 64.101 s
success write out page data — 0.019 s
success write out redirect data — 0.001 s
success onPostBootstrap — 0.001 s
info bootstrap finished - 72.059 s
success Building CSS — 8.484 s

success Building production JavaScript bundles — 13.570 s

success Building static HTML for pages — 7.427 s

info Done building in 101.704 sec

101.77 real        98.59 user         4.22 sys

1526960128  maximum resident set size

From these results it looks like concurrency of 2 works best, though maybe setting it to something like Math.min(os.cpus().length, 32)(this gives number of logical, not physical cores) might be a better idea.

jquense · 2017-11-21T18:51:50Z

packages/gatsby/src/internal-plugins/query-runner/page-query-runner.js

@@ -90,8 +90,9 @@ const runQueriesForIds = ids => {
    return Promise.resolve()
  }
  const state = store.getState()
-  return Promise.all(
-    ids.map(id => {
+  return Promise.mapSeries(


we should maybe try and not depend on bluebird's static methods as it makes switching to the native promise eventually harder

KyleAMathews · 2017-11-23T14:45:37Z

Deploy preview failed.

Built with commit db8a1ae4b69555935f1061dafa41731725df094e

https://app.netlify.com/sites/using-glamor/deploys/5a1c3fc28198760772b713bd

Bouncey · 2017-11-23T16:19:31Z

I can confirm this fixes memory issues when trying to build ~1470 pages 👍

KyleAMathews · 2017-11-28T02:09:04Z

This is really great! Thanks for diving into Gatsby's innards and finding the problem — been thinking a lot recently about how to add automated perf testing to Gatsby which will make it a lot easier to track down problem spots like this + prevent regressions.

Per @jquense's comment — could you use async.js' concurrency controls instead of Bluebird's non-standard Promise concurrency?

KyleAMathews · 2017-11-28T02:47:59Z

Also the # of CPUs don't matter since JS is single-threaded. So please back that out.

szimek · 2017-11-28T11:34:15Z

@KyleAMathews I don't really know why you don't want to keep using bluebird, if this code runs only on the server side (you could also import is as Bluebird instead of Promise), but sure, I can switch it to use async.js. What should I set concurrency to? 2? 4? something else?

jquense · 2017-11-28T12:59:29Z

I'm ok using bluebird as a utils libraries the issue is that it's assuming the global promise object is a bluebird promise, which makes switching to just the platform promise harder. E.g. We want a lodash approach not prototype extensions

jquense · 2017-11-28T13:00:18Z

(and yes other code does this too :) we want to limit its spread)

KyleAMathews · 2017-11-28T16:37:17Z

I don't really know why you don't want to keep using bluebird

Yeah — we don't want to use Bluebird's promise extensions. The only reason we switched to it was part for performance and part for its better error messages — both of which are improving in Node core so I can definitely see us pulling out Bluebird in the future which would speed up Gatsby some as we'd be loading less JS.

szimek · 2017-11-28T17:36:12Z

@KyleAMathews @jquense Please review it again :) async API is based on callbacks, so the code is a bit uglier than previously.

KyleAMathews · 2017-11-28T23:35:52Z

Looks great! Thanks for the investigation and staying with us on the back and forth on this PR! :-)

szimek force-pushed the fix-out-of-memory-error branch from 04917e5 to 5161f69 Compare November 20, 2017 10:18

jquense reviewed Nov 21, 2017

View reviewed changes

szimek force-pushed the fix-out-of-memory-error branch 2 times, most recently from d800ff6 to 8ab36c6 Compare November 23, 2017 09:55

szimek mentioned this pull request Nov 27, 2017

1.9.9 -> 1.9.100, now 'Javascript heap out of memory' #2796

Closed

szimek force-pushed the fix-out-of-memory-error branch from db8a1ae to e4265ed Compare November 27, 2017 20:57

szimek force-pushed the fix-out-of-memory-error branch from e4265ed to 78fd94a Compare November 28, 2017 17:16

Fix out-of-memory error when building hundreds of pages

e51c389

szimek force-pushed the fix-out-of-memory-error branch from 78fd94a to e51c389 Compare November 28, 2017 19:23

KyleAMathews merged commit 8517e09 into gatsbyjs:master Nov 28, 2017

KyleAMathews mentioned this pull request Dec 2, 2017

Upgrade from gatsby@1.9.123 to gatsby@1.9.127 seemingly breaks JSON parsing when there's a significant amount of JSON #3094

Closed

KyleAMathews mentioned this pull request Dec 20, 2017

Intermittent Netlify failures freeCodeCamp/guide#7925

Closed

2 tasks

szimek deleted the fix-out-of-memory-error branch April 9, 2019 09:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix out-of-memory error when building hundreds of pages #2969

Fix out-of-memory error when building hundreds of pages #2969

szimek commented Nov 19, 2017 •

edited

Loading

gatsbybot commented Nov 19, 2017 •

edited

Loading

gatsbybot commented Nov 19, 2017 •

edited

Loading

KyleAMathews commented Nov 21, 2017

KyleAMathews commented Nov 21, 2017

KyleAMathews commented Nov 21, 2017

szimek commented Nov 21, 2017

szimek commented Nov 21, 2017 •

edited

Loading

jquense Nov 21, 2017

KyleAMathews commented Nov 23, 2017 •

edited

Loading

Bouncey commented Nov 23, 2017

KyleAMathews commented Nov 28, 2017

KyleAMathews commented Nov 28, 2017

szimek commented Nov 28, 2017 •

edited

Loading

jquense commented Nov 28, 2017

jquense commented Nov 28, 2017

KyleAMathews commented Nov 28, 2017

szimek commented Nov 28, 2017

KyleAMathews commented Nov 28, 2017

Fix out-of-memory error when building hundreds of pages #2969

Fix out-of-memory error when building hundreds of pages #2969

Conversation

szimek commented Nov 19, 2017 • edited Loading

gatsbybot commented Nov 19, 2017 • edited Loading

gatsbybot commented Nov 19, 2017 • edited Loading

KyleAMathews commented Nov 21, 2017

KyleAMathews commented Nov 21, 2017

KyleAMathews commented Nov 21, 2017

szimek commented Nov 21, 2017

szimek commented Nov 21, 2017 • edited Loading

jquense Nov 21, 2017

Choose a reason for hiding this comment

KyleAMathews commented Nov 23, 2017 • edited Loading

Bouncey commented Nov 23, 2017

KyleAMathews commented Nov 28, 2017

KyleAMathews commented Nov 28, 2017

szimek commented Nov 28, 2017 • edited Loading

jquense commented Nov 28, 2017

jquense commented Nov 28, 2017

KyleAMathews commented Nov 28, 2017

szimek commented Nov 28, 2017

KyleAMathews commented Nov 28, 2017

szimek commented Nov 19, 2017 •

edited

Loading

gatsbybot commented Nov 19, 2017 •

edited

Loading

gatsbybot commented Nov 19, 2017 •

edited

Loading

szimek commented Nov 21, 2017 •

edited

Loading

KyleAMathews commented Nov 23, 2017 •

edited

Loading

szimek commented Nov 28, 2017 •

edited

Loading