Stack size limit on generation of more then 10000 pages

[kukoba]

In our project there are about 6000 pages (2000 pages * 3 languages). It's development state and production site will have about 20-25 languages (so, > 40000 pages)

For now, during develop google chrome returns error in dev console

Uncaught RangeError: Maximum call stack size exceeded

Firefox works well (because has bigger stack size limit)

And build state return error:

WebpackError: Maximum call stack size exceeded

For one language (~2000 pages) all work fine.
For build tried to use options of nodejs:

--max-old-space-size=100000 --stack-size=65000

End system-wide:

ulimit -s 65000

But it did not help.

Spent some time for investigation and seems that will be spent much more.
Has anybody experience of generation sites with huge amount of pages (more then 10000)?
Are there any idea how to fix it fast?

As alternative of generation of such amount of static pages I see using SSR (most part of pages it's almost the same pages with trading instruments information) But do not want to use this approach because in this case we will not be able to use pure S3 for hosting the site.

Will be grateful for any help

[misscs]

@CALLIKA We are not generating that many pages, but we've run into the rate limit issue. Here's what worked for us:

1. Ensure you are using the latest version of Gatsby.
2. Adjust and tweek --max_old_space_size in order to get yours work. We are generating ~2000 pages and this is how we've updated our build script

"build": "node --max_old_space_size=12000 ./node_modules/.bin/gatsby build --prefix-paths",

Ultimately, we are going to move the generation to an integration server so we can keep a limited subset of data locally.

Hope this helps!

[kukoba]

@misscs , thank you for response!

Ultimately, we are going to move the generation to an integration server so we can keep a limited subset of data locally.

Yep, we have two types of build & develop tasks (with :dev suffix & without it). So, develop:dev runs develop with only for one instrument. It helps for development, and seems that build task will be work fine eventually on our jenkins, but issue with google chrome is little bit worse...

[KyleAMathews]

Errors in the browser are different than errors in the build process.

The browser stack overflow is probably from the async-requires.js module.

[KyleAMathews]

I haven't investigated larger paged sites like this much yet but know in general the solution will be to break up how we're loading in our mapping of page resource requires.

[kukoba]

@KyleAMathews , It's absolutely clear that they are different and thank you for hint

[benedfit]

@KyleAMathews are there any techniques that you can think of that could help those of us running large sites now? Thankfully I've built my site in a way that it falls back to a functional state in those browsers that exceed the call size

[charlesgiroux]

OS: Windows 10
Node -v: 7.8.0
Gatsby: 1.9.158

I am not able to exceed 1600 pages. I am using the createPages API to create pages based on a JSON model that I find too complex to use with GraphQL (I am new at GraphQL).

I have tried:

node --max_old_space_size=12000 --stack-size=100000 ./node_modules/gatsby/dist/bin/gatsby.js build --prefix-paths

But it also crashes:

npm ERR! Exit status 3221225725
npm ERR!

I will try later from another machine.

[charlesgiroux]

Quick update: seems like Windows is more sensitive to this issue. I was able to generate 2600 files on OSX without having to increase Node's stack-size or memory.

[FullstackJack]

Hitting "max file descriptors limit" perhaps?
https://stackoverflow.com/questions/31108693/increasing-no-of-file-handles-in-windows-7-64-bit

[elrumordelaluz]

this worked for me:

node --max_old_space_size=12000 --stack-size=100000 ./node_modules/gatsby/dist/bin/gatsby.js build

However could be awesome to have a way to build production ready big website (I mean with lot of pages) without those size tweaks.

The error I get building with raw gatsby build is:

error Generating JavaScript bundles failed


  Error: app-6fe500fbb4d0b81e62cd.js from UglifyJs
  RangeError: Maximum call stack size exceeded
      at AST_Call.eval [as print] (eval at <anonymous> (/Users/lio/Sites/site-static/node_modules/webp  ack/node_modules/uglify-js/tools/node.js:28:1), <anonymous>:4250:34)
      at eval (eval at <anonymous> (/Users/lio/Sites/site-static/node_modules/webpack/node_modules/ugl  ify-js/tools/node.js:28:1), <anonymous>:4953:20)
      at doit (eval at <anonymous> (/Users/lio/Sites/site-static/node_modules/webpack/node_modules/ugl  ify-js/tools/node.js:28:1), <anonymous>:4247:13)
      at AST_Assign.eval [as print] (eval at <anonymous> (/Users/lio/Sites/site-static/node_modules/we  bpack/node_modules/uglify-js/tools/node.js:28:1), <anonymous>:4253:13)
      at AST_Seq.eval [as _do_print] (eval at <anonymous> (/Users/lio/Sites/site-static/node_modules/w  ebpack/node_modules/uglify-js/tools/node.js:28:1), <anonymous>:4874:18)
      at eval (eval at <anonymous> (/Users/lio/Sites/site-static/node_modules/webpack/node_modules/ugl  ify-js/tools/node.js:28:1), <anonymous>:4885:14)
      at doit (eval at <anonymous> (/Users/lio/Sites/site-static/node_modules/webpack/node_modules/ugl  ify-js/tools/node.js:28:1), <anonymous>:4247:13)
      at AST_Seq.eval [as print] (eval at <anonymous> (/Users/lio/Sites/site-static/node_modules/webpa  ck/node_modules/uglify-js/tools/node.js:28:1), <anonymous>:4253:13)
      at AST_Seq.eval [as _do_print] (eval at <anonymous> (/Users/lio/Sites/site-static/node_modules/w  ebpack/node_modules/uglify-js/tools/node.js:28:1), <anonymous>:4881:22)
      at eval (eval at <anonymous> (/Users/lio/Sites/site-static/node_modules/webpack/node_modules/ugl  ify-js/tools/node.js:28:1), <anonymous>:4885:14)
      at doit (eval at <anonymous> (/Users/lio/Sites/site-static/node_modules/webpack/node_modules/ugl  ify-js/tools/node.js:28:1), <anonymous>:4247:13)
      at AST_Seq.eval [as print] (eval at <anonymous> (/Users/lio/Sites/site-static/node_modules/webpa  ck/node_modules/uglify-js/tools/node.js:28:1), <anonymous>:4253:13)
      at AST_Seq.eval [as _do_print] (eval at <anonymous> (/Users/lio/Sites/site-static/node_modules/w  ebpack/node_modules/uglify-js/tools/node.js:28:1), <anonymous>:4881:22)
      at eval (eval at <anonymous> (/Users/lio/Sites/site-static/node_modules/webpack/node_modules/ugl  ify-js/tools/node.js:28:1), <anonymous>:4885:14)
      at doit (eval at <anonymous> (/Users/lio/Sites/site-static/node_modules/webpack/node_modules/ugl  ify-js/tools/node.js:28:1), <anonymous>:4247:13)
      at AST_Seq.eval [as print] (eval at <anonymous> (/Users/lio/Sites/site-static/node_modules/webpa  ck/node_modules/uglify-js/tools/node.js:28:1), <anonymous>:4253:13)
      at AST_Seq.eval [as _do_print] (eval at <anonymous> (/Users/lio/Sites/site-static/node_modules/w  ebpack/node_modules/uglify-js/tools/node.js:28:1), <anonymous>:4881:22)
      at eval (eval at <anonymous> (/Users/lio/Sites/site-static/node_modules/webpack/node_modules/ugl  ify-js/tools/node.js:28:1), <anonymous>:4885:14)
      at doit (eval at <anonymous> (/Users/lio/Sites/site-static/node_modules/webpack/node_modules/ugl  ify-js/tools/node.js:28:1), <anonymous>:4247:13)
      at AST_Seq.eval [as print] (eval at <anonymous> (/Users/lio/Sites/site-static/node_modules/webpa  ck/node_modules/uglify-js/tools/node.js:28:1), <anonymous>:4253:13)
      at AST_Seq.eval [as _do_print] (eval at <anonymous> (/Users/lio/Sites/site-static/node_modules/w  ebpack/node_modules/uglify-js/tools/node.js:28:1), <anonymous>:4881:22)
      at eval (eval at <anonymous> (/Users/lio/Sites/site-static/node_modules/webpack/node_modules/ugl  ify-js/tools/node.js:28:1), <anonymous>:4885:14)

---

I haven't investigated larger paged sites like this much yet but know in general the solution will be to break up how we're loading in our mapping of page resource requires.

@KyleAMathews could you expand a little more on this. I'm interested since I am creating lots of pages fetched from a custom API end point and some loop to paginate them and so on.

Thanks in advance,

[nkristoffersen]

I'm looking to generate 10,000 - 20,000 pages in Gatsby. Is that possible? Will V2 have better support for large sites? Or should I reduce the scope of generated pages and dynamically render missing pages client side?

Loving Gatsby btw, I build several GatsbyJS sites per month. :-)

[KyleAMathews]

V2 does indeed have much better support for large sites!

Our create-pages/markdown benchmarks can build a 10k site in less than a minute now after #6226!

I'll close this issue out actually as 10-20k page sites shouldn't generally be a problem any longer. Please open new issues if any of you have trouble with larger sites!