Eslint is slow when used with webpack

[mieszko4]

I am wondering if it is possible to boost up performance when using eslint-plugin-import with webpack.

In my project:

• I have 4500 modules (find node_modules/ -type f -name "package.json" | wc -l)
• I have 500 files that I am eslinting
• I use 8 aliases in webpack config resolve.alias
• I use 8 loader rules in webpack config module.rules.

Eslinting lasts ~30s, and I can see that import/no-unresolved takes most of the time:

Rule                              | Time (ms) | Relative
:---------------------------------|----------:|--------:
import/no-unresolved              |  4983.197 |    28.2%
react/prop-types                  |  1694.653 |     9.6%
react/sort-comp                   |  1321.233 |     7.5%
react/no-multi-comp               |  1264.966 |     7.2%
react/require-render-return       |  1221.941 |     6.9%
import/no-named-as-default        |   665.776 |     3.8%
react/no-deprecated               |   524.655 |     3.0%
indent                            |   261.081 |     1.5%
import/no-named-as-default-member |   248.505 |     1.4%
react/jsx-curly-spacing           |   198.665 |     1.1%

Any suggestions on improvement?

[ljharb]

If you disable the webpack aliases and the resolver config for them, does it go faster?

[mieszko4]

I found the problem!
I am using webpack.config.babel.js and the problem is that it reads and transpiles the entire webpack config each time it resolves the rule. And there are lots of calls, in my case over 150 of them.

I suggest the config is read once and then cached. I will make a pr.

[ljharb]

Glad you figured it out; seems like a webpack issue.

[ljharb]

Wait, or are you saying that eslint-plugin-import reads your webpack config?

[mieszko4]

Yes, eslint-import-resolver-webpack reads the config each time exports.resolve() is called.
The improvement could be that it reads it once and exports.resolve() calles the cached version.

[ljharb]

Sounds like a great improvement, thanks!

[mieszko4]

After some further investigation I concluded that caching it would not speed it up (too much) since require() that is used caches the call already. It is simply slow because of babel the first time it is called.
Each subsequent time it is quick as it uses the cached version already.

I've created a pr anyway as it moves out some common processing.

[mieszko4]

Actually, correction I was testing it wrong. With this pr I got an improvement of over 2s.

[benmosher]

fyi: caching it permanently will impair things like eslint-loader and eslint_d that hold things in memory for a long time. I use eslint_d for linting in Sublime to improve performance.

caching it impermanently (i.e. maybe for 30s tops) would be fine.

[ljharb]

Could it be cached permanently but with an md5 hash of the file's contents, so any change busts the cache?

[benmosher]

lol yep, just suggested that over on #789 😁

[mieszko4]

Hmm. using md5 of webpack config file will not make it quicker then.
This is because quite some time seems to be spent on resolving the webpack config file.
require used in the code caches the file that is required.
So at first pass it is slow because of babel, it caches the required webpack config, then at each subsequent pass it is slow because of file path resolution.

[benmosher]

ahhh, right, I forgot that every file doesn't necessarily have the same webpack config. part of the resolver is finding the "right" one. you're right, would need to do that every time anyway

[mieszko4]

I guess then that there is nothing much that can be done here, i.e. at least my pr (#789) does not make sense.
Perhaps only giving an optional parameter to this rule, something like { cachable: true }

[benmosher]

fwiw, eslint_d substantially improves performance for me in normal (webpack) use. There are many caches that this plugin builds already, and eslint_d allows them to exist across invocations. (no help for CI but it's great for local use)

[mieszko4]

thanx @benmosher. I am using atom.io with https://github.com/Adezandee/fast-eslint and it works nice for me. I had a problem with https://github.com/AtomLinter/linter-eslint though, it was unuseably slow.
CI is a different story, but 5s longer build is fine.

[mikhailChibel]

@mieszko4 how did you get the table with rule execution times?

[mieszko4]

@mikhailChibel prefix the call with TIMING=1

[kamronbatman]

This is still a major problem. We have ./index.mjs that re-exports a component and I have a feeling that simply adding re-exports is making the linter exponentially slower.

[kamronbatman]

Tested 2.12.0 and it looks like I am having a different issue that is causing slowness.

[benmosher]

@kamronbatman 🤦‍♂️ ohh nooo! any ideas?

[benmosher]

so #1091 was driven by my own results dropping timing logs into the webpack resolver and looking at what took the most time; the stuff I cached reduced my times by 80%, though it was only ~5sec of my 60s total lint time and so it only shaved ~4s off.

if you could either use the Node profiler (though I found this difficult and of limited utility) or just add some timing captures and logs and see what regions of code specifically are slowing you down, that would be the most helpful. I have already fixed the big waste that I could see.

also: did you upgrade your webpack resolver to 0.10? most of the fixes (if not all of them) were in there, and not in v2.12 of the plugin.

[gfx]

In my production, import/cycle is extremely slow:

TIMING=true npx eslint --ext .ts,.tsx src

Rule	Time (ms)	Relative
import/no-cycle	22167.840	62.9%
prettier/prettier	8538.422	24.2%
@typescript-eslint/await-thenable	1832.930	5.2%
react/no-deprecated	553.444	1.6%
import/no-deprecated	503.936	1.4%

It takes 22 seconds!

[ljharb]

@gfx the first rule will always be slow because it’s building up an import graph; see #1408 for typescript-specific slowness.

[gilbarbara]

I upgraded my webpack config to be a factory (exporting a function instead of a simple object like CRA does) and my ESLint timing went through the roof.

So instead of setting the webpack config factory directly to import/resolver, I'm using a file where I import the factory and export the result.

// webpack.config.eslint.js
const configFactory = require('./webpack.config');

module.exports = configFactory('development');

Before

Rule	Time (ms)	Relative
import/no-unresolved	82850.597	54.6%
import/no-named-as-default-member	47993.517	31.6%
indent	3229.228	2.1%
import/named	3056.267	2.0%
react/void-dom-elements-no-children	1000.009	0.7%
react/no-deprecated	858.790	0.6%
react/jsx-no-bind	784.099	0.5%
react/no-direct-mutation-state	731.675	0.5%
react/destructuring-assignment	593.046	0.4%
react/sort-comp	483.998	0.3%

After

Rule	Time (ms)	Relative
import/no-unresolved	10543.458	32.5%
import/no-named-as-default-member	3781.838	11.7%
indent	2865.881	8.8%
react/void-dom-elements-no-children	817.273	2.5%
react/no-deprecated	810.339	2.5%
react/jsx-no-bind	759.052	2.3%
react/no-direct-mutation-state	696.619	2.1%
react/destructuring-assignment	589.438	1.8%
import/named	446.033	1.4%
react/no-typos	416.777	1.3%

[mosherc]

Just wanted to bump this with another data point. Added this along with import/no-deprecated and our CI lint steps went from a couple minutes to 20-25 minutes. We were deprecating a few components that were in a bunch of places, but "fixing" the deprecation did not reduce the time at all. Removing the webpack setting in our eslint reduced the time back to normal, but obviously now it will not lint our imports that use webpack aliases. This makes the webpack resolver difficult to justify since it slows us down so much.

[ljharb]

@mosherc this happens when you have any rules enabled that use the import map (ie, that traverse all of your modules).

[mosherc]

Ok so it really is between using no-deprecated and having long linting times, or removing that rule/webpack resolver to go back to shorter linting times? Those are my only choices and there's no way to improve performance? I tried some of the ideas in the comments and got no noticeable improvement.

[ljharb]

Usually when a codebase is large enough to run into these problems, I suggest using jest and jest-runner-eslint to parallelize linting.