io.js remote debuging is EXTREMELY slow compared to Node.js 0.10

[unbornchikken]

Hello,

If I run a debugging session in WebStorm or the recently fixed (node-inspector/node-inspector#563) node-inspector (--debug-brk), booting of the application takes forever. Even processing of the few require lines on top of my app.js file takes forever. I can se that io.js process is eating up a whole core of my i7 CPU. It takes almost two minutes to get the server started. In Node.js 0.10 this was around 7-10 seconds. I've tried Node.js 0.12. It's interesting because it's very slow too, but a slightly faster than what I'm experiencing in io.js.

Are there some parameters to speed thing out?

[Ndrou]

I have exactly the same problem... more than one minute to start my express project with --debug-brk in Webstorm :-(

[Ndrou]

I tested many versions of node. The problem seems to appear from 0.11.15.

[unbornchikken]

Sadly this bug makes node-inspector kinda useless POS with io.js. I don't even know a sane method to debug my applications in io.js, I simply don't have hours to wait for a breakpoint get hit. Switched back to Node.js 0.10 :(

I can confirm that this issue present with io.js 1.2 on Linux x64, Linux x86, Windows 8 x64 (and with Node.js 0.12 on those platforms).

[bnoordhuis]

I can't reproduce with the built-in debugger on a medium-sized application. What does the WebStorm debugger do that the built-in one doesn't?

Also, /cc @bajtos.

[unbornchikken]

It's using remote debugging exactly like node-inspector, it opens io.js with --debug-brk. This issue is not related to WebStorm it's related to remote debugging. I can reproduce this bug with node-inpector, which has an io.js compatible version in a PR there: node-inspector/node-inspector#563

[bnoordhuis]

That's why I cc'd @bajtos, he's the maintainer of node-inspector. It's still unclear to me why external debuggers are problematic when the built-in debugger seems to be working fine.

EDIT: To be clear, the built-in debugger is a 'remote' debugger as well; it runs in a separate process.

[unbornchikken]

I get it, sorry. I've just tested one of our medium sized express application with node-inspector and WebStorm on Linux (by using debug-brk and manual start). It was 7 seconds to get server started with Node.js 0.10 on both debuggers. It was 37 seconds to get server started on node-inpector. It was 43 seconds to get server started on WebStorm debugger (I think it's slower slightly because of the nolazy flag).

[bajtos]

@unbornchikken can you try the following please?

Install node-inspector@0.7 and run it with DEBUG=node-inspector:protocol:*. Save the logs for both 0.10 and 0.12. Then compare the logs and look for requests/responses that take unreasonable long time to complete in 0.12 compared to 0.10.

You can also upload the logs as a gist, so that we can take a look too. Please don't paste them as a comment in this issue, because they are long and they would make the comment page difficult to read.

[unbornchikken]

I've tried my best, but there was some issues. I was not able to log anything with Node.js 0.12, because it didn't honoured debug-brk, it started the application immediately. So I created logs with Node.js 0.10.30 + node-inspector@0.7, and with io.js 1.2 + node-inspector@node-inspector/node-inspector#563.

It's uploaded the results along with shell scripts there: http://1drv.ms/1AX33g8

This time Node.js 0.10 server startup time was slightly slower, maybe because it was debug mode enabled, but WebStorm started the server within 7 seconds in debug mode as a said before. io.js was the same as before.

[unbornchikken]

Tomorrow I'm gonna make logs by using Fiddler when debugging my project in WebStorm. Its debugger consistently supports all versions of node and io.js, maybe we will figure out what's the cause of this better with it.

[unbornchikken]

Ok, guys, there is the stuff. I have full pcap captures of Node.js 0.10.36 and io.js 1.2.0 debugging sessions. I've made it with RawCap, it shows a WebStorm debugging session from beginning to get a restify server started. I placed a breakpoint there.

io.js: http://1drv.ms/1w4n04n
Node.js 0.10.36: http://1drv.ms/1w4nyXW

You can open them in Wireshark, and can filter to communication streams by using the following filter:

tcp.stream eq 0 && tcp.len > 0

The whole session length is:

• node.js 0.10.36: 11 seconds
• io.js 1.2.0: 78 secons

This mean it takes 7x more time to get breakpoint hit in io.js! If this is not a bug, then what?

I cannot se any notable difference between the communication (Follow TCP stream in Wireshark), in io.js it just slow.

I'd appreciate if someone with deep v8 debugger knowledge could take a look at this, thanks!

[unbornchikken]

I've opened a discussing on JetBrains board too: https://devnet.jetbrains.com/thread/460705

[develar]

I am developer of JetBrains JS debuggers.

WebStorm (IDEA) is slower (start) than node inspector due to two reasons:

1. V8 debugger protocol is ugly (compare to WIP). So, we bootstrap our own implementation (using evaluate) on start. Yes, our protocol should be pulled to V8 (or node/io.js), but it works without problems.
2. WebStorm loads global variables on start (it is required for live console). You can disable it — pass vm property: -Djs.debugger.load.global.variables.on.start=false

I don't experience slow down on Mac OS X, but we have user report https://youtrack.jetbrains.com/issue/WEB-14539#comment=27-925591

There is some difference between WebStorm 9 and WebStorm 10 (139 branch and 140), but in this area nothing was changed in the past 6 months.

[unbornchikken]

Do you have any idea why does WebStorm debugger hit the same breakpoint in the same project much slower in io.js than in node 0.10?

[develar]

@unbornchikken I have not yet looked closely, but we don't use v8 implementation of RPC "break" event, we use our own efficient implementation, so, the problem is not on the RPC level (and not due to some changes in this area in io.js), but in the V8 core itself.

I cannot reproduce the problem: WebStorm 140, OS X, nodejs 0.12 or io.js 1.2.0 — start debug and stop on breakpoint are not slow.

[unbornchikken]

I can reproduce this in Windows 8 x64, Linux Mint x86 and Linux Mint x64. Please take a look to my attached network captures. Those have been taken on Windows 8 x64 / WebStrom 9.0.3.

[develar]

@unbornchikken Please try WebStorm 10 EAP (https://confluence.jetbrains.com/display/WI/WebStorm+EAP).

You can enable lazy compilation (we disable it due to V8 bug https://code.google.com/p/v8/issues/detail?id=2825) — please open registry (Help -> Find action -> type in registry) and set js.debugger.v8.lazy.compilation to true and check if this makes the performance any better (latest WebStorm 10 EAP only).

[unbornchikken]

I've tried 140.2543. Same issue. In this version I was not able to find the mentioned debugger setting. Maybe the latest version that you mentioned is not published yet.

[ubaltaci]

I found out initial time to start debug session in WS for new v8's ( 3.28+ ) is too slow compared to old v8`s. ( In my case node 0.11.14 and older versions works as expected ) Based on our projects I saw 2 to 5 times increase in start time.

OS X 10.10.3
WS-140.2543

[rvagg]

can I suggest you try with and without IPv6 enabled on the machine(s) in question as that's one major delta between versions that could play in to this (totally stabbing in the dark here)

[unbornchikken]

I've tried debugging with disabled IPv6 (http://superuser.com/questions/586144/disable-ipv6-loopback-on-windows-7-64-bit) and get the same result.

[unbornchikken]

I think I successfully narrowed down the problem. The slow part is where the application working on requireing the source files (on top of app.js for example). After this point the debugging behaves exactly like it was in Node 0.10 performance wise. Stepping over source lines or stepping into functions are fast as always.

[dougwilson]

AFAIK this sounds like this: jshttp/mime-types#16 which was that the WebStorm debugger was ungodly slow when JSON.parse is called on an enormous JSON file...

[unbornchikken]

Im using the same instance of Webstorm against different versions of node. With 0.10 debugger is fast. With 0.12/io.js debugger is slow. Also, if you see my opening post, io.js process is eating up my CPU core not Webstorm. And please note, im experiencing the same issue with node-inspector too, starting of the debugging session takes a good one and a half minute in a medium sized project.

Im using Webstorm there because its debugger consistently supports all versions of node and io.js.

[gabooh]

Exactly the same problem here. Very slow to start, fine then.

[Ndrou]

For complete the tests, we've tested too on the lastest WebStorm 9 version and on the Webstorm 10 EAP, same problem. I don't think Webstorm is the problem. As said @unbornchikken, I also noticed requireing the source files is the part that slowed the debugger. Just start io with --debug-brk on a medium project with many require, and you will see the problem !
The problem has occurred since node 0.11.15.

[bnoordhuis]

Let's try a quick sanity check. Can one of you who is on Linux try the following:

$ perf record -c 50000 -g -i iojs --perf_basic_prof --debug-brk app.js

Connect the debugger right away and when it's done connecting, ^C iojs, then run perf report -F overhead -g flat --stdio > report.txt and paste the output in a gist. Remove -F overhead if your perf doesn't know about the -F switch.

EDIT: Fixed typo in command.

[gabooh]

@bnoordhuis done, the the resulting text file is 11mb. So it's available here gzipped : http://dl.free.fr/hZnaR8l8U

I've stopped the execution once the debugger was connected, not after application init ended (which would be much, much longer).

edit: using iojs 1.3 and node-inspector to connect
edit2: BTW in case you're trying, there is a typo in the first command, it's : perf record

[bnoordhuis]

@gabooh Thanks, but I can't seem to get a download going and my high school French is not good enough to figure out what's going wrong. Can you perhaps gist the first few thousand lines?

[unbornchikken]

I can only chech this on monday.

[jamesongithub]

what does js.debugger.v8.use.any.breakpoint do? is there any drawbacks to disabling?

[develar]

@yangguo-chromium Thanks! "
I can confirm that with node 4.1 the issue is fixed." https://youtrack.jetbrains.com/issue/WEB-16076#comment=27-1118156

[Fishrock123]

Sounds like this is fixed then? Let us know so we can re-open if this is still a problem! :)

[hilkeheremans]

Yep, still not having issues so assuming it is fixed. Back in debugging heaven.

Thanks for the feedback everyone!

[unbornchikken]

It's fine at my side as well, thanks for everyone involved! 👍

[o0x2a]

It seems the problem is back with node.js v5.0. Debugging is so slow that almost useless using node v5.0 and latest version of webstorm v11.0.1 .

[unbornchikken]

Confirmed: #3875

[hilkeheremans]

The issue occurs again on node v5.0.0, not on v4.1.0 or v4.1.1. Tested with WebStorm v11.0.1. See above, I also confirm on my end.

[unbornchikken]

@hilkeheremans #3875

[hilkeheremans]

@unbornchikken Right, sorry!

[hashseed]

Can someone check whether this helps? https://codereview.chromium.org/1454673002/