Replace the custom flamegraph viewer with speedscope #100

jlfwong · 2018-07-08T06:22:32Z

This PR replaces the custom flamegraph viewer introduced by @aroben in #74 (which sounds like it was a huge improvement over the existing SVG viewer!) with an integration of a high performance, WebGL-based, language agnostic profile viewer that I've been working on called speedscope: https://github.com/jlfwong/speedscope.

If you don't think this is a good fit for this repository, no worries! I'm happy to either just close this PR, or to change it to be a dramatically reduced version which just makes it easier to output a format that speedscope can consume (basically just the JSON serialized version of a stackprof profile; speedscope has built-in code to handle import of stackprof profiles: https://github.com/jlfwong/speedscope/blob/master/import/stackprof.ts)

A script is included (vendor/speedscope/update.sh) to handle updating speedscope in the future to pull in the latest version.

It should be able to easily handle profiles at least as large as the existing viewer, and zooming & panning should remain 60fps within those very large profiles through efficient use of the GPU for rendering.

Test Plan:
Ran the following:

$ ruby sample.rb
$ bundle exec bin/stackprof --flamegraph-viewer=/tmp/stackprof.dump

I also preserved the original workflow of doing a separate compilation & opening step:

$ ruby sample.rb
$ bundle exec bin/stackprof --flamegraph=/tmp/stackprof.dump > /tmp/flamegraph
$ bundle exec bin/stackprof --flamegraph-viewer=/tmp/flamegraph

This should open the profile in-browser in both Linux and OS X using whatever your default configured browser is. I've only tested on OS X to date on Firefox & Chrome, but it shouldn't be OS dependent.

Screenshots:

jlfwong · 2018-07-20T03:06:16Z

@tmm1 @aroben Ping? Is there someone else I should be sending this to?

tmm1 · 2018-07-20T03:08:42Z

I haven't looked at this code, but in theory this is fine by me.

I'm not using or maintaining this library anymore. @itsderek23 and @tenderlove are running the show here now.

tenderlove · 2018-07-25T22:20:54Z

This is really awesome. I'm 👍 on this, but I need to test it with our app first

itsderek23 · 2018-07-25T22:27:10Z

Yes - the viewer looks great @jlfwong.

I haven't used the current flamegraph viewer significantly.

@jlfwong - have you ran this against output from a Rails app? That's pretty common in the Ruby world.

NickLaMuro · 2018-07-25T22:39:44Z

I have some large samples sitting around somewhere that I have run against https://github.com/ManageIQ/manageiq , so I can give that a shot at some point tonight and report back.

(Update: I will do this tomorrow... it is late 🌝 )

NickLaMuro

So I think I might have more to say, but since I don't even have merge rights and have just been "nerd sniped" into reviewing because I use this repo myself regularly, I gave some comments.

Mostly personal opinion here, so feel free to take it or leave it.

Seems cool regardless!

-Nick

NickLaMuro · 2018-07-25T22:44:34Z

bin/stackprof

-    puts("open file://#{File.expand_path('../../lib/stackprof/flamegraph/viewer.html', __FILE__)}?data=#{File.expand_path(file)}")
-    exit
-  }
+  o.on('--flamegraph', "open a viewer for the flamegraph of the given profile"){ options[:format] = :flamegraph }


I am personally 👎 on this options consolidation change (not that I have any real pull in the final say...)

Reason being is I have used this to integrate with other repos, and having this in two separate steps is kind of ideal when you want to just save it to a path your your choosing (currently this forces the use of Tmpdir), and open at your convenience. Allowing you to choose a dir also allows you to organize your samples, even when using the private Stackprof.print_flamegraph interface, and now that is not available.

To maybe meet halfway:

Could you keep the options that exist currently, but change --flamegraph-viewer to optionally accept a file, and if one isn't provided, it will run a form of the Tmpdir code you currently have and open the file.

Thanks for the feedback! I don't have strong opinions here, and happy to do whatever would yield the best user experience here.

Open to guidance for what the commandline flags should be here and what the semantics should be.

and having this in two separate steps is kind of ideal when you want to just save it to a path your your choosing

My understanding is that if you want to collect multiple profiles for later viewing, you can collect them as the raw stackprof output into a directory of your choosing, then run the --flamegraph command to view them. I'm not sure what you would want to do with the intermediate form here other than open them to view as a flamegraph.

The intermediate form is a JavaScript ball that isn't usable (AFAIK) from anything except the flamegraph viewer.

The intermediate form is a JavaScript ball that isn't usable (AFAIK) from anything except the flamegraph viewer.

I always kind of thought of the --flamegraph flag command as the "compile" step of the flamegraph, and the --flamegraph-viewer flag as the execution. The "compile" step could be slow, depending on how big your stackprof blob is (you were talking about 100MB blobs, and I know the feeling myself), you might not want to repeat that process if you are looking at data over time, or with a before and after.

Sidenote (and I am NOT suggesting doing this in this PR), in regards to the "Javascript ball" being unusable, I have wondered if it made sense to have a option flag to make it a single .html file artifact instead of javsacript file but inlining the scripts (so speedscope in this case). Seems like it would avoid the whole viewer issue, and it is what the original perl version of the flamegraph does as well by making it an SVG.

I don't have any strong feelings about this change, but I'd prefer not to break people's workflow.

NickLaMuro · 2018-07-25T22:52:35Z

lib/stackprof/report.rb


-          flamegraph_row(f, x - row_width, y, row_width, row_prev)
-        end
+      if not `which xdg-open`.empty?


To avoid shelling out too much, could you implement this:

https://stackoverflow.com/a/5471032/3574689

Not that stackprof has Windows support anyway, but would save this being something needing to be changed in the future if that ever becomes the case.

The commands being checked (open and xdg-open) here are unlikely to be implemented in Windows. This will still need to be modified for Windows support to run some command that will successfully open the browser, so I'm not sure if the proposed solution helps much on Windows.

Made most of my case over in this comment, but did want to emphasize that this and the other comment was more of a suggestion, not a requirement or even a "strongly worded request".

NickLaMuro · 2018-07-25T22:57:29Z

vendor/speedscope/update.sh

@@ -0,0 +1,16 @@
+#!/bin/bash


Could this possibly turned into a Ruby script/rake task?

The Gem::Package::TarReader should be able to handle the Tar portion of things (cross platform), and then you can basically use FileUtils for the rest.

Obviously npm is still required and would need a shell out, but that is just a given and as you have stated it is just a developer dependency.

It could be, but I'm not sure it's worth doing -- is the target benefit to make it possible for a Windows-only maintainer to run the update script?

So there are a few non-"but what about WIN-DOSE" reasons I tend to always push for this myself, and it very much is pedantic in most cases... BUT YOU ASKED FOR IT! (he says sarcastically):

(I am writing a lot here, but again this is just an explanation to my personal opinionated stance and just what I do myself and why. Zero pressure to make the change and feel free to take it or leave it.)

Shelling out has some cost associated with it. Next to nothing 99% of the time, but depending on how much you need to do it, it can add up. In your case, nothing here of note or value where this is actually an issue, just an FYI.

When I use "Windows" as an example, bit is more about removing inconsistencies between platforms, and since we know Ruby is going to be used by who ever is running this script, we can assume it will be there as a more consistent constant (#redundantWordsAreRedundant). For a few examples in your case:

A user might have some weird version of /bin/bash (probably a bad example for this use case...)

which might not exist on the $PATH on some machines, or the user might be running with sudo

weird user aliases being loaded that mess with the calls in the script

Specifically to the rake suggestion, this is simply for dev/maintainer consistency. I know that you did put a README together for the speedscope stuff specifically, but from personal experience, most people don't RTFM. But assuming they would at least do a rake -T when they clone the project (which rake is pretty much accepted as the build pipeline tool for ruby projects), they can get at a glance what general dev tasks are available.

Further more, if it is written in ruby, we then are just doing Ruby -> npm for that one time we need to shell out, instead of Ruby -> shell -> npm/tar/.... Less moving parts and less places for failure.

</2cents>

@tenderlove Do you have opinions on this front? I don't spend too much time in the ruby ecosystem, so I'm not sure what the expectations are on this front. I'm happy to do this if you feel strongly, but would otherwise bias towards not doing it, since not doing it less work 😅

@jlfwong I think you meant to direct this at me, but I can put something together for you if it is decided this would be preferred.

Sorry, I misread the past comment and just noticed your true intent. Again, apologize for the extra noise.

I did, however, put together some sample code to show how this could be done, but again, no pressure to actually implement this, and it was mostly a interesting exercise for myself.

# Rakefile (I personally put this starting around line 11) def untar(tarfile) # Little bit of a hack with the `.new` to get this to work Gem::Package.new("").open_tar_gz tarfile do |tar| tar.each { |file| yield file } end end desc "Update speedscope assets" task :update_speedscope do rm_rf "vendor/speedscope" mkdir_p "vendor/speedscope" cd "vendor/speedscope" do sh "npm pack speedscope" File.open Dir.glob("speedscope*.tgz").first do |tarfile| untar tarfile do |file| next unless file.full_name =~ /^package\/(LICENSE|dist\/release\/.*(html|css|js|png))$/ File.write File.basename(file.full_name), file.read end rm tarfile.path end end end

One note, this does change the scoping of the resulting directory structure from vendor/speedscope/speedscope to just vendor/speedscope, since the nested dir seemed redundant (but I could be missing something). I would assume that some changes in report.rb would be necessary if this is a reasonable change, otherwise a few tweaks to this task could be made to emulate what already exists.

Thanks for writing up sample code! I always appreciate it when people are willing to take the time to write code to support their ideas.

That said, I'm not planning on changing this to a rake task unless this is considered a blocker for merging.

My reasoning here is that I'm intending to make similar changes to several different profilers in many different languages (e.g. pyflame), and would prefer the update scripts to look as similar as possible. Changing them to be specific to the language of the profiler would make that more difficult.

NickLaMuro · 2018-07-25T22:59:14Z

lib/stackprof/report.rb

+      File.open(tmp_js_path, "w") do |tmp_js_file|
+        tmp_js_file.write(js_source)
+      end
+      puts "Creating temp file #{tmp_js_path}"


Pedantic: Seems like this should be removed, or possibly hidden behind a --debug/--verbose flag.

Yep! Printing the temp file for the HTML file is useful if the browser open fails for some reason, but I agree this path isn't particularly helpful for anything other than my personal debugging while writing this :)

Noticed this after I re-read my review, but maybe putting this down as an else case for when you are doing the which checks and which can't find a program doing an "open", so at least the output file is being printed and something is shown as an output.

jlfwong · 2018-07-25T23:46:08Z

@jlfwong - have you ran this against output from a Rails app? That's pretty common in the Ruby world.

I have not, but have this integrated into https://github.com/jlfwong/rack-mini-profiler, which we use to profile a sinatra app regularly at Figma.

We also use speedscope at Figma to open 100MB+ profiles generated by Chrome, which it handles relatively gracefully.

tenderlove

We need to make sure the viewer and JSON are separate (it looks like this PR will do that), but also ensure we can print the flamegraph JSON data to an IO of our choosing. I think this PR removes the that ability (though I could be wrong because the diff is pretty large 😅).

As long as we have an API that can ensure those things, then I'm happy. 😊

tenderlove · 2018-07-27T18:05:09Z

bin/stackprof

-    puts("open file://#{File.expand_path('../../lib/stackprof/flamegraph/viewer.html', __FILE__)}?data=#{File.expand_path(file)}")
-    exit
-  }
+  o.on('--flamegraph', "open a viewer for the flamegraph of the given profile"){ options[:format] = :flamegraph }


I don't have any strong feelings about this change, but I'd prefer not to break people's workflow.

tenderlove · 2018-07-27T18:05:19Z

lib/stackprof/report.rb

@@ -80,65 +83,36 @@ def print_stackcollapse
      end
    end

-    def print_flamegraph(f=STDOUT, skip_common=true)


We need to maintain this API. We're using flamegraphs in production by serving a static asset (the flamegraph viewer) and it makes a request to and endpoint that serves up the flamegraph JSON for a particular page. We use this method to print the flamegraph JSON to the socket

Can you elaborate on the workflow you use? This function as it currently exists doesn't print JSON directly, but rather prints a JavaScript function invocation whose only argument happens to be valid JSON.

Do you take the output of this and strip the flamegraph( at the beginning and the trailing ) at the end then serve it as true JSON?

Or, when you say

it makes a request to and endpoint that serves up the flamegraph JSON for a particular page

do you mean that you set up the flamegraph viewer to include a <script> which references the exact output of print_flamegraph?

Both before and after this PR, the data and the viewer are separated, and before and after this PR the data is valid JavaScript but not valid JSON without modification.

So I think the request here is that "the step for extracting the JavaScript file and the step for opening the browser need to be separate steps so that consumers of stackprof as a library can serve the JavaScript file manually".

If that's the request, I'm going to open up a different code pathway for this.

Speedscope has multiple ways of loading data.

Dropping local files in

Browsing for local files

Specifying files via (possibly CORS) XHR URLs, which would must not contain the JavaScript function invocation wrapper (documented here: https://github.com/jlfwong/speedscope#importing-via-url)

Specifying files via file:// protocol script URLs which must contain the Javascript function invocation wrapper, which is what this PR does since you can't make XHR requests to file:// protocol URLs.

So if I understand you correctly, here's a proposed remediation:

Add a method called get_speedscope_json which returns a JSON string (not a JavaScript string) which could then be served over HTTP

Use that method in view_flamegraph_in_browser to retain the behavior in the PR as it stands

Then to upgrade your integration with stackprof, it would require changing the call-site to use get_speedscope_json instead of print_flamegraph_json, and also change the static assets that are being served right now to serve speedscope rather than serving the existing flamegraph viewer.

How does that path forward sound to you?

Yes, this sounds perfect!

This should be complete now

tenderlove · 2018-07-31T22:11:03Z

I've tried integrating this in to production, but it seems like speedscope.c6a476e8.js tries to dynamically download other stuff (and our content policy blocks it). Is there a way we can combine it in to one script?

jlfwong · 2018-07-31T23:49:43Z

I've tried integrating this in to production, but it seems like speedscope.c6a476e8.js tries to dynamically download other stuff (and our content policy blocks it). Is there a way we can combine it in to one script?

Hmm. It's going to be a bit tricky, but it's doable. The reason it downloads other things is to speed up loading so that it can present a UI and allow users to browse for a file without needing to wait for all the code that does the actual importing.

If I'm going down that path anyway, would you prefer that everything is inlined into the .html file instead?

I'll look into it when I change the codepaths to split get_speedscope_json and print_flamegraph_json.

jlfwong · 2018-08-01T08:33:41Z

@tenderlove To be clear, are you talking about Content-Security-Policy? If so, can you provide what Content-Security-Policy header you're working with?

I'm curious how you have this set up with the current flamechart viewer, given that I thought it loads scripts dynamically too?

jlfwong · 2018-08-02T05:32:38Z

@tenderlove My working assumption is that you have a Content-Security-Policy header which does not include strict-dynamic. Is that right? https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/script-src#strict-dynamic

jlfwong · 2018-08-02T17:43:56Z

bin/stackprof

-  o.on('--flamegraph-viewer [f.js]', String, "open html viewer for flamegraph output\n\n"){ |file|
-    puts("open file://#{File.expand_path('../../lib/stackprof/flamegraph/viewer.html', __FILE__)}?data=#{File.expand_path(file)}")
+  o.on('--flamegraph', "output format for consumption by --flamegraph-viewer"){ options[:format] = :flamegraph }
+  o.on('--flamegraph-viewer [profile-path]', "open a viewer for the flamegraph of the given profile"){ |f|


Okay, I switched this back to preserve the original switches, with the added ability to use --flamegraph-viewer directly on a profile file without needing to do the compile step first, but also supporting doing the compile step first to preserve people's existing workflows

jlfwong · 2018-08-02T17:45:55Z

Okay, @tenderlove I updated the PR using a build which shouldn't do any dynamic script loading. The work is based on an open PR on speedscope, so I pulled in the tarball for it manually rather than using npm pack speedscope. If does end up meeting your needs, I'll land the PR and push to npm.

Here's the relevant PR on speedscope if you're curious: jlfwong/speedscope#113

jlfwong · 2018-08-05T17:02:13Z

@tenderlove I'm also interested to know if the CSP policy you have set up prohibits the use of eval. That might end up being a bigger problem since one of the core libraries I depend upon uses eval: https://github.com/regl-project/regl

jlfwong · 2018-08-09T16:41:41Z

@tenderlove ^ ping

tenderlove · 2018-08-15T22:04:14Z

@jlfwong hey, sorry it took so long to get back to you. Yes, our CSP won't allow eval. I'm not totally sure what to do here. I really like the viewer in this PR, but it sounds like we may not be able to use it in production.

My goal is that folks at work can just click a link and see flame graphs of the page.

I really want this in stackprof because it's hands down better than the existing one. Maybe we could keep an API that outputs data that will work with the old viewer?

jlfwong · 2018-08-15T22:21:59Z

@tenderlove Got it. Yeah, that makes sense!

My goal is that folks at work can just click a link and see flame graphs of the page.

Definitely a noble goal :)

A couple of alternative options to potentially consider. You've probably thought of these already, but just to make sure that these are all show-stoppers:

Have a different CSP specifically for the page loading speedscope
Host speedscope on a totally different domain without the same CSP protection, and then allow access to the API endpoint which serves the profile file cross-domain via an Access-Control-Allow-Origin header. I have a friend which uses speedscope at another company in exactly this way: their profiles are uploaded to S3, and a header is set on the S3 bucket to allow them to be accessed cross domain. This opens a different variety of attack vector that might be unreasonable to open, even if done only for that specific endpoint, but speedscope being hosted on a different domain would guard against cookie jacking attacks via a theoretical XSS hole.

If those are both no-gos, then I'll look into either preserving the existing flamechart viewer or into switching the WebGL abstraction I'm using to one which doesn't use eval.

jlfwong · 2018-08-19T19:33:23Z

@tenderlove Okay, I've updated the PR to use a version which does not use eval.

To validate that it was working, I opened it via a local server and specified the following header:

Content-Security-Policy: script-src 'self';

I was able to validate that the version before the removal of eval calls did not work with that policy, and that the version now in this PR does. It doesn't seem like the dynamic script loading was the problem, I think it was just the eval calls, which is great because it means that I don't have to maintain two different builds :)

Can you see if this now works for GitHub's content security policy?

If it turns out that I'm wrong and that both the eval calls and the dynamic file loading were causing issues, I can easily switch this to a build that has both removed.

jlfwong · 2018-09-26T21:48:12Z

@tenderlove ping! ^

xsidax · 2023-01-27T17:51:19Z

This seems rather interesting, is this proposal alive?

Jamie Wong added 6 commits July 7, 2018 21:59

Remove old flamegraph viewer

5aa74f7

Update sample.rb to be usable for outputting a flamegraph

0103364

Import speedscope and include instructions on how to update it

7ab9f9e

Integrate into the stackprof command

c64829b

Actually open the browser

f705bfe

Upgrade to speedscope 0.1.2

71aa3e3

jlfwong mentioned this pull request Jul 17, 2018

Include an output format for speedscope rbspy/rbspy#161

Merged

jlfwong requested a review from tmm1 July 20, 2018 03:06

jlfwong assigned tmm1 Jul 20, 2018

jlfwong requested a review from aroben July 20, 2018 03:06

jlfwong assigned aroben Jul 20, 2018

tmm1 unassigned aroben Jul 20, 2018

tmm1 removed the request for review from aroben July 20, 2018 03:08

jlfwong requested review from tenderlove and itsderek23 and removed request for tmm1 July 20, 2018 03:13

jlfwong assigned tenderlove and itsderek23 and unassigned tmm1 Jul 20, 2018

NickLaMuro reviewed Jul 25, 2018

View reviewed changes

tenderlove requested changes Jul 27, 2018

View reviewed changes

jlfwong mentioned this pull request Aug 1, 2018

Add a single file (or single JS file at least) build option jlfwong/speedscope#108

Open

Jamie Wong added 2 commits August 2, 2018 10:37

Retain the original switches

ee4b6b6

Switch to a single file build

bd69ced

jlfwong commented Aug 2, 2018

View reviewed changes

jlfwong mentioned this pull request Aug 19, 2018

Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed regl-project/regl#491

Closed

Jamie Wong added 2 commits August 19, 2018 12:25

Update to a version which does not use eval

78ef0a9

Add missing txt file

213b8e6

jlfwong mentioned this pull request Jan 5, 2019

Add --json format #103

Merged

aroben mentioned this pull request May 29, 2019

Performance issues for large graphs SamSaffron/flamegraph#30

Open

jlfwong mentioned this pull request Apr 13, 2020

Allow speedscope to be used as a library instead of a standalone application jlfwong/speedscope#16

Open

Replace the custom flamegraph viewer with speedscope #100

Are you sure you want to change the base?

Replace the custom flamegraph viewer with speedscope #100

Conversation

jlfwong commented Jul 8, 2018 • edited Loading

jlfwong commented Jul 20, 2018

tmm1 commented Jul 20, 2018

tenderlove commented Jul 25, 2018

itsderek23 commented Jul 25, 2018

NickLaMuro commented Jul 25, 2018 • edited Loading

NickLaMuro left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jlfwong Jul 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jlfwong Jul 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NickLaMuro Jul 26, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NickLaMuro Jul 27, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NickLaMuro Jul 26, 2018 • edited Loading

Choose a reason for hiding this comment

jlfwong commented Jul 25, 2018

tenderlove left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jlfwong Jul 27, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tenderlove commented Jul 31, 2018

jlfwong commented Jul 31, 2018

jlfwong commented Aug 1, 2018

jlfwong commented Aug 2, 2018

Choose a reason for hiding this comment

jlfwong commented Aug 2, 2018

jlfwong commented Aug 5, 2018

jlfwong commented Aug 9, 2018

tenderlove commented Aug 15, 2018

jlfwong commented Aug 15, 2018 • edited Loading

jlfwong commented Aug 19, 2018 • edited Loading

jlfwong commented Sep 26, 2018

xsidax commented Jan 27, 2023

jlfwong commented Jul 8, 2018 •

edited

Loading

NickLaMuro commented Jul 25, 2018 •

edited

Loading

jlfwong Jul 25, 2018 •

edited

Loading

jlfwong Jul 25, 2018 •

edited

Loading

NickLaMuro Jul 26, 2018 •

edited

Loading

NickLaMuro Jul 27, 2018 •

edited

Loading

NickLaMuro Jul 26, 2018 •

edited

Loading

jlfwong Jul 27, 2018 •

edited

Loading

jlfwong commented Aug 15, 2018 •

edited

Loading

jlfwong commented Aug 19, 2018 •

edited

Loading