Segmentation fault in sassc-2.2.1 when I visit ActiveAdmin in production

[ndbroadbent]

Hello,

I upgraded sassc to 2.2.1, and my production Rails app is now crashing when I visit my ActiveAdmin dashboard. Here are the crash logs from my server: https://gist.github.com/ndbroadbent/3d0f78235dc9056ca9eb9464d53b4a9c

/app/vendor/bundle/ruby/2.5.0/gems/sassc-2.2.1/lib/sassc/engine.rb:42: [BUG] Segmentation fault at 0x0000000000000000

Versions:

• sassc: 2.2.1, 2.0.1
• ruby: 2.5.5p157
• activeadmin: 2.4.0
• haml: 5.1.2

I'm not too sure why ActiveAdmin is calling SassC in production, but I think it might be part of a third-party template that needs to be rendered at runtime.

Another thing to note is that this consistently crashes my app on every request.

Thanks for your help!

Potentially related issues:

• Segment Fault in 2.0.1 - 2.1.0pre3 and ruby 2.6.2p47 #133
• Segmentation fault on "RAILS_ENV=production assets:precompile" sassc-rails#122
• Segfault rails/sprockets#633

[ndbroadbent]

Sorry, this is also happening with sassc 2.0.1. I upgraded a few gem dependencies at the same time (including ActiveAdmin), so I'm not sure what started causing this crash, but it does seem to be coming from sassc or libsass.

[ahorek]

I reported the same problem a year ago rails/sprockets#581 and I propose a fix, but there's still no response #162 and rails/sprockets#630

the second option is disabling concurrency, it could still break in some cases, but it'll be much less likely

Rails.application.config.assets.configure do |env|
  env.export_concurrent = false
end

[ndbroadbent]

Thanks for the suggestion @ahorek! Unfortunately I don't think that export_concurrent setting would work for me, because this is not related to sprockets in my case. It's not happening in the assets:precompile task, it's actually happening when I make a request during production, because ActiveAdmin renders some templates (and SCSS) at runtime.

[ahorek]

hmm, do you really need dynamic assets compiled in production?
in your case sassc is acually called from tilt https://github.com/rtomayko/tilt/blob/master/lib/tilt/sass.rb#L28
maybe one of your gems is mutating sassc functions ( https://github.com/sass/sassc-ruby/blob/master/lib/sassc/script/functions.rb ) at runtime the same way as sprockets do?

it would be helpful if you have a repro script...

[ndbroadbent]

hmm, do you really need dynamic assets compiled in production?

Yes I'm not sure why ActiveAdmin needs to compile sass in production (this is not coming from my own code or assets.) I would like to know if there's a way to disable it, or maybe precompile these assets. But this seems to be the default behavior in the latest version of ActiveAdmin.

I think it might be difficult to consistently reproduce this crash, but I could try to put together a Docker image and test Rails app.

[ndbroadbent]

Ahhhhhh this is happening because I was lazy and added some inline SCSS in my HAML template:

:scss
  .filters {
    margin-top: 18px;
    margin-bottom: 10px;
    margin-left: 39px;

    .button {
      font-size: 12px;
    }
  }

At least this explains why my ActiveAdmin view is running sassc. So a workaround is to just move this style into my precompiled assets, which should solve the problem for now.

[ndbroadbent]

I've tracked down the cause of the issue. This started happening after I updated the sqreen Ruby gem (https://www.sqreen.com). This gem adds some instrumentation and wraps some methods in Rails, so their code was causing this multi-threading issue that resulted in a segmentation fault (probably a similar issue to sprockets, and other issues that people have reported.)

If anyone wants to look into this, you can see all the middleware and instrumentation lines in the stack trace: sqreen-1.18.1. The sqreen code isn't open source on GitHub, but you can view the source code by running gem install sqreen and then gem open sqreen.

[lloeki]

@ndbroadbent Sqreen Ruby agent engineer here. We tracked down this issue previously to conflicting unmangled libc++ symbols, and fixed it in libsqreen 0.3.0.0.3 (a native dependency of the Sqreen gem). This should be resolved with a bundle update.

BTW we’re working on progressively opening our GitHub repos but we need a git history audit first.

[Lorel]

@ndbroadbent fix confirmed here 🎉
just deployed the version 1.18.2 of the gem sqreen, using libsqreen version 0.3.0.0.3, and I've just succeeded to precompile my assets