[elastic-agent] apm-server permissions for forking java-attacher

[stuartnelson3]

Recent apm-server releases now come with the java-attacher.jar bundled along, so that an apm-server can fork and run a configured java-attacher.

In standalone mode, this requires that the user provide permissions to the apm-server to fork a process, otherwise the operation errors:

{"log.level":"error","@timestamp":"2021-12-29T08:53:47.771Z","log.logger":"beater","log.origin":{"file.name":"beater/beater.go","file.line":439},"message":"failed to run java attacher: fork/exec /usr/bin/java: operation not permitted","service.name":"apm-server","ecs.version":"1.6.0"}

Currently, when running apm-server in managed mode (under the elastic-agent), these permissions are not set, even though the user has the ability to configure the java-attacher from kibana.

Are there plans for how to address this?

@felixbarny @ruflin

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[jlind23]

@stuartnelson3 from what i understand even if you set those specific permissions through Kibana it does not end up in the Agent config right?

[stuartnelson3]

I'm not sure if setting these permissions is possible through kibana; i believe all services/integrations are running with a static set of allowed systemcalls.

[stuartnelson3]

Revert the following PR once this issue has been resolved: elastic/kibana#122164

[jlind23]

I do need @ruflin opinion here :-)

[stuartnelson3]

After doing some straceing, adding the following syscalls allows apm-server to fork the java-attacher process:

        - "fork" # apm-server
        - "execve" # apm-server
        - "prlimit64" # java-attacher
        - "socketpair" # java-attacher
        - "vfork" # java-attacher
        - "getegid" # java-attacher

The full policy is available here: https://gist.github.com/stuartnelson3/0fecf0f25198b3b791aa3faa59f96cae

Only the commented syscalls (copied above) are not part of the default amd64 policy.

For completeness, we would have to also have a list for each of the following:

• 386
• arm
• arm64

[ruflin]

In standalone mode, this requires that the user provide permissions to the apm-server to fork a process

Can you share an example on how this is exactly done?

When running under Elastic Agent, what is the expected behaviour. Should the apm-server always have these permissions or only when enabled through the UI? @Mpdreamz Would be great if you could also chime in here.

[jlind23]

@stuartnelson3 We will discuss about it during the 8.2 release cycle. We will then get in touch with you to find a proper solution.

[simitt]

Adding some more context to this. As of today, standalone beats offer a beta feature to enable seccomp filtering and configure a seccomp policy (default action and syscalls with actions and names); for example metricbeat seccomp support. Despite this being a beta feature, seccomp is enabled by default and applies a default policy, if not customized via configuration.

When the Elastic Agent manages processes, there is currently no way of providing a custom seccomp policy per process (beat), but seccomp is still enabled by default. This locks the permissions to the default policy and is more restrictive than it was for standalone beats.
What we are trying to figure out is:

• Is the current behavior intentional or should seccomp filtering not be enabled by default for every package/process?
• Are there any plans to work on a strategy towards how custom seccomp policies can be provided? I believe this would be interesting on a per package level but also to be user configurable.

This is currently blocking the APM team from making progress with enabling the Java Auto Attacher feature (elastic/apm-server#7023).

[ph]

Is the current behavior intentional or should seccomp filtering not be enabled by default for every package/process?

Sorry, this is not intentional, to clarify even more I am still surprised seccomp was still in beta, it's been in the code for at least ~3 years, this is also something we should take care.

We do not have a plan or strategy at how we would lock down elastic-agent or process, custom seccomp policies seems a good way to do it, and making it user-configurable is something we should do.

AFAIK Elastic-Agent doesn't enable seccomp rules, only beats does it at the moment, so we should be able to find a way to enable that at the apm-server level.

We had a discussion in elastic/beats#29809 to allow developers to a apm-server.elastic-agent.yml were you set up configuration that elastic-agent would pick up. I believe this could solve your current issue.

We can also discuss if we can create seccomp rules associated with a specific integration but that would require a more deeper discussion, I think we should eventually need it.

I've created #90 on our side to track the work on it.

[andrewkroh]

Yeah, seccomp is definitely stable so the beta doc label can be removed.

We can also discuss if we can create seccomp rules associated with a specific integration but that would require a more deeper discussion, I think we should eventually need it.

The current implementation has the Beat process setting its seccomp policy early in the startup. Once it's set it cannot be changed unless the process is restarted. So if Agent wanted to modify the policy to align with the requirements of a particular integration or use-case then Agent would be required to restart the process with a different seccomp policy. I wanted to mention this because IIUC the Beat processes are not restarted when integration policies are changed.

[felixbarny]

I have just realized that the attacher also forks itself, which might require additional permissions.

Background:
Attaching a Java agent is only possible if the attacher and the target JVM run under the same OS user. Even the root user can't attach to JVMs that run under another user. Therefore, when the attacher wants to attach to a JVM, it needs to switch user (via sudo -u) and call itself.

Question: when we allow APM Server to fork the java attacher, what are the seccomp permissions for the java attacher? Can it fork itself? To we need to add another explicit inclusion rule?

[ph]

I wanted to mention this because IIUC the Beat processes are not restarted when integration policies are changed.

This is what I though, I believe we could force a restart on theses options change, I believe we do it for output. IIRC.

[ph]

@felixbarny from the secomp(2) documentation.

If fork(2) or clone(2) is allowed by the filter, any child
processes will be constrained to the same system call
filters as the parent.  If execve(2) is allowed, the
existing filters will be preserved across a call to
 execve(2).

I have to dig deeper but you can restrict a child more looking at https://stackoverflow.com/questions/65153051/install-seccomp-filter-in-child

[felixbarny]

I wanted to mention this because IIUC the Beat processes are not restarted when integration policies are changed.

This is what I though, I believe we could force a restart on theses options change, I believe we do it for output. IIRC.

IINM, APM Server already restarts when anything in the policy changes. @axw can you confirm?

[axw]

@felixbarny no, policy changes don't restart the process. The server creates a new listening socket, new aggregation processors, etc., to which new requests are sent; and then drains the existing listening socket's requests. We could restart the process, but I think I'd rather just always allow the java-attacher program to be executed.

[stuartnelson3]

Background:
Attaching a Java agent is only possible if the attacher and the target JVM run under the same OS user. Even the root user can't attach to JVMs that run under another user. Therefore, when the attacher wants to attach to a JVM, it needs to switch user (via sudo -u) and call itself.

Looking at the code, it appears that the root user is exempt from this check, and allowed to attach to a processor require of the user/group that started it? https://hg.openjdk.java.net/jdk/jdk/file/932418820c80/src/jdk.attach/linux/native/libattach/VirtualMachineImpl.c#l161

Question: when we allow APM Server to fork the java attacher, what are the seccomp permissions for the java attacher? Can it fork itself? To we need to add another explicit inclusion rule?

#95

The attacher requires vfork to fork itself.

[felixbarny]

Looking at the code, it appears that the root user is exempt from this check, and allowed to attach to a processor require of the user/group that started it?

Great question, Stuart. The capability for a root user to attach to other users has been added via https://bugs.openjdk.java.net/browse/JDK-8197387 and is not available for all the Java versions that we support.
Also, in the future we might want to avoid using the root user in favor of a dedicated user that can be configured to be able to switch to a subset of users.
But we could potentially try to attach without forking if run as root and fall back to forking on error.

[felixbarny]

Also, on Windows and macOS, the temporary directory is not shared across users. Instead, each user has their own and the name of the directory is not necessarily known to the root user. The attachment procedure involves creating a temp file and then writing to a socket that the target VM creates in the temp dir. Therefore, to my knowledge, on Windows and macOS, there's not a good alternative around self-forking/switching user/sudo -u.

[felixbarny]

@stuartnelson3 can this be closed now that we have decided to remove seccomp from APM Server (elastic/apm-server#7305)?

[stuartnelson3]

yep! this is no longer needed.