[Elastic Agent] Helm Charts

[ph]

The infra team is currently responsible for maintaining the k8s Helm carts for our beats, with the move to the Elastic Agent we would need to take care of that in the team.

Steps TBD

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ph]

@exekias @blakerouse This should be included in the K8s group discussion.

[kuisathaverat]

@ph we have made one for the clusters maybe you can reuse stuff.

[ph]

@kuisathaverat oh nice, we should probably make it public somewhere. :)

[kuisathaverat]

@kuisathaverat oh nice, we should probably make it public somewhere. :)

I think https://github.com/elastic/helm-charts is the place

[nkammah]

The Stack Infra release team handles deploying the elastic/helm-charts. Since the helm-charts are part of the Unified Release Process, the repository follows the stack branching model - you can read more about it here.

The main thing we need from you is to have a PR against the main branch for the elastic-agent chart. The Stack Infra release team will handle the backport to 7.x and 7.11 once it's merged on the main branch).

We do run tests against the charts - you can see more in the ES chart for example.

cc @jmlrt

[nkammah]

bump cc @kuisathaverat @ph

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[kuisathaverat]

we need the Elastic Agent helm chart to our PoC with Otel so we have started to make a couple of Helm charts based on the k8s deployment at https://github.com/elastic/beats/tree/master/deploy/kubernetes/elastic-agent-standalone and https://github.com/elastic/beats/tree/master/deploy/kubernetes/elastic-agent , the PR with the helm chart is WIP

[kuisathaverat]

I will need some help here to configure the Elastic Agent. So far, I have two Helm charts fleet and standalone, both deploy and daemonset, or only one of them. The Elastic Agent starts and report to Elasticsearch its logs and metrics. Here is where I need some help, I found some stuff that I dunno if it is correct:

• standalone:
  • DaemonSet
    • Report k8s nodes metrics
    • Report pod metrics
    • Report Elastic Agent logs

• Deployment
  • Report Elastic Agent pod logs (no metrics)

So I have the k8s node cluster metrics and the Elastic Agent logs, and pod metrics, I do not have any log other than the Elastic Agent log. The pod deployment only report info about it self I think is not useful.

• fleet (system+kubernetes integrations enabled):

• DaemonSet
  • Report k8s nodes metrics
  • Report Elastic Agent logs

• Deployment
  • Report Elastic Agent pod logs (no metrics)

We only have k8s cluster node metrics and Elastic Agent logs+metrics, the deployment report only data about it self not useful.

The use case I want to resolve is the same I have resolved on the oblt clusters with Filebeat and Metricbeat, on those clusters we have deployed a Daemonset deploy per k8s cluster node and only a Deployment for the whole cluster, this configuration is the same for Filebeat and Metricbeat. So in a k8s cluster of 3 nodes we have 4 Filebeat (3 DaemonSet + Deployment) and 4 Metricbeat. This configuration give us k8s nodes cluster metrics, pods Metrics, and pods logs. How I can have a similar configuration with the Elastic Agent? I have check the documentation but does not have too much about k8s.

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[ph]

@kuisathaverat I will let @ChrsMark answer this, he has been pushing the k8s story from our side.

[kuisathaverat]

We are testing the Elastic Agent standalone deployed with this Helm chart h in our Opentelemetry PoC, it is reporting logs and metrics correctly with the latest changes in the configuration. What are the next steps to make it official?

We have another version that works with fleet but it only reports metrics, I guess, this one will need changes to with the fleet server and grab the logs.

cc @ph

[ph]

@kuisathaverat this is super thanks!

[ChrsMark]

That's great @kuisathaverat !

One question: I see that you use 7.12-SNAPSHOT right? Is there any reason for this? leaderelection provider will only work with 7.13.

[kuisathaverat]

the real deployment is using 7.13.0-SNAPSHOT I have to update the values to ake that version the default, but it is not urgent I guess we will change the version more times before we are in a final version.

[jmlrt]

What are the next steps to make it official?

👋🏻 @kuisathaverat, to make it official, this chart should be moved to a public repository (can be elastic/helm-charts along the other current charts or directly into Beats repo as for ECK chart.

Then we would need to update helm-charts-publish job to also handle Elastic Agent chart release and publish it to https://helm.elastic.co repo.

[maggieghamry]

@jmlrt @kuisathaverat is this chart available anywhere yet?

[kuisathaverat]

No, it is not public yet.

[maggieghamry]

@kuisathaverat thank you! I see all checks passed - do we know what's preventing this from getting integrated?

[kuisathaverat]

something is going on with filebeat and the log module that makes the filebeat process restart continuously, I think is something with GKE or k8s 1.19

[adammike]

is this available at all yet?

[kuisathaverat]

not officially, we are using it on the oblt clusters, maybe is time to make a PR to the official helm charts repo

[adammike]

I don't have access to that repository. I'm having a terrible time getting the elastic-agent to properly discover my Kubernetes logs in either fleet managed (preferred) or standalone mode.

[kuisathaverat]

After talking with @nkammah, the recommendation nowadays is to add the Helm Chart to the product repo in a folder like https://github.com/elastic/cloud-on-k8s/tree/master/deploy

@ph If you agree I can make a couple of PRs to the beats repo to put those helm charts in the older https://github.com/elastic/beats/tree/master/deploy/kubernetes into a new folder named charts or helm-charts

[ruflin]

+1 on moving the helm charts into the repository to develop these in public. Having it in the public repo should not mean directly we support these. That part on how we support it we still need to figure out (@nimarezainia @jlind23 ). @kuisathaverat Would be great if you could take on the move.

@adammike Sorry the repos that @kuisathaverat are private but hopefully this gets resolved with the above discussion.

[kuisathaverat]

I have made a couple of draft PRs, they are WIP, I have to create an example of deploy and the README for both Helm Charts

[nkammah]

@kuisathaverat any update on the above ?

[DaveSys911]

@kuisathaverat @nkammah Watching this, as we also need fleet helm charts.

Linking in @kuisathaverat draft prs

Elastic-agent managed helm chart:
#28102

Elastic-agent standalone helm chart:
#28103

Have a few questions:

• Any reason to have separate helm charts for standalone/fleet managed ?

• Would you like some help finishing & merging those charts?

[nkammah]

Hi @DaveSys911 - We're in the process of migrating the ownership of the o11y helm-charts to the Observability solution. The o11y robots will be assisting the product teams in getting up to speed (cc @jlind23 @ruflin) - a kick-off meeting will be happening in the coming days.

[simitt]

@ph @ruflin @kuisathaverat can you please confirm that the plan is to support apm-server with the Elastic Agent Helm charts, as it was suggested in https://github.com/elastic/observability-robots/issues/1002#issuecomment-1029911813?

[kuisathaverat]

@ph @ruflin @kuisathaverat can you please confirm that the plan is to support apm-server with the Elastic Agent Helm charts, as it was suggested in https://github.com/elastic/observability-robots/issues/1002#issuecomment-1029911813?

Currently is supported we deploy it with the Helm chart in the oblt clusters.

[simitt]

Apologies for asking once more Ivan, I am not certain if that means that the elastic-agent control plane team is owning the Elastic Agent Helm chart AND ensures that the APM Server is available automatically through this helm chart (either always by default or by choice).
Can someone please confirm this. It has been my understanding so far, just want to ensure there is no misunderstanding on this.

[venom02]

Hi, any update on the helm charts for the elastic agents?
Thank you

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[nickbabkin]

Hi,

looking for the working helm-chart for fleet server, am I at the right place? :)

[ph]

@simitt I am not sure I follow the issue, but if elastic agent is installed via helm if a user enables the apm integration it will get the server or not?

[simitt]

I was hoping for a solution where the user wouldn't need to take additional steps for installing APM Server, but have it installed and setup via just using the helm chart.

[kuisathaverat]

I was hoping for a solution where the user wouldn't need to take additional steps for installing APM Server, but have it installed and setup via just using the helm chart.

Since the configuration is not in the Elastic Agent pod, it makes things complicated. You need a Kibana instance up and running with the APM integration installed, an API key, a Service Token, and an Elastic Agent policy with APM integration support created before installing the Elastic Agent Helm chart.

[simitt]

That's what I was afraid of. The goal is to provide an easy setup option for users on k8s without manual interaction though. I don't believe that this is an APM specific requirement, but will be interesting for any other integration management too. What is the suggested solution for this if not planned via Helm charts?

[kuisathaverat]

What is the suggested solution for this if not planned via Helm charts?

We can workaround part of those requirements by adding k8s jobs, init containers, or something along these lines, but it will be kind of a hack that will depend on the Stack version and probably difficult to maintain backward compatibility (as we know), also there are requirements that are not possible to resolve like the Kibana instance with APM integration configured, some that probably we should not resolve like the API Key creation, so we can workaround the Service Token creation and the policy. The best way to resolve these requirements is with a k8s operator, in our case ECK, that is designed to resolve more complex orchestration scenarios.
There is a possibility of using ECK in the helm chart but IIRC forces to deploy the whole stack on k8s that probably is not the 100% of the cases.
IMO the Elastic Agent should provide some command in the CLI (or Elastic Agent configuration options) to resolve these prerequisites in the Elastic Agent side independently of Kibana, so if I have an API key and an Elastic Agent I can configure everything from the Agent side, it'd make the onboarding simpler and avoid to go backend forward between the Elastic Agent and Kibana.

[kuisathaverat]

@ruflin after the official communication about Elastic Helm Charts I think we can close this issue