es-index-cleaner:latest is not working as expected #3228
Comments
|
@maksim-paskal, do you mind providing logs and more information on what is not working? cc @pavolloffay |
|
Sure. As I mension before - we use jaeger installation in kubernetes with elasticsearch backend and rollover - we install jaeger with helm chart with enabled esIndexCleaner to clear old indexes in our elasticsearch backend to reduce disk usage ...
esIndexCleaner:
enabled: true
schedule: "0 * * * *"
numberOfDays: 1
extraEnv:
- name: ES_TLS_ENABLED
value: "true"
- name: ES_TLS_SKIP_HOST_VERIFY
value: "true"
- name: ROLLOVER
value: 'true'
...It's create CronJob with this specification ...
spec:
containers:
- args:
- "1"
- http://elasticsearch:9200
env:
- name: ES_TLS_ENABLED
value: "true"
- name: ES_TLS_SKIP_HOST_VERIFY
value: "true"
- name: ROLLOVER
value: "true"
- name: ES_SERVER_URLS
value: http://elasticsearch:9200
- name: ES_USERNAME
value: elastic
- name: ES_PASSWORD
valueFrom:
secretKeyRef:
key: password
name: jaeger-elasticsearch
- name: ES_INDEX_PREFIX
value: jaeger-prod
image: jaegertracing/jaeger-es-index-cleaner:latest
...It's successfuly works until today - we notice that our disk usage significantly increased - we realize that old jaeger indexes not removing - cronjob logs I think that the problem in es-index-cleaner golang implementation (#3192) this implementation do not use environment variables that was used in old python implementation - we clear old indexes only with dowgrading to python implementation |
|
@maksim-paskal thanks for reporting this. Based on the provided configuration it seems that you want to run the index cleaner every hour https://crontab.guru/#0__** and keep the data for one day? |
|
yes. We use only last 24 hours for traces. |
The golang implementation uses the same env variables. It's configured via cobra/viper like in the other jaeger binaries. Here are the docker image SHAs it seems that docker pull jaegertracing/jaeger-es-index-cleaner:1 1 ↵ ploffay@fedora
1: Pulling from jaegertracing/jaeger-es-index-cleaner
4e9f2cdf4387: Already exists
bd863d97c41b: Already exists
7abde820361d: Pull complete
Digest: sha256:520279aa82234dd08e3439907e2ec229c6501524a7886f133fd7c897c86667c8
Status: Downloaded newer image for jaegertracing/jaeger-es-index-cleaner:1
docker.io/jaegertracing/jaeger-es-index-cleaner:1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~ » docker pull jaegertracing/jaeger-es-index-cleaner:1.26 ploffay@fedora
1.26: Pulling from jaegertracing/jaeger-es-index-cleaner
Digest: sha256:520279aa82234dd08e3439907e2ec229c6501524a7886f133fd7c897c86667c8
Status: Downloaded newer image for jaegertracing/jaeger-es-index-cleaner:1.26
docker.io/jaegertracing/jaeger-es-index-cleaner:1.26
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~ » docker pull jaegertracing/jaeger-es-index-cleaner:1.26.0 ploffay@fedora
1.26.0: Pulling from jaegertracing/jaeger-es-index-cleaner
Digest: sha256:520279aa82234dd08e3439907e2ec229c6501524a7886f133fd7c897c86667c8
Status: Downloaded newer image for jaegertracing/jaeger-es-index-cleaner:1.26.0
docker.io/jaegertracing/jaeger-es-index-cleaner:1.26.0 |
|
It seems that image I check pod with image todays logs of @pavolloffay I think the problem in environment env:
- name: ES_INDEX_PREFIX
value: jaeger-prodit should be env:
- name: INDEX_PREFIX
value: jaeger-prodBut I do not understand how it can work in python implementation early... Today I try to run it with kubectl run test --image jaegertracing/jaeger-es-index-cleaner@sha256:5e0adb0e8623dd334709943999a6b9d3e283090efcc05a7f90c34804a7e71758 --restart Never -it --rm --command sh
# and run
export ES_TLS_ENABLED="true"
export ES_TLS_SKIP_HOST_VERIFY="true"
export ROLLOVER="true"
export ES_SERVER_URLS="http://elasticsearch-master.elk.svc.cluster.local:9200"
export ES_USERNAME="test"
export ES_PASSWORD="test"
export ES_INDEX_PREFIX="jaeger-prod"
python3 /es-index-cleaner/esCleaner.py "1" http://elasticsearch-master.elk.svc.cluster.local:9200result with go implementation, with this environment - @pavolloffay Sorry for this noise, it seems some elasticsearch specific... It's not a bug for go implementation |
|
The python implementation uses docker run jaegertracing/jaeger-es-index-cleaner:1 help ploffay@fedora
Error: wrong number of arguments
Usage:
jaeger-es-index-cleaner NUM_OF_DAYS http://HOSTNAME:PORT [flags]
Flags:
--archive Whether to remove archive indices. It works only for rollover
--es.password string The password required by storage
--es.tls.ca string Path to a TLS CA (Certification Authority) file used to verify the remote server(s) (by default will use the system truststore)
--es.tls.cert string Path to a TLS Certificate file, used to identify this process to the remote server(s)
--es.tls.enabled Enable TLS when talking to the remote server(s)
--es.tls.key string Path to a TLS Private Key file, used to identify this process to the remote server(s)
--es.tls.server-name string Override the TLS server name we expect in the certificate of the remote server(s)
--es.tls.skip-host-verify (insecure) Skip server's certificate chain and host name verification
--es.username string The username required by storage
-h, --help help for jaeger-es-index-cleaner
--index-date-separator string Index date separator (default "-")
--index-prefix string Index prefix
--rollover Whether to remove indices created by rollover
--timeout int Number of seconds to wait for master node response (default 120)
2021/09/09 14:27:47 wrong number of argumentsThe
It can be a bug in golang implemenation as well. The date comparison is done in our implementation not in ES. I will try to have a look at it. |
|
I have the same error. With the tag: Thanks! |
|
I have submitted #3261 to fix this issue. But I am not sure if the PR fixes the issue or not. I have built docker image with changes in that PR - |
|
Hi, it doesn't work: It should delete this index Thanks |
|
Here are the commands that I used to unsuccessfully tried to reproduce the issue (the indices were deleted): docker run -it --rm -e "ES_JAVA_OPTS=-Xms2g -Xmx2g" -p 9200:9200 -p 9300:9300 -e "http.host=0.0.0.0" -e "discovery.type=single-node" -e "xpack.security.enabled=false" --name=elasticsearch docker.elastic.co/elasticsearch/elasticsearch:5.6.10
docker run --net=host -e INDEX_PREFIX=jaeger-prod jaegertracing/jaeger-es-rollover:1.26.0 init http://localhost:9200
docker run --net=host -e CONDITIONS='{}' -e INDEX_PREFIX=jaeger-prod jaegertracing/jaeger-es-rollover:1.26.0 rollover http://localhost:9200
docker run --net=host -e INDEX_PREFIX=jaeger-prod -e ROLLOVER=true jaegertracing/jaeger-es-index-cleaner:1.26.0 0 http://localhost:9200
# {"level":"info","ts":1631540557.3836107,"caller":"./main.go:83","msg":"Indices before this date will be deleted","date":1631577600}
# {"level":"info","ts":1631540557.38408,"caller":"./main.go:98","msg":"Deleting indices","indices":[{"Index":"jaeger-prod-jaeger-span-000001","CreationTime":"2021-09-13T13:42:19.232Z","Aliases":{"jaeger-prod-jaeger-span-read":true}},{"Index":"jaeger-prod-jaeger-service-000001","CreationTime":"2021-09-13T13:42:19.476Z","Aliases":{"jaeger-prod-jaeger-service-read":true}}]} |
|
@pavolloffay my issue resolved with changing pod env name from @ichasco-heytrade did you use rollover? https://www.jaegertracing.io/docs/1.26/deployment/#elasticsearch-rollover |
|
@ichasco-heytrade did you configure |
|
Hi, yes, I am using rollover. I have tested with the env I have tested also with the |
|
For me only works with:
I don't need to set |
It does not seem like you are using rollover:
The rollover indices do not have a date in the index name. |
The job is scheduled |
|
The job might be scheduled but the indices list from the es-cleaner does not show that the deployment was using the rollover - #3228 (comment) |
|
@ichasco-heytrade here is my reproducer for #3228 (comment) to remove PR #3264 To remove the docker run --rm -it -e INDEX_PREFIX=jaeger pavolloffay/jaeger-es-index-cleaner:PR3261 3 <es-url>Do not enable rollover bc the index is not managed by the rollover. |
|
Now works!. Thanks! |
|
thanks for the configuration. I am closing this. |
|
Hi, This is still failing. The PR hasn't been merged. Is there any reason? Thanks! |
|
I resolved my issue by adding the ROLLOVER='true' environment variable, using the golang version of the index cleaner |
We use jaeger installed in kubernetes cluster with elasticsearch backend with rollover and now - old indexes not removed. Helm chart use jaegertracing/jaeger-es-index-cleaner:latest image for removing old indexes - and this not working as expected
Expected behavior
Old indexes must be removed as described in documentation
Version (please complete the following information):
Additional context
As hotfix - downgrade to image
jaegertracing/jaeger-es-index-cleaner:1#3192
The text was updated successfully, but these errors were encountered: