Documentation on Logging?

[ethernoy]

Is your proposal related to a problem?

I use Thanos receive in my cluster with 6 Prometheus instances writing to it. The Thanos receive pod has an ephemeral storage limit of 700MB, a PVC storage of 50GB, and was started in debug log level.
When I stop the receiver, wait several hours, then start it again, its console is flooded with out of order sample debug log with full request body, then soon it is evicted due to lack of ephemeral storage, and be flooded with out of order sample again. I suspect the ephemeral storage was used up by logging file, but failed to found one. To workaround this issue, I turned off debug logging, then start receive again, then the issue is gone.

Several questions in mind:

1. There is no documentation about logging in Thanos, hence I can't find whether there is local log file, or where it is.
2. There is no spec about logging in Thanos apart from log level, hence I can't disable local logging if there is any.

Describe the solution you'd like

Some explanation on logging in Thanos?

[yeya24]

Hi, Thanos doesn't have any log file and all logs are written to stdout.
How is your ephemeral storage mounted to the pod?

[ethernoy]

Hi, Thanos doesn't have any log file and all logs are written to stdout.
How is your ephemeral storage mounted to the pod?

Actually there is no explicit mounting of ephemeral storage to the pod. I only mounted a pvc and a volume for secret to the pod. Will Thanos receiver stores local cache data?

[yeya24]

Hi, Thanos doesn't have any log file and all logs are written to stdout.
How is your ephemeral storage mounted to the pod?

Actually there is no explicit mounting of ephemeral storage to the pod. I only mounted a pvc and a volume for secret to the pod. Will Thanos receiver stores local cache data?

Receiver is just a TSDB and it only uses the data dir IMO. You have a pvc so it is fine.

[ethernoy]

Hi, Thanos doesn't have any log file and all logs are written to stdout.
How is your ephemeral storage mounted to the pod?

Actually there is no explicit mounting of ephemeral storage to the pod. I only mounted a pvc and a volume for secret to the pod. Will Thanos receiver stores local cache data?

Receiver is just a TSDB and it only uses the data dir IMO. You have a pvc so it is fine.

But the receiver got evicted for quite several times due to lack of ephemeral storage when Prometheus instances were trying to write old TSDB data to receiver. I was using 0.17.1 by the way.

[kakkoyun]

Sounds fair to me. We should definitely document more. cc @yashrsharma44

[kakkoyun]

Let's have a common place to put logging documentation and reference to this section from each command. operating/observability looks like a good candidate. i.e operating/observability/tracing|logging|mixin ...

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗

[yashrsharma44]

Let's have a common place to put logging documentation and reference to this section from each command. operating/observability looks like a good candidate. i.e operating/observability/tracing|logging|mixin ...

I was thinking of writing a docs section for the query logging and logging in general, should I go ahead with the above structuring for the docs for logging?
cc @kakkoyun

[sbirl]

I believe something more needs to be written up, not only for logging, but for all options.

I am new to Thanos, and I would love to know what it is capable of. (I am to type to really dig in deep to a product)
The tiny descriptions associated with each parameter found under README do not provide enough information.
Moreover, I can find no links to more detailed information -- not even via search engines.

Example 1: logFormat. What options are available? The default logfmt tells me nothing.
Example 2: bucketweb.extraFlags. I cannot find anything about any flags if I search (Google, DDG, etc) for "bucketweb.extraFlags".

PLEASE provide a place to detailed documentation for all parameters!