Add stdout or file out with the latest telemetry metrics

[robert-zaremba]

Summary

Follow up to: #12447

Today, we can inspect telemetry only through Prometheus or REST API server.

Problem Definition

For debugging, or internal purposes it would be helpful to be able to:

• start in memory telemetry sink without enabling API server
• be able to inspect telemetry

Proposal

Add additional metric display methods.
Extend app config (app.toml):

[telemetry]
# print interval metrics to stdout if set to true
stdout = false 

# save interval metric to file if not empty. will overwrite the file during each interval write
fileOutput = "filename"

The stdout mode will mix the metrics output in the log output (using INFO) with non-pretty printed JSON formatted text.

The fileOutput will save metrics to file with pretty printed JSON format.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[alexanderbez]

Is there a reason you can't just hit the endpoint?

[robert-zaremba]

Is there a reason you can't just hit the endpoint?

If API server is not active then you can't do it. Workaround would be to enable api server and not export the ports in the firewall

[alexanderbez]

Yes, I see that as a perfectly fine solution. As long as this file approach isn't too much technical overhead, then it's fine IMO

[robert-zaremba]

It's not an overhead. It's just dumping data to a file. That's fast and happens in parallel in a go routine. Moreover by default it will be disabled.

[tac0turtle]

i dont fully see the need for this, what is the overhead that turning on the server is causing?

[tac0turtle]

closing this as we should opt to use the default instead of implement custom items. This is inline with reducing maintenance burden of the current repo