Metrics double_registration (storage_log_written_bytes)

[BenPope]

FAIL test: RandomNodeOperationsTest.test_node_operations.enable_failures=True (3/153 runs)
failure at 2022-12-24T07:32:38.127Z: <BadLogLines nodes=docker-rp-12(1) example="ERROR 2022-12-24 07:14:49,202 [shard 0] cluster - controller_backend.cc:722 - [{kafka/topic-abletlvgrs/12}] exception while executing partition operation: {type: update, ntp: {kafka/topic-abletlvgrs/12}, offset: 476, new_assignment: { id: 12, group_id: 127, replicas: {{node_id: 1, shard: 0}, {node_id: 2, shard: 1}, {node_id: 4, shard: 1}} }, previous_replica_set: {{{node_id: 5, shard: 0}, {node_id: 2, shard: 1}, {node_id: 4, shard: 1}}}} - seastar::metrics::double_registration (registering metrics twice for metrics: storage_log_written_bytes)">
on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/20343#018542d1-a236-4ea4-aba2-6f4e33c128ea

[jcsp]

Without having inspected the logs, this is probably a case of quickly deleting then recreating the same NTP, such that the new storage log is getting created before the old one is destroyed.

[0xdiba]

If it helps in any way, we've seen this happen in the wild with other metrics too
eg: [shard 0] seastar - Exceptional future ignored: seastar::metrics::double_registration (registering metrics twice for metrics: kafka_consumer_group_consumers)

[BenPope]

This is related: #5939

[mmaslankaprv]

it looks like the partition shared pointer is being held alive by fetch request handler. Even tho the partition is removed and all the subsequent reads will fail, keeping pointer alive prevents metrics from being deleted. Maybe we we should explicitly deregister ntp metrics in disk_log_impl::remove() ?

[piyushredpanda]

Is that a question for @jcsp?

[mmaslankaprv]

@jcsp what do you think ?