Deduplication of keys

[MarcusNoble]

I've encountered a problem where it's possible to have the object sent from Fluent Bit to contain duplicated keys.

Using the following filters results in this issues:

    [FILTER]
        Name record_modifier
        Match kube.*
        Record cluster_name example-cluster-name

    [FILTER]
        Name nest
        Match kube.*
        Operation nest
        Wildcard cluster_name
        Nest_under kubernetes

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off

The resulting object sent for processing looks something like this (unrelated keys removed for brevity):

{
    "kubernetes": {
      "annotations": {
        "kubernetes_io/psp": "eks.privileged",
        "linkerd_io/inject": "enabled"
      },
      "container_name": "newrelic-infra",
      "labels": {
        "app_kubernetes_io/name": "newrelic"
      },
      "namespace_name": "kube-monitoring"
    },
    "kubernetes": {
      "cluster_name": "example-cluster-name",
    }
}

I'm not sure how the data from the C code is converted to Go but I've confirmed that these duplicate keys do get passed through.

[MarcusNoble]

So the more I think about this I think it's the responsibility of the Fluent Bit Nest filter plugin, rather than this output plugin.
It seems there are already related issues open:

• Nesting a key under an existing key results in duplicate keys fluent/fluent-bit#1177
• Duplicated JSON fields on record fluent/fluent-bit#1835

[hossain-rayhan]

Hi @MarcusNoble ,
Thanks for linking the original issues from the fluent-bit core. It seems like those issues are still open and fluent-bit maintainer is working for them. As we don't have a plan to implement it as a output plugin feature, I am closing this issue. Feel free to re-open if you have any question.

[edsiper]

FYI:

I've pushed fix 1d148860a8825d5f80aef40efd0d6d2812419740 in Fluent Bit to handle duplicated keys in a map. The workaround is basically: when converting the data to JSON, if a duplicated key is found, only pack the last value found for that key.

This will be part of v1.6 release next week.

[PettitWesley]

Thank you @edsiper!