Move labels and annotations under kubernetes.namespace.

[MichaelKatsoulis]

What does this PR do?

As part of #13911 , after discussions in #16558 (comment)
this PR

• Updates kubernetes.namespace from keyword to group field.
• Moves kubernetes.namespace which was the name of the namespace to kubernetes.namespace.name field.
• Renames namespace_labels and namespace_annotations metadata to namespace.labels and namespace.annotations

Why is it important?

Until now namespace labels and annotation where under namespace_labels and namespace_annotations.
As those fields belong to namespace they should be under the namespace group field.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

A. Verify that events from add_kuberentes_metadata are enriched properly

0. Use updated manifests from https://github.com/elastic/beats/tree/master/deploy/kubernetes
1. Deploy Filebeat on kubernetes (cluster should be created with kubadm [ie a kind cluster] ) and configure log collection like this:

    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
            add_resource_metadata:
              namespace:
                include_annotations: ["description"]

Annotate default namespace like this kubectl annotate ns default description='it is the default'
2. Ensure that kubernetes.namespace.name, kubernetes.namespace.labels, kubernetes.namespace.annotations are being populated properly.

B. Verify that events from autodiscover provider are enriched properly

0. Use updated manifests from https://github.com/elastic/beats/tree/master/deploy/kubernetes
1. Deploy Filebeat on kubernetes (cluster should be created with kubadm [ie a kind cluster] ) and configure log collection like this:

    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log

2. Ensure that kubernetes.namespace.name, kubernetes.namespace.labels, are being populated properly.

Related issues

• Relates Define mapping for new k8s resources #16558

Screenshots

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[botelastic]

This pull request doesn't have a Team:<team> label.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-10-11T10:53:23.456+0000

• Duration: 149 min 31 sec

• Commit: 9776bd9

Test stats 🧪

Test	Results
Failed	0
Passed	53413
Skipped	5324
Total	58737

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

[ChrsMark]

Let's track this one and all related changes at #16483

[mergify]

This pull request does not have a backport label. Could you fix it @MichaelKatsoulis? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[P1llus]

jenkins run the tests

[ChrsMark]

@kaiyan-sheng this is part of #16483 and #13911 but I guess it needs an update in Kibana's side, at Inventory view where we can group Pods by namespace so do you know who we need to inform about this?

[ChrsMark]

lgtm, good job here!
Please add a changelog entry before merging.

[ChrsMark]

If not intentional please revert it.

[kaiyan-sheng]

@ChrsMark Let's create a Kibana issue for this since it will break the inventory UI in master branch. I would ping @simianhacker for this 🙂