Remove viper and mapstructure from rpk

[r-vasquez]

Cover letter

This PR removes the dependency of viper and mapstructure for our config file handling. Now we have control over Write and Read operations in our config file.

We introduced in #4894 the Weak Types, which allow us to handle weakly typed parameters.

It also includes the upgrade from Yaml.v2 to Yaml.v3, one of the key major changes in this library is the indentation of 4 instead of 2. So our config files will look different (but valid although)

Fixes #4531, Fixes #4176, Fixes #5202

Also, now rpk prevents the user to change ownership of config files, so with this it:

Fixes #4046, Fixes #3064, Fixes #1919

What is changing

Config File:

Now you will be able to detect if your configuration file contains some weak types that were allowed before but now we want to enforce strict typing. e.g:

redpanda:
    kafka_api:
        name: test
        address: 127.0.0.1
        port: 4664

will be parsed as a YAML list

redpanda:
    kafka_api:
        - name: test
          address: 127.0.0.1
          port: 4664

As you can see above, the indent is now with 4 spaces instead of 2, this is enforced in the new yaml.v3 package and still valid.

Old code will write redpanda.seed_servers.node_id, this property is no longer needed/supported. We are stripping it out from the config, so:

- node_id: 1
  host:
    address: 192.168.10.1
    port: 33145
- node_id: 2
  host:
    address: 192.168.10.2
    port: 33145
- node_id: 3
  host:
    address: 192.168.10.3
    port: 33145

Will become:

- host:
    address: 192.168.10.1
    port: 33145
- host:
    address: 192.168.10.2
    port: 33145
- host:
    address: 192.168.10.3
    port: 33145

Behavior changes/notes:

When running params.Load() method we are doing some backward compatibility changes, one of them is to copy rpk.tls to rpk.kafka_api.tls and rpk.amin_api.tls. This is already in place and working for some commands, with this PR will work for ALL commands that load the config file.

The old manager created a directory when flag --config was passed, we will keep this behavior for now.

Also removed an undocumented and unused way of changing config parameters in camelCase using --format json in rpk config set.

Ducktape Tests:

• Changed expected configuration order and indentation.
• Added logs for failure scenarios.
• Changed expected value for redpanda.seed_servers (see above)

K8S Configurator:

• Removed manager as a dependency to use only config.Params and config.Config methods to load and manipulate configuration.

RPK Go Code:

• Removed Viper and Mapstructure
• Upgrade yaml from v2 to v3
• Removed config.Manager interface and it’s implementations.
• Deleted old/unused config/license.go

Release notes

• rpk preserves now the file ownership of the configuration file.
• rpk will print warnings of errors in configuration file parameters.
• rpk: bugfix, redpanda start no longer strips empty tls values.

[twmb]

Some preliminary comments and questions!

[twmb]

Why doesn't this properly call weak unmarshaling for socket addresses, etc? I think you brought up that setting redpanda.seed_servers is different than setting redpanda with a value containing seed servers?

[r-vasquez]

It doesn't call weak unmarshaling for any of the one_or_many weak types, since we created other intermediary types for them:

[]NamedSocketAddress -> NamedSocketAddresses
[]SeedServer -> SeedServers
... 

We use the intermediary types in the general config.Config type, not in every field, e.g : https://github.com/redpanda-data/redpanda/blob/dev/src/go/rpk/pkg/config/weak.go#L316L335

[twmb]

I have an idea for how to solve this for the yaml case, but not the json. In the yaml case, when we return, we could have a huge type switch:

case []string => return weakStrings(xyz)
case []NamedSocketAddress => return namedSocketAddresses(xyz)

this only works for yaml because weak decoding is only implemented for yaml
this theoretically could be supported for json in a followup PR if we want,
but I think this should be tracked as a separate issue: allow setting slices with one element in set

[r-vasquez]

Created: #5264

[r-vasquez]

Failure in: https://buildkite.com/redpanda/redpanda/builds/11813#0181ad2c-892a-4d2e-a49e-3e5a71f4cc95

Possibly related to: #2406, checking if the same issue or if I should create a new one.

[twmb]

Awesome

[emaxerrno]

@r-vasquez awesome work!