redpanda admin config print --hosts 0 uses localhost

[travisdowns]

Version & Environment

Redpanda version: v21.11.10 (rev edc2c49)

What went wrong?

rpk redpanda admin config print --host 0 tries to access localhost rather than the configured API endpoint in the config YAML.

What should have happened instead?

The admin config should have been printed based on connecting to the local redpanda, as per the help output:

--host string   either an index into admin_api hosts to issue the request to, or a hostname

How to reproduce the issue?

1. Vanilla RP install from .deb
2. rpk redpanda admin config print --host 0
3. Gives error:

rpk redpanda admin config print --host 0
unable to request configuration: Get "http://127.0.0.1:9644/v1/config": dial tcp 127.0.0.1:9644: connect: connection refused

Additional information

First part of redpanda.yaml showing admin API endpoint config (cat /etc/redpanda/redpanda.yaml):

cluster_id: redpanda
config_file: /etc/redpanda/redpanda.yaml
node_uuid: PCQDXX5tijj4VLLLBVb2gQXgvfege2MyozyPEphJ2jM2JcXbY
organization: redpanda-test
pandaproxy: {}
redpanda:
  admin:
  - address: 172.31.46.143
    port: 9644
  advertised_kafka_api:
  - address: 34.223.42.124
    name: primary
    port: 9092
  - address: 172.31.46.143
    name: private
    port: 9093
  advertised_rpc_api:
    address: 172.31.46.143
    port: 33145

<truncated>

[r-vasquez]

@travisdowns The admin API endpoint that rpk redpanda admin config print --host 0 takes is from rpk.admin_api.addresses in the config file.

I tried with:

config_file: /etc/redpanda/redpanda.yaml
node_uuid: PYe1gLX9p9pbGrr9Kb61CxEeqRH7oca9bZzHyfRTRtSVMCcA5
pandaproxy: {}
redpanda:
  admin:
  - address: 0.0.0.0
    port: 9644
  <...>
rpk:
  admin_api:  # <-- Takes this
    addresses:
      - 172.31.46.143
  coredump_dir: /var/lib/redpanda/coredump
  enable_memory_locking: false
  enable_usage_stats: true
  <...>

And it works, I'll be making changes in the help output to make this clear.

[travisdowns]

@r-vasquez - but how does this rpk.admin_api entry get populated? It is not populated in a vanilla install: is the expectation that the user is adding these entries by hand?

I feel the experience here is poor because most other rpk commands take their parameters from the local configuration when possible, but then rpk admin works differently?

[twmb]

rpk.admin_api is populated manually, although in the future I hope we have a combo of two things: (1) a wizard to setup your rpk configuration, and (2) the ability to discover admin ports from kafka ports, and kafka ports from admin ports.

rpk.kafka_api IMO is a bit of an anomaly by peeking into the redpanda section. Fundamentally, redpanda and rpk are two separate configurations for different programs. I know we've had some goals to reduce or perhaps even eliminate the redpanda configuration entirely, and my own goal is to separate out rpk configuration into its own dedicated file. If this happens, it'll be much more obvious that these configurations are for different programs -- at the moment, configuring both in the same file leads to confusing situations like this.

[travisdowns]

rpk.kafka_api IMO is a bit of an anomaly by peeking into the redpanda section. Fundamentally, redpanda and rpk are two separate configurations for different programs. I know we've had some goals to reduce or perhaps even eliminate the redpanda configuration entirely, and my own goal is to separate out rpk configuration into its own dedicated file. If this happens, it'll be much more obvious that these configurations are for different programs -- at the moment, configuring both in the same file leads to confusing situations like this.

I agree with what you're saying about keeping the rpk config separate, but the "peek inside" behavior of the other rpk commands does set the precedent that stuff "just works" when you deploy redpanda and then use rpk on the node and I don't think this is an uncommon experience, so it is a bit jarring then that the admin commands work differently.

In any case, if that's the behavior, I guess that's the behavior and this is not a bug but just by design and changing the help text makes sense. Though I guess it does not answer the question of why it tries to connect to localhost? Shouldn't it give an error that host 0 does not exist in the config section?

[twmb]

"just works"
This should still be the precedent if we allow the admin or kafka ports to discover each other. I want to keep the "just works" behavior but avoid introducing anything that'll be gnarly to remove for backcompat purposes.

localhost
This is another fallout from how the current rpk configuration defaults to 127.0.0.1:9644 -- which makes things just work, until they don't.

IME, having magic is nice until there's a problem, but then once there's a problem, the magic makes it very confusing for what to fix. My longer term preference is to have as minimal magic as possible, but make it very easy to set and obvious things up the first time. This would result in a better learning experience as to what's actually happening when you do anything (that's my hope).