rpk should preserve config file ownership:

[ajfabbri]

Version & Environment

Redpanda version: latest

What went wrong?

Failure starting redpanda with an error similar to:

 Failure during startup: std::__1::system_error (error system:13, open: Permission denied)

This has been reported multiple times, typically when rpk is run as one user (e.g. sudo) and redpanda is run as another.

What should have happened instead?

If a user does sudo rpk command that may backup / rewrite / rename configuration files, rpk should probably propagate the original ownership and permissions of the file (modulo a sanity mask).

[Akaame]

Hello I wanted to contribute to this but I am not sure if I am getting the scenario right

1. ls -l /etc/redpanda verify ownership and permissions of config file(s)
2. Run any command that changes config. Example: sudo rpk redpanda config set redpanda.node_id 2
3. ls -l /etc/redpanda does not show any indication that ownership and permissions of this file has changed.

---

When I try to run redpanda --redpanda-cfg /etc/redpanda/redpanda.yaml, I get either

Error: System check 'Data directory is writable' failed. Required: true, Current false

or the permission denied error exactly same with yours

syschecks - Writing pid file "/var/lib/redpanda/data/pid.lock"
Failure during startup: std::__1::system_error (error system:13, open: Permission denied)

both of which indicate more or less the same thing. A good first step may be to wrap this error and give a more helpful feedback to the user.

[twmb]

Possibly related to #1919

[rick20]

Hi, got this same error today. May I know how to solve this issue?

[twmb]

@rick20, a few questions:

• What rpk version are you on (rpk version)?
• What are the perms on the dir that owns redpanda.yaml?
• What are the perms on redpanda.yaml?
• What user are you running rpk as?

[rick20]

Hi @twmb

What rpk version are you on (rpk version)?

v22.1.3

What are the perms on the dir that owns redpanda.yaml?

Owner of the dir volume is a non-root user with 775 or 777 permission
This is the kafka service in docker-compose.yaml

  kafka:
    container_name: kafka
    image: vectorized/redpanda:v22.1.3
    restart: "no"
    ports:
      - "19092:19092"
    command:
      - redpanda
      - start
      - --overprovisioned
      - --kafka-addr
      - DOCKER_NETWORK://0.0.0.0:9092,LOCALHOST_NETWORK://0.0.0.0:19092
      - --advertise-kafka-addr
      - DOCKER_NETWORK://kafka:9092,LOCALHOST_NETWORK://127.0.0.1:19092
    volumes:
      - /home/bdf/kafka-data:/var/lib/redpanda/data

What are the perms on redpanda.yaml?

I don't manage the redpanda.yaml

What user are you running rpk as?

redpanda

[twmb]

Can you upgrade to the latest rpk / redpanda? I don't think the fix was backported to v22.1, which is ~1yr old now.

[rick20]

Hi @twmb I have tested using latest redpanda.
I even copy the exact same docker-compose.yml configuration right from https://docs.redpanda.com/docs/platform/quickstart/quick-start-docker/#run-with-docker-compose
The only thing I add is a volume which is mapped to root-owned folder and 775 permission.
Here's my docker-compose.yml:

version: '3.7'
services:
  redpanda:
    # NOTE: Use the latest version here!
    image: docker.redpanda.com/vectorized/redpanda:latest
    container_name: redpanda-1
    command:
    - redpanda
    - start
    - --smp
    - '1'
    - --reserve-memory
    - 0M
    - --overprovisioned
    - --set redpanda.empty_seed_starts_cluster=false
    - --seeds "redpanda-1:33145"
    - --kafka-addr
    - PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
    - --advertise-kafka-addr
    - PLAINTEXT://redpanda:29092,OUTSIDE://localhost:9092
    - --pandaproxy-addr
    - PLAINTEXT://0.0.0.0:28082,OUTSIDE://0.0.0.0:8082
    - --advertise-pandaproxy-addr
    - PLAINTEXT://redpanda:28082,OUTSIDE://localhost:8082
    - --advertise-rpc-addr redpanda-1:33145
    ports:
    - 8081:8081
    - 8082:8082
    - 9092:9092
    - 28082:28082
    - 29092:29092
    volumes:
      - /data/kafka-data:/var/lib/redpanda/data

The host directory:

$:~/redpanda$ ls -al /data/
total 12
drwxrwxr-x  3 root root 4096 Jan 15 05:06 .
drwxr-xr-x 20 root root 4096 Jan 15 05:06 ..
drwxrwxr-x  2 root root 4096 Jan 15 05:06 kafka-data

I have tested using non-root user as well but still no luck.

docker compose logs redpanda still showing the same error:

...
redpanda-1  | INFO  2023-01-15 05:07:16,373 [shard 0] main - application.cc:527 - schema_registry_client.retry_base_backoff_ms:100	- Delay (in milliseconds) for initial retry backoff
redpanda-1  | INFO  2023-01-15 05:07:16,373 [shard 0] main - application.cc:527 - schema_registry_client.sasl_mechanism:	- The SASL mechanism to use when connecting
redpanda-1  | INFO  2023-01-15 05:07:16,373 [shard 0] main - application.cc:527 - schema_registry_client.scram_password:	- Password to use for SCRAM authentication mechanisms
redpanda-1  | INFO  2023-01-15 05:07:16,373 [shard 0] main - application.cc:527 - schema_registry_client.scram_username:	- Username to use for SCRAM authentication mechanisms
redpanda-1  | INFO  2023-01-15 05:07:16,373 [shard 0] syschecks - Writing pid file "/var/lib/redpanda/data/pid.lock"
redpanda-1  | INFO  2023-01-15 05:07:16,373 [shard 0] main - application.cc:319 - Shutdown complete.
redpanda-1  | ERROR 2023-01-15 05:07:16,373 [shard 0] main - application.cc:333 - Failure during startup: std::__1::system_error (error system:13, open: Permission denied)

[rick20]

fixed by changing the ownership of the directory to 101:101 (systemd-resolve:systemd-journal) in Ubuntu 22.04

[r-vasquez]

@rick20 The problem is that the default UID:GID of the image: docker.redpanda.com/vectorized/redpanda:latest is 101:101

I followed your example without binding the data directory and it worked:

$ docker-compose up -d

$ docker exec -it redpanda-1 id
uid=101(redpanda) gid=101(redpanda) groups=101(redpanda)

So that's why you have to change the ownership of the root-owned directory.

If I try binding a root-owned directory I get the same error:

$ sudo mkdir test
$ pwd
/tmp/4046-repro
$ ls -l
total 4
-rw-r--r--. 1 rvasquez rvasquez 907 Jan 16 14:06 docker-compose.yaml
drwxr-xr-x. 2 root     root      40 Jan 16 14:05 test


$ docker-compose up
redpanda-1  | ERROR 2023-01-16 19:08:31,551 [shard 0] main - application.cc:333 - Failure during startup: std::__1::system_error (error system:13, open: Permission denied)

I hope this clarify a little bit.

[jlnunez89]

Just for anyone stuck on this- I was getting:

"Failure during startup: std::__1::system_error (error system:13, open: Permission denied"

and I got past it by setting up fsGroup: 101 in the security context:

spec:
  securityContext:
    fsGroup: 101

particularly, I'm using Kompose, so that means I ended up doing this in my docker-compose file:

[...]

  redpanda:
    image: docker.redpanda.com/redpandadata/redpanda:latest
    labels:
      kompose.security-context.fsgroup: 101
    command:
      - redpanda start
      [...]