Unable to specify the backend engine using the CLI argument '--backend-engine' #3195

liv-io · 2024-01-13T10:12:47Z

Component

agent

Describe the bug

When running woodpacker-agent without a configuration file but only command-line arguments, the backend engine cannot be defined. Somehow the argument --backend-engine does not work and it falls back to local.

The following command does not select the docker backend:

exec /usr/local/bin/woodpecker-agent --grpc-token abcdef0123456789 --backend-engine docker --backend-docker-ipv6 false --backend-docker-host unix://run/podman/podman.sock --backend-docker-tls-verify false --grpc-skip-insecure true --healthcheck true --hostname woodpecker-agent --log-level info --grpc-secure false

While the agent starts, it immediately prints the following message to the logs (backend 'local'):

9:57AM INF Starting Woodpecker agent with version '2.1.1' and backend 'local' using platform 'linux/amd64' running up to 1 pipelines in parallel

When I export the environment variable DOCKER_HOST and run the same command again, the correct backend is selected:

export DOCKER_HOST="unix://run/podman/podman.sock"

exec /usr/local/bin/woodpecker-agent --grpc-token abcdef0123456789 --backend-engine docker --backend-docker-ipv6 false --backend-docker-host unix://run/podman/podman.sock --backend-docker-tls-verify false --grpc-skip-insecure true --healthcheck true --hostname woodpecker-agent --log-level info --grpc-secure false

The log output is (backend 'docker'):

10:07AM INF Starting Woodpecker agent with version '2.1.1' and backend 'docker' using platform 'linux/amd64' running up to 1 pipelines in parallel

System Info

Output of https://<your-woodpecker-instance>/version:

source	"https://github.com/woodpecker-ci/woodpecker"
version	"2.1.1"

Additional info on my setup:

Operating System: Debian 12.4
Podman: 4.3.1

Containerfile:
https://github.com/liv-io/oci-containers/blob/main/woodpecker-agent/Containerfile
Entrypoint/CMD script:
https://github.com/liv-io/oci-containers/blob/main/woodpecker-agent/cmd.sh

Additional context

No response

Validations

Read the Contributing Guidelines.
Read the docs.
Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
Checked that the bug isn't fixed in the next version already [https://woodpecker-ci.org/faq#which-version-of-woodpecker-should-i-use]
Check that this is a concrete bug. For Q&A join our Discord Chat Server or the Matrix room.

The text was updated successfully, but these errors were encountered:

qwerty287 · 2024-06-06T13:01:03Z

@liv-io I found a bug that could be related to this.

It's currently open as PR: #3770

Is it possible for you to try this out and check if it fixes the bug? Or do you still have this issue?

liv-io · 2024-06-09T14:36:26Z

Is it possible for you to try this out and check if it fixes the bug? Or do you still have this issue?

@qwerty287 Yes, certainly. Thanks for your efforts. Please give me a few days, but I should be able to send you a feedback hopefully this week.

liv-io · 2024-06-15T12:15:12Z

@qwerty287 Your branch has already been merged into 2.6.0, right?

I quickly made the update from 2.5.0 to 2.6.0 and removed the DOCKER_HOST environment variable.

When starting woodpecker-agent it immediately exits with:

{"level":"info","time":"2024-06-15T12:09:14Z","message":"log level: info"}
{"level":"info","time":"2024-06-15T12:09:14Z","message":"no agent config found at '/etc/woodpecker/agent.conf', start with defaults"}
{"level":"error","engine":"docker","time":"2024-06-15T12:09:14Z","message":"selected backend engine is unavailable"}
{"level":"fatal","error":"selected backend engine docker is unavailable","time":"2024-06-15T12:09:14Z","message":"error running agent"}

If I add the environment variable again, the woodpecker-agent starts fine:

export DOCKER_HOST="unix://run/podman/podman.sock"

qwerty287 · 2024-06-16T05:25:41Z

Your branch has already been merged into 2.6.0, right?

Yes.

When starting woodpecker-agent it immediately exits with:

Did you start with the CLI argument?

If you don't use "standard" docker (e.g. podman) it is necessary to set an env var or the CLI argument. It only worked with DOCKER_HOST before, and now it should also work with the CLI argument and WOODPECKER_BACKEND_DOCKER_HOST.

liv-io · 2024-06-16T08:57:30Z

Did you start with the CLI argument?

Yes, I believe so. This is the command I am running.

Command:

exec /usr/local/bin/woodpecker-agent \
--grpc-token <token> \
--backend-engine docker \
--backend-docker-ipv6 false \
--backend-docker-host unix://run/podman/podman.sock \
--backend-docker-tls-verify false \
--grpc-skip-insecure true \
--healthcheck true \
--hostname woodpecker-agent \
--log-level info \
--grpc-secure false

Output:

8:49AM INF log level: info
8:49AM INF no agent config found at '/etc/woodpecker/agent.conf', start with defaults
8:49AM ERR selected backend engine is unavailable engine=docker
8:49AM FTL error running agent error="selected backend engine docker is unavailable"

The moment I export the environment variable:

Command:

export DOCKER_HOST="unix://run/podman/podman.sock"

exec /usr/local/bin/woodpecker-agent \
--grpc-token <token> \
--backend-engine docker \
--backend-docker-ipv6 false \
--backend-docker-host unix://run/podman/podman.sock \
--backend-docker-tls-verify false \
--grpc-skip-insecure true \
--healthcheck true \
--hostname woodpecker-agent \
--log-level info \
--grpc-secure false

Output:

8:51AM INF log level: info
8:51AM INF no agent config found at '/etc/woodpecker/agent.conf', start with defaults
8:51AM ERR could not persist agent config at '/etc/woodpecker/agent.conf' error="open /etc/woodpecker/agent.conf: no such file or directory"
8:51AM INF starting Woodpecker agent with version '2.6.0' and backend 'docker' using platform 'linux/amd64' running up to 1 pipelines in parallel

qwerty287 · 2024-06-16T16:08:47Z

Is it possible for you to compile/run a custom binary so we can some debugging prints?

liv-io · 2024-06-17T15:22:25Z

Is it possible for you to compile/run a custom binary so we can some debugging prints?

Yes, certainly. It is a busy week for me, but I can try to build/run a custom release. Is there a branch I can try?

qwerty287 · 2024-06-17T15:54:11Z

Not yet, I'll try some things later this week but it's also busy for me ;)

qwerty287 · 2024-06-20T06:30:41Z

@liv-io Here you go: https://github.com/qwerty287/woodpecker/tree/debug-print

I only changed 3 lines.

liv-io · 2024-06-23T18:31:56Z

@qwerty287 Thanks a lot!

This is the output I get.

Without environment variable:

6:27PM INF log level: info
6:27PM INF no agent config found at '/etc/woodpecker/agent.conf', start with defaults
DOCKER DEBUG: docker host: ''
DOCKER DEBUG: cli context ok
DOCKER DEBUG: flag not set, checking default
6:27PM ERR selected backend engine is unavailable engine=docker
6:27PM FTL error running agent error="selected backend engine docker is unavailable"

With environment variable:

6:29PM INF log level: info
6:29PM INF no agent config found at '/etc/woodpecker/agent.conf', start with defaults
DOCKER DEBUG: docker host: 'unix://run/podman/podman.sock'
DOCKER DEBUG: cli context ok
6:29PM ERR could not persist agent config at '/etc/woodpecker/agent.conf' error="open /etc/woodpecker/agent.conf: no such file or directory"
6:29PM INF starting Woodpecker agent with version 'dev' and backend 'docker' using platform 'linux/amd64' running up to 1 pipelines in parallel

Let me know if I need to provide additional details.

Thanks for your help.

qwerty287 · 2024-06-23T19:02:32Z

Did you use the CLI flag in both cases? If not, can you try this?

liv-io · 2024-06-23T19:50:00Z

It seems to work without the --backend-engine docker flag.

Command:

/usr/local/bin/woodpecker-agent \
--grpc-token <token> \
--backend-docker-ipv6 false \
--backend-docker-host unix://run/podman/podman.sock \
--backend-docker-tls-verify false \
--grpc-skip-insecure true \
--healthcheck true \
--hostname woodpecker-agent \
--log-level info \
--grpc-secure false

Output:

7:44PM INF log level: info
7:44PM INF no agent config found at '/etc/woodpecker/agent.conf', start with defaults
DOCKER DEBUG: cli context ok
DOCKER DEBUG: flag not set, checking default
DOCKER DEBUG: docker host: ''
7:44PM ERR could not persist agent config at '/etc/woodpecker/agent.conf' error="open /etc/woodpecker/agent.conf: no such file or directory"
7:44PM INF starting Woodpecker agent with version 'dev' and backend 'local' using platform 'linux/amd64' running up to 1 pipelines in parallel

Also when adding the environment variable (again without the --backend-engine docker flag).

Command:

export DOCKER_HOST="unix://run/podman/podman.sock"

/usr/local/bin/woodpecker-agent \
--grpc-token <token> \
--backend-docker-ipv6 false \
--backend-docker-host unix://run/podman/podman.sock \
--backend-docker-tls-verify false \
--grpc-skip-insecure true \
--healthcheck true \
--hostname woodpecker-agent \
--log-level info \
--grpc-secure false

Output:

7:48PM INF log level: info
7:48PM INF no agent config found at '/etc/woodpecker/agent.conf', start with defaults
DOCKER DEBUG: cli context ok
DOCKER DEBUG: docker host: 'unix://run/podman/podman.sock'
DOCKER DEBUG: cli context ok
7:48PM ERR could not persist agent config at '/etc/woodpecker/agent.conf' error="open /etc/woodpecker/agent.conf: no such file or directory"
7:48PM INF starting Woodpecker agent with version 'dev' and backend 'docker' using platform 'linux/amd64' running up to 1 pipelines in parallel

qwerty287 · 2024-06-26T06:15:18Z

I mean the backend-docker-host flag.

This one is the problem. In your case, the engine is selected correctly, but it then fails because the docker backend says it's not available. It should work however.

Can you check the branch again? I added some more prints

liv-io · 2024-06-30T16:27:48Z

I don't quite understand. Sorry for confusing things.

Since I am not using Docker, but Podman I thought specifying the --backend-docker-host flag (or the environment variable export DOCKER_HOST="unix://run/podman/podman.sock") is a must since the path is different.

I ran a version of your latest branch/commit. These are the results:

All flags, without environment variable:

/usr/local/bin/woodpecker-agent --grpc-token <token> --backend-engine docker --backend-docker-ipv6 false --backend-docker-host unix://run/podman/podman.sock --backend-docker-tls-verify false --grpc-skip-insecure true --healthcheck true --hostname woodpecker-agent --log-level info --grpc-secure false

4:11PM INF log level: info
4:11PM INF no agent config found at '/etc/woodpecker/agent.conf', start with defaults
DOCKER DEBUG: docker host: ''
DOCKER DEBUG: cli context ok
DOCKER DEBUG: IsSet false, Str
DOCKER DEBUG: flag not set, checking default
4:11PM ERR selected backend engine is unavailable engine=docker
4:11PM FTL error running agent error="selected backend engine docker is unavailable"

All flags, with environment variable:

export DOCKER_HOST="unix://run/podman/podman.sock"
/usr/local/bin/woodpecker-agent --grpc-token <token> --backend-engine docker --backend-docker-ipv6 false --backend-docker-host unix://run/podman/podman.sock --backend-docker-tls-verify false --grpc-skip-insecure true --healthcheck true --hostname woodpecker-agent --log-level info --grpc-secure false

4:11PM INF log level: info
4:11PM INF no agent config found at '/etc/woodpecker/agent.conf', start with defaults
DOCKER DEBUG: docker host: 'unix://run/podman/podman.sock'
DOCKER DEBUG: cli context ok
DOCKER DEBUG: IsSet true, Str unix://run/podman/podman.sock
4:11PM ERR could not persist agent config at '/etc/woodpecker/agent.conf' error="open /etc/woodpecker/agent.conf: no such file or directory"
4:11PM INF starting Woodpecker agent with version 'dev' and backend 'docker' using platform 'linux/amd64' running up to 1 pipelines in parallel

Without --backend-docker-host, without environment variable:

/usr/local/bin/woodpecker-agent --grpc-token <token> --backend-engine docker --backend-docker-ipv6 false --backend-docker-tls-verify false --grpc-skip-insecure true --healthcheck true --hostname woodpecker-agent --log-level info --grpc-secure false

4:11PM INF log level: info
4:11PM INF no agent config found at '/etc/woodpecker/agent.conf', start with defaults
DOCKER DEBUG: docker host: ''
DOCKER DEBUG: cli context ok
DOCKER DEBUG: IsSet false, Str
DOCKER DEBUG: flag not set, checking default
4:11PM ERR selected backend engine is unavailable engine=docker
4:11PM FTL error running agent error="selected backend engine docker is unavailable"

Without --backend-docker-host, with environment variable:

export DOCKER_HOST="unix://run/podman/podman.sock"
/usr/local/bin/woodpecker-agent --grpc-token <token> --backend-engine docker --backend-docker-ipv6 false --backend-docker-tls-verify false --grpc-skip-insecure true --healthcheck true --hostname woodpecker-agent --log-level info --grpc-secure false

4:12PM INF log level: info
4:12PM INF no agent config found at '/etc/woodpecker/agent.conf', start with defaults
DOCKER DEBUG: docker host: 'unix://run/podman/podman.sock'
DOCKER DEBUG: cli context ok
DOCKER DEBUG: IsSet true, Str unix://run/podman/podman.sock
4:12PM ERR could not persist agent config at '/etc/woodpecker/agent.conf' error="open /etc/woodpecker/agent.conf: no such file or directory"
4:12PM INF starting Woodpecker agent with version 'dev' and backend 'docker' using platform 'linux/amd64' running up to 1 pipelines in parallel

Without --backend-engine docker, without environment variable:

/usr/local/bin/woodpecker-agent --grpc-token <token> --backend-engine docker --backend-docker-ipv6 false --backend-docker-host unix://run/podman/podman.sock --backend-docker-tls-verify false --grpc-skip-insecure true --healthcheck true --hostname woodpecker-agent --log-level info --grpc-secure false

4:24PM INF log level: info
4:24PM INF no agent config found at '/etc/woodpecker/agent.conf', start with defaults
DOCKER DEBUG: docker host: ''
DOCKER DEBUG: cli context ok
DOCKER DEBUG: IsSet false, Str
DOCKER DEBUG: flag not set, checking default
4:24PM ERR selected backend engine is unavailable engine=docker
4:24PM FTL error running agent error="selected backend engine docker is unavailable"

Without --backend-engine docker, with environment variable:

export DOCKER_HOST="unix://run/podman/podman.sock"
/usr/local/bin/woodpecker-agent --grpc-token <token> --backend-engine docker --backend-docker-ipv6 false --backend-docker-host unix://run/podman/podman.sock --backend-docker-tls-verify false --grpc-skip-insecure true --healthcheck true --hostname woodpecker-agent --log-level info --grpc-secure false

4:25PM INF log level: info
4:25PM INF no agent config found at '/etc/woodpecker/agent.conf', start with defaults
DOCKER DEBUG: docker host: 'unix://run/podman/podman.sock'
DOCKER DEBUG: cli context ok
DOCKER DEBUG: IsSet true, Str unix://run/podman/podman.sock
4:25PM ERR could not persist agent config at '/etc/woodpecker/agent.conf' error="open /etc/woodpecker/agent.conf: no such file or directory"
4:25PM INF starting Woodpecker agent with version 'dev' and backend 'docker' using platform 'linux/amd64' running up to 1 pipelines in parallel

qwerty287 · 2024-06-30T17:27:36Z

The problem is: The backend is correctly selected, but the docker backend says it is not available. If you set a custom host, it should actually work however…
The cli parser lib does not recognise the flag somehow.

The only reason I can currently imagine is that there is some problem with the boolean flags. Can you try running the agent without all options that are not required? E.g. backend-docker-ipv6.

The only case that's interesting here is without env var, but with flag.

liv-io · 2024-07-01T07:53:36Z

Huh, it seems like your intuition was right. Thanks a lot for your help by the way!

I simply removed the --backend-docker-ipv6 false flag (without specifying the env var) and this is the output:

7:44AM INF log level: info
7:44AM INF no agent config found at '/etc/woodpecker/agent.conf', start with defaults
DOCKER DEBUG: docker host: 'unix://run/podman/podman.sock'
DOCKER DEBUG: cli context ok
DOCKER DEBUG: IsSet true, Str unix://run/podman/podman.sock
7:44AM ERR could not persist agent config at '/etc/woodpecker/agent.conf' error="open /etc/woodpecker/agent.conf: no such file or directory"
7:44AM INF starting Woodpecker agent with version 'dev' and backend 'docker' using platform 'linux/amd64' running up to 1 pipelines in parallel

It selects the docker backend (and the socket path is correct too).

Do you want me to test more variations of my command?

qwerty287 · 2024-07-01T10:05:00Z

Can you try --backend-docker-ipv6=false?

liv-io · 2024-07-01T13:46:41Z

Seems to work 👍

/usr/local/bin/woodpecker-agent --grpc-token <token> --backend-engine docker --backend-docker-ipv6=false --backend-docker-host unix://run/podman/podman.sock --backend-docker-tls-verify false --grpc-skip-insecure true --healthcheck true --hostname woodpecker-agent --log-level info --grpc-secure false

1:44PM INF log level: info
1:44PM INF no agent config found at '/etc/woodpecker/agent.conf', start with defaults
DOCKER DEBUG: docker host: 'unix://run/podman/podman.sock'
DOCKER DEBUG: cli context ok
DOCKER DEBUG: IsSet true, Str unix://run/podman/podman.sock
1:44PM ERR could not persist agent config at '/etc/woodpecker/agent.conf' error="open /etc/woodpecker/agent.conf: no such file or directory"
1:44PM INF starting Woodpecker agent with version 'dev' and backend 'docker' using platform 'linux/amd64' running up to 1 pipelines in parallel

Am I supposed to use the equal sign for all -- (double dash) options?

qwerty287 · 2024-07-01T19:18:42Z

You can do, that makes it more clear.

I can open an issue in the parser library to find out why this doesn't work if you think that's helpful

liv-io · 2024-07-03T15:48:33Z

The woodpecker-agent --help command does not mention the equal sign for any of the commands.

...
   --backend-docker-ipv6                               backend docker enable IPV6 (default: false) [$WOODPECKER_BACKEND_DOCKER_ENABLE_IPV6]
   --backend-docker-network value                      backend docker network [$WOODPECKER_BACKEND_DOCKER_NETWORK]
   --backend-docker-volumes value                      backend docker volumes (comma separated) [$WOODPECKER_BACKEND_DOCKER_VOLUMES]
...

Would you update the output or leave as is?

I can open an issue in the parser library to find out why this doesn't work if you think that's helpful

Do you think both options (with and without equal sign) should work? If yes, maybe we it is worth creating a ticket. Which parser library is being used?

qwerty287 · 2024-07-04T05:51:59Z

Do you think both options (with and without equal sign) should work? If yes, maybe we it is worth creating a ticket.

I need to check that too first, but I think it should work.

Which parser library is being used?

https://github.com/urfave/cli

liv-io · 2024-07-06T13:28:03Z

I need to check that too first, but I think it should work.

It seems adding an equal sign to all "long options" (with double dash) does not work. I can provide additional details if this is valuable.

Thanks for finding the library!

qwerty287 · 2024-08-08T16:17:07Z

Hey @liv-io, can you try to reproduce this with the latest version? Because we switched to v3 of the cli library, so it would be good to know whether it works there.

liv-io · 2024-09-18T13:40:56Z

@qwerty287 I've added an equal sign to all long options and it seems to work. Thanks a lot for your help. I will close this ticket. Cheers.

PS. Tested with version 2.7.1

liv-io added the bug Something isn't working label Jan 13, 2024

qwerty287 added this to the 2.x.x milestone Jan 14, 2024

qwerty287 mentioned this issue Jun 6, 2024

Respect cli argument when checking docker backend availability #3770

Merged

qwerty287 added the feedback this needs feedback label Jun 6, 2024

liv-io closed this as completed Sep 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to specify the backend engine using the CLI argument '--backend-engine' #3195

Unable to specify the backend engine using the CLI argument '--backend-engine' #3195

liv-io commented Jan 13, 2024 •

edited

Loading

qwerty287 commented Jun 6, 2024

liv-io commented Jun 9, 2024 •

edited

Loading

liv-io commented Jun 15, 2024

qwerty287 commented Jun 16, 2024

liv-io commented Jun 16, 2024

qwerty287 commented Jun 16, 2024

liv-io commented Jun 17, 2024

qwerty287 commented Jun 17, 2024

qwerty287 commented Jun 20, 2024

liv-io commented Jun 23, 2024 •

edited

Loading

qwerty287 commented Jun 23, 2024

liv-io commented Jun 23, 2024 •

edited

Loading

qwerty287 commented Jun 26, 2024

liv-io commented Jun 30, 2024

qwerty287 commented Jun 30, 2024

liv-io commented Jul 1, 2024

qwerty287 commented Jul 1, 2024

liv-io commented Jul 1, 2024

qwerty287 commented Jul 1, 2024

liv-io commented Jul 3, 2024 •

edited

Loading

qwerty287 commented Jul 4, 2024

liv-io commented Jul 6, 2024

qwerty287 commented Aug 8, 2024

liv-io commented Sep 18, 2024 •

edited

Loading

Unable to specify the backend engine using the CLI argument '--backend-engine' #3195

Unable to specify the backend engine using the CLI argument '--backend-engine' #3195

Comments

liv-io commented Jan 13, 2024 • edited Loading

Component

Describe the bug

System Info

Additional context

Validations

qwerty287 commented Jun 6, 2024

liv-io commented Jun 9, 2024 • edited Loading

liv-io commented Jun 15, 2024

qwerty287 commented Jun 16, 2024

liv-io commented Jun 16, 2024

qwerty287 commented Jun 16, 2024

liv-io commented Jun 17, 2024

qwerty287 commented Jun 17, 2024

qwerty287 commented Jun 20, 2024

liv-io commented Jun 23, 2024 • edited Loading

qwerty287 commented Jun 23, 2024

liv-io commented Jun 23, 2024 • edited Loading

qwerty287 commented Jun 26, 2024

liv-io commented Jun 30, 2024

qwerty287 commented Jun 30, 2024

liv-io commented Jul 1, 2024

qwerty287 commented Jul 1, 2024

liv-io commented Jul 1, 2024

qwerty287 commented Jul 1, 2024

liv-io commented Jul 3, 2024 • edited Loading

qwerty287 commented Jul 4, 2024

liv-io commented Jul 6, 2024

qwerty287 commented Aug 8, 2024

liv-io commented Sep 18, 2024 • edited Loading

liv-io commented Jan 13, 2024 •

edited

Loading

liv-io commented Jun 9, 2024 •

edited

Loading

liv-io commented Jun 23, 2024 •

edited

Loading

liv-io commented Jun 23, 2024 •

edited

Loading

liv-io commented Jul 3, 2024 •

edited

Loading

liv-io commented Sep 18, 2024 •

edited

Loading