Runners not scaling when EphemeralRunner status is Failed

[bryanjuho]

When EphemeralRunners status is marked as Failed, runners fail to scale to the minimum (3 in my case) number.

I guess Runners tend to fail when every once in a while, but when pods start to fail and reach the minRunners count, it doesn't shutdown automatically (which is what I expect).

According to the runner-set-listener logs, replicas count shows -1 when all three of my runners had failed.

{"namespace": "devops", "name": "runner-set-n6bk6", "replicas": -1}

Any ideas? Thanks!

[DamianPalyskaSpotOn]

I had the same problem. The solution was to bump the runner image version to the latest.

[bryanjuho]

Unfortunately it did not, I ended up creating a CronJob resource to delete failed runners periodically.

[krupakar1329]

`2024-05-16T15:50:01Z INFO EphemeralRunner EphemeralRunner has failed more than 5 times. Marking it as failed {"ephemeralrunner": {"name":"pd-cluster-znz6x-runner-4ctcc","namespace":"github-arc-runner-xxx"}}
2024-05-16T15:50:01Z INFO EphemeralRunner Updating ephemeral runner status to Failed {"ephemeralrunner": {"name":"pd-cluster-znz6x-runner-4ctcc","namespace":"github-arc-runner-x"}}xx
2024-05-16T15:50:01Z INFO EphemeralRunner Removing the runner from the service {"ephemeralrunner": {"name":"pd-cluster-znz6x-runner-4ctcc","namespace":"github-arc-runner-xxx"}}
2024-05-16T15:50:01Z INFO EphemeralRunnerSet Ephemeral runner counts {"ephemeralrunnerset": {"name":"pd-cluster-znz6x","namespace":"github-arc-runner-x"}, "pending"xx: 0, "running": 0, "finished": 0, "failed": 4, "deleting": 0}
2024-05-16T15:50:01Z INFO EphemeralRunnerSet Scaling comparison {"ephemeralrunnerset": {"name":"pd-cluster-znz6x","namespace":"github-arc-runner-xxx"}, "current": 4, "desired": 4}
2024-05-16T15:50:01Z INFO EphemeralRunnerSet Updating status with current runners count {"ephemeralrunnerset": {"name":"pd-cluster-znz6x","namespace":"github-arc-runner-xxx"}, "count": 4}

runners are not seen in giithub , runner pods are getting deleted. any clue
`

[bryanjuho]

I was never able to find out why they fail, but since they autoscale, I just delete them.

[alexgaganashvili]

After trying to figure out why new runners do not get scheduled, I found out that that Failed ephemeralrunners were to blame. I opened a new discussion thread yesterday. I started thinking of creating a cron job and see in this post a confirmation that that's what I'll most likely have to do.
Btw, what's interesting is that I'm not running into this issue on a different cluster. So I'm wondering what the root cause it.
Whatever it is, it would be great if Failed ephemeralrunners were automatically cleaned up.
fyi: @nikola-jokic

[koraypinarci]

I have the same issue, and i created a cron-job that cleans up all failed ephemeralrunners so that the jobs can continue running. It would be better, of course, if ARC could recognize and clean them up by itself.

[kiranbayram]

Hello there @bryanjuho , could you please provide the details how exactly you set up the CronJob and what commands you're running in it? Thanks in advance. Regards 🙏

[bryanjuho]

Hi, I'm sure there's a better way to do this but this is how I've done it.
hope this helps

The Cronjob resource

# CronJob to delete failed runners
apiVersion: batch/v1
kind: CronJob
metadata:
  name: delete-failed-ephemeral-runners
  namespace: development
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: runner-manager
          containers:
          - name: delete-failed-runners
            image: my-registry/delete-runners:latest
            imagePullPolicy: Always
          restartPolicy: OnFailure

The Dockerfile for the delete-runners image

FROM debian:bookworm-slim

# Install kubectl
RUN apt-get update && apt-get install -y curl \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

RUN curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" && \
    chmod +x kubectl && \
    mv kubectl /usr/local/bin/

# Add your script to the container
WORKDIR /app
COPY delete-failed-runners.sh .

RUN chmod +x delete-failed-runners.sh

ENTRYPOINT ["/app/delete-failed-runners.sh"]

Lastly the shell script (delete-failed-runners.sh)

#!/bin/bash

# Namespace where your EphemeralRunners are deployed
NAMESPACE="development"

# Get the list of EphemeralRunner resources with status 'Failed'
failed_runners=$(kubectl get EphemeralRunner -n $NAMESPACE --no-headers -o custom-columns=":metadata.name,:status.phase" | grep Failed | awk '{print $1}')

for runner in $failed_runners; do
    echo "Deleting failed EphemeralRunner: $runner"
    kubectl delete EphemeralRunner $runner -n $NAMESPACE
done

[kiranbayram]

This is great 🎉 thanks so much 🙏 @bryanjuho I will look into it

[krupakar1329]

Seeing the issue again Found below in pod logs. 2.317.0 version is deprecated

[RUNNER 2024-09-17 09:15:23Z INFO Terminal] WRITE LINE: Current runner version: '2.317.0'
Current runner version: '2.317.0'
[RUNNER 2024-09-17 09:15:23Z INFO Terminal] WRITE LINE: 2024-09-17 09:15:23Z: Listening for Jobs
2024-09-17 09:15:23Z: Listening for Jobs
[RUNNER 2024-09-17 09:15:23Z INFO JobDispatcher] Set runner/worker IPC timeout to 30 seconds.
[RUNNER 2024-09-17 09:15:23Z ERR  GitHubActionsService] GET request to https://pipelinesghubeus4.actions.githubusercontent.com/GcLiJYpRBu4UULaLKeoKHekWwfyzWN1jFTCRGiRwdwkcnwbFRH/_apis/distributedtask/pools/5/messages?sessionId=d795c87b-01b9-4d44-988a-ad7ebf46b674&status=Online&runnerVersion=2.317.0&os=Linux&architecture=X64&disableUpdate=true failed. HTTP Status: Forbidden
[RUNNER 2024-09-17 09:15:23Z INFO JobDispatcher] Shutting down JobDispatcher. Make sure all WorkerDispatcher has finished.
[RUNNER 2024-09-17 09:15:23Z INFO Runner] Deleting Runner Session...
[RUNNER 2024-09-17 09:15:23Z ERR  Terminal] WRITE ERROR: An error occured: Runner version v2.317.0 is deprecated and cannot receive messages.
An error occured: Runner version v2.317.0 is deprecated and cannot receive messages.
[RUNNER 2024-09-17 09:15:23Z ERR  Listener] GitHub.DistributedTask.WebApi.AccessDeniedException: Runner version v2.317.0 is deprecated and cannot receive messages.
[RUNNER 2024-09-17 09:15:23Z ERR  Listener]    at GitHub.Services.WebApi.VssHttpClientBase.HandleResponseAsync(HttpResponseMessage response, CancellationToken cancellationToken)
[RUNNER 2024-09-17 09:15:23Z ERR  Listener]    at GitHub.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
[RUNNER 2024-09-17 09:15:23Z ERR  Listener]    at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
[RUNNER 2024-09-17 09:15:23Z ERR  Listener]    at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable`1 additionalHeaders, Guid locationId, Object routeValues, ApiResourceVersion version, HttpContent content, IEnumerable`1 queryParameters, Object userState, CancellationToken cancellationToken)
[RUNNER 2024-09-17 09:15:23Z ERR  Listener]    at GitHub.Runner.Listener.MessageListener.GetNextMessageAsync(CancellationToken token)
[RUNNER 2024-09-17 09:15:23Z ERR  Listener]    at GitHub.Runner.Listener.Runner.RunAsync(RunnerSettings settings, Boolean runOnce)
[RUNNER 2024-09-17 09:15:23Z ERR  Listener]    at GitHub.Runner.Listener.Runner.RunAsync(RunnerSettings settings, Boolean runOnce)
[RUNNER 2024-09-17 09:15:23Z ERR  Listener]    at GitHub.Runner.Listener.Runner.RunAsync(RunnerSettings settings, Boolean runOnce)
[RUNNER 2024-09-17 09:15:23Z ERR  Listener]    at GitHub.Runner.Listener.Runner.ExecuteCommand(CommandSettings command)
[RUNNER 2024-09-17 09:15:23Z ERR  Listener]    at GitHub.Runner.Listener.Program.MainAsync(IHostContext context, String[] args)
Runner listener exit with terminated error, stop the service, no retry needed.

[medounesa]

i have same issue after update to the last version 2.319.1 but the ephemeral runner still offline