-
-
Notifications
You must be signed in to change notification settings - Fork 372
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
aptly 0.9.5 "ERROR: can't open database: resource temporarily unavailable" #401
Comments
I think aptly exit code should be non zero, but the error itself is quite expected: aptly does locking on DB and fails to get it. In general, I would recommend running aptly in API mode for cicd. |
This sounds like database contention. I've encountered this several times when I use Jenkins to update various debs and push to aptly. I only have one aptly instance and when jobs try to update aptly at the same time, they fail because aptly only lets the job that got to it first have the mutex. The solution was to have one job that consumes artifacts of all jobs and pushes to aptly so that way it's one job continually using aptly and not multiple racing for access. Having multiple slaves try to ssh and edit aptly at the same time might be causing your problem. |
We could have tried to add an option to try acquiring DB lock for some time before giving up, should be relatively easy. That way we could avoid simple failures like that. Thoughts? |
As long as the time allowance for trying to access the DB can be set manually, this could work. The only problem I can see is with timing is if the timeout isn't long enough. In the Jenkins example I mentioned, the current deb and package library is extremely large (I'm currently in the middle of coming up with solutions to purge and make it smaller) and so if you do any kind of update to aptly or the db, it takes upwards of tens of minutes (especially |
Yes, I'm looking towards something like |
I'm running a bunch of instances like this (context is stress testing unrelated part of CI system;
each build slave ssh's to the apt repo machine and runs aptly to suck the new debs in):
aptly repo add -force-replace=true -remove-files=true repo-buildhost4.oblong.com-xenial-dev incoming-unique.tmpdir.13432
I think I noticed that aptly doesn't like multiple instances running, so It's wrapped with a shell script to not run when any other aptly process is present. The shell script is racy and sometimes lets more than one instance through, and aptly fails with:
Loading mirrors, local repos, snapshots and published repos...
ERROR: can't open database: resource temporarily unavailable
Is this expected? Isn't aptly supposed to do its own locking?
(Also, oddly, when this happens, aptly doesn't seem to exit with nonzero status, so my script doesn't notice the problem.)
The text was updated successfully, but these errors were encountered: