Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loopd becomes unresponsive after some time #767

Open
Jossec101 opened this issue Jun 3, 2024 · 12 comments
Open

Loopd becomes unresponsive after some time #767

Jossec101 opened this issue Jun 3, 2024 · 12 comments

Comments

@Jossec101
Copy link

Jossec101 commented Jun 3, 2024

Hey!

After some time (we have not measured how much yet), our loop is having issues doing loop outs, getinfo breaks as in the log below:

lnd:/# loop getinfo
[loop] rpc error: code = Unknown desc = sql: Scan error on column index 20, name "publication_deadline": unsupported Scan, storing driver.Value type string into type *time.Time

Additionally, invoking loop out gives this error:

cltv delta below minimum: 98

It somehow resolves if we kill loopd and some swaps got broadcast onchain

System information

Versions:
LND 0.17.5 and loop 0.27.1 for amd64 linux containers on kubernetes (AWS)

@Jossec101 Jossec101 changed the title Loop becomes unresponsive after some time Loopd becomes unresponsive after some time Jun 3, 2024
@hieblmi
Copy link
Collaborator

hieblmi commented Jun 3, 2024

Thanks for raising the issue @Jossec101. Have you been running this setup for a while or did this behavior recently just appear or did you upgrade any versions of your stack?

@Jossec101
Copy link
Author

Thanks for raising the issue @Jossec101. Have you been running this setup for a while or did this behavior recently just appear or did you upgrade any versions of your stack?

Nope, we have running loopd for more than 1 year upgrading from time to time to newer versions, recently we just started to see this behaviour.

@bhandras
Copy link
Member

bhandras commented Jun 3, 2024

Are you using sqlite or PostgreSQL? If PostgreSQL, could you please provide the version that your interfacing with?

@Jossec101
Copy link
Author

Are you using sqlite or PostgreSQL? If PostgreSQL, could you please provide the version that your interfacing with?

SQLite

@Jossec101
Copy link
Author

Wiping up the db fixed this issue, it looks to be a corruption over time

@bhandras
Copy link
Member

bhandras commented Jun 5, 2024

Thanks for the heads-up @Jossec101. If I understand correctly you're referring to filesystem corruption? Did you identify the source of the corruption? Or do you maybe mean that the DB file itself was changed outside of loopd?

@Jossec101
Copy link
Author

Thanks for the heads-up @Jossec101. If I understand correctly you're referring to filesystem corruption? Did you identify the source of the corruption? Or do you maybe mean that the DB file itself was changed outside of loopd?

I mean that we stopped loopd, moved the sqlite to a safe place and loopd recreated a new one and that solved the issue.

@Jossec101
Copy link
Author

It happened again today, bump

@bhandras
Copy link
Member

bhandras commented Jul 1, 2024

Did you get the same error as reported in #767 (comment) ?

@Jossec101
Copy link
Author

Jossec101 commented Jul 2, 2024

Yes, and and the db actually corrupted after a few hours since last comment but we upgraded to loop 0.28, I will report back if it happens again

@Jossec101
Copy link
Author

Jossec101 commented Jul 3, 2024

It has happened again, actually the frequency between db corruptions is increasing 🤔

@bhandras
Copy link
Member

bhandras commented Jul 3, 2024

I believe it may not be a DB corruption but perhaps a Go sqlite bug as it seemingly fails to parse the timestamp value. Do you still have your DB? Perhaps could you do a select and see what publication deadlines are there? Or if you could share the DB with me, I'm happy to take a look. I'm on both Keybase and Slack.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants