Loop out fee sent, HTLCs remains pending yet LITD says it failed

[MButcho]

I looped out 5 mil sats with custom BTC address and no confirmation target. After 7 hours, HTLCs are still pending, fee was paid 7 hours ago but https://terminal.lightning.engineering/liquidity/ says it failed.

Actually loop out the amount to custom BTC address, I have done it many times before

1. Choose 5 050 000 sats with slider
2. Click Loop (one time manual)
3. Click advanced, entered custom bc1q... BTC address, leave confirmation target empty
4. Clicked finish or confirm or something like that
5. Retarted LITD

Happend for the 1st time.

Standalone

lnd-mode=remote

LiT version: 0.12.2-alpha commit=v0.12.2-alpha / website

physical HW, SSD

Ubuntu 22.04 LTS

Loop Out Amount
5,050,000 sats
Status
Failed
Failure Reason
Off-chain Failure
Initiated
Jan 25, 2024 1:37 PM
Last Update
Jan 25, 2024 2:56 PM
ID
896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a
HTLC Address
bc1plmdpe96jdk4xh922f4w2gfngq3y3kpgk3ezx3nk2vsgs3xp9ew3q65d8se
Swap Server Node Pubkey
021c97a90a411ff2b10dc2a8e32de2f29d2fa49d41bfbb52bd416e460db0747d0d

Fee
Payment to: 03fb2a0ca79c005f493f
(7 hours ago)
30,000
Created:
7 hours ago (2024.01.25 - 13:37:46)
Destination Node:
03fb2a0ca79c005f493f
Fee:
48
Fee msats:
48000 millisats
Hops:
2
Hop 1:
cyberdyne.sh
Hop 2:
LOOP
Id:
d2f61c75edb32eb820f9f03a44325c429a45e4cb27ff98e8e5077ef1b9fe17db
Is Outgoing:
true
Secret:
xxx
M Tokens:
30000000 millisats

2024-01-25 13:37:43.673 [INF] LOOPD: Loop out request received
2024-01-25 13:37:43.701 [INF] LOOP: LoopOut 0.05050000 BTC to bc1q... (channels: )
2024-01-25 13:37:43.968 [INF] LOOP: Initiating swap request at height 827310: amt=0.05050000 BTC, expiry=827410
2024-01-25 13:37:44.369 [INF] LOOP: 896ec1 Htlc address (P2TR): bc1plmdpe96jdk4xh922f4w2gfngq3y3kpgk3ezx3nk2vsgs3xp9ew3q65d8se
2024-01-25 13:37:44.415 [INF] LOOP: 896ec1 Server message: Use AutoLoop - it is really good, give it a try
2024-01-25 13:37:44.418 [INF] LOOP: 896ec1 Loop out swap state: Initiated
2024-01-25 13:37:44.421 [INF] LOOP: 896ec1 Sending swap payment lnbc50263260n1pjmy4scpp539hvzkm32v9jqe75307k8pjmcxupej960ljqqqk95vuxscu2mx9qdy9wdmkzupq95s8xcmjd9c8gw3qx5cnyvrxv4jxzvtr8ymn2v3kv3skzdnz8y6ngcf5vs6kxcf5xgmrvwpsxs6rjvtzxq6nzd3cv56rgd3cvdjkxcfkxscnzvpc8yurydtrvfsnycqzxgxq97zvuqrzjqgwf02g2gy0l9vgdc25wxt0z72wjlfyagxlmk54ag9hyvrdsw37smapyqqqqqqqq2qqqqqqqqqqqqqqq9qsp5sn2wq73qrcr3q3h7v9eqr49dllzmmaa6xmc24996jssxwrwf7ksq9qyyssqx3u9kk8jswkn44c8nzxw03heez9tcfr4kzf8mz62qn66l42z48lp7wjvdkd9d75ldyc5qe09u9vwx0cyltdgsqw3mkm02fhc7wlfgvgpx2naal
2024-01-25 13:37:44.642 [INF] LOOP: 896ec1 Server update: SERVER_INITIATED received, timestamp: 2024-01-25 13:37:44.245419666 +0100 CET
2024-01-25 13:37:44.649 [INF] LOOP: 896ec1 Server recommended routing plugin: Low/High
2024-01-25 13:37:44.649 [INF] LOOP: 896ec1 Sending prepayment lnbc300u1pjmy4scpp56tmpca0dkvhtsg8e7qaygvjug2dytextylle3689qal0rw07zldsdq2wpex2urp0ycqzxgxq97zvuqrzjqgwf02g2gy0l9vgdc25wxt0z72wjlfyagxlmk54ag9hyvrdsw37smapyqqqqqqqq2qqqqqqqqqqqqqqq9qsp5d6243690c6t73hdxv9ffsahvxfk5mu88hn99zzx4k28cjlt0s4hq9qyyssqnx4k50g7tdrfz4gan5f0dgz7z2q8d8lwl38z0sv0ftnndtqzxaq99dwphk23lsxa60qeed8jppty07chul6fydntyexv5slcpr3790sp908jxg
2024-01-25 13:37:44.650 [INF] LOOP: 896ec1 Register 1 conf ntfn for swap script on chain (hh=827310)
2024-01-25 13:37:44.653 [INF] LOOP: 896ec1 Checking preimage reveal height 827390 exceeded (height 827310)
2024-01-25 13:37:44.653 [INF] LOOP: 896ec1 Waiting for either htlc on-chain confirmation or off-chain payment failure
2024-01-25 13:37:46.459 [INF] LOOP: 896ec1 Acquired routing plugin Low/High for payment 896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a
2024-01-25 13:37:46.460 [INF] LOOP: 896ec1 Payment (896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a) try count 1/3 (plugin=Low/High)
2024-01-25 13:37:46.460 [INF] LOOP: 896ec1 Payment (d2f61c75edb32eb820f9f03a44325c429a45e4cb27ff98e8e5077ef1b9fe17db) try count 1/1 (plugin=None)
2024-01-25 13:37:46.500 [INF] LOOP: 896ec1 Payment d2f61c75edb32eb820f9f03a44325c429a45e4cb27ff98e8e5077ef1b9fe17db: state=IN_FLIGHT, inflight_htlcs=0, inflight_amt=0 mSAT
2024-01-25 13:37:46.500 [INF] LOOP: 896ec1 Payment 896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a: state=IN_FLIGHT, inflight_htlcs=0, inflight_amt=0 mSAT
2024-01-25 13:37:46.764 [INF] LOOP: 896ec1 Payment d2f61c75edb32eb820f9f03a44325c429a45e4cb27ff98e8e5077ef1b9fe17db: state=IN_FLIGHT, inflight_htlcs=1, inflight_amt=30000000 mSAT
2024-01-25 13:37:46.799 [INF] LOOP: 896ec1 Payment 896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a: state=IN_FLIGHT, inflight_htlcs=1, inflight_amt=2513163000 mSAT
2024-01-25 13:37:46.856 [INF] LOOP: 896ec1 Payment 896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a: state=IN_FLIGHT, inflight_htlcs=2, inflight_amt=3769744500 mSAT
2024-01-25 13:37:46.915 [INF] LOOP: 896ec1 Payment 896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a: state=IN_FLIGHT, inflight_htlcs=3, inflight_amt=5026326000 mSAT
2024-01-25 13:37:47.291 [INF] LOOP: 896ec1 Payment 896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a: state=IN_FLIGHT, inflight_htlcs=2, inflight_amt=3769744500 mSAT
2024-01-25 13:37:47.352 [INF] LOOP: 896ec1 Payment 896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a: state=IN_FLIGHT, inflight_htlcs=3, inflight_amt=5026326000 mSAT
2024-01-25 13:37:47.922 [INF] LOOP: 896ec1 Payment 896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a: state=IN_FLIGHT, inflight_htlcs=2, inflight_amt=3769744500 mSAT
2024-01-25 13:37:47.991 [INF] LOOP: 896ec1 Payment 896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a: state=IN_FLIGHT, inflight_htlcs=3, inflight_amt=5026326000 mSAT
2024-01-25 13:37:48.967 [INF] LOOP: 896ec1 Server update: SERVER_HTLC_PUBLISHED received, timestamp: 2024-01-25 13:37:48.862091221 +0100 CET
2024-01-25 13:38:14.647 [ERR] LOOP: 896ec1 swap subscription error: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: INTERNAL_ERROR

After reboot

2024-01-25 14:56:04.974 [INF] RPCS: Starting RPC Server
2024-01-25 14:56:04.974 [INF] LITD: Starting LiT macaroon service
2024-01-25 14:56:05.111 [INF] AUTO: Starting Autopilot Client
2024-01-25 14:56:05.394 [INF] LOOP: 896ec1 Acquired routing plugin Low/High for payment 896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a
2024-01-25 14:56:05.395 [INF] LOOP: 896ec1 Payment (d2f61c75edb32eb820f9f03a44325c429a45e4cb27ff98e8e5077ef1b9fe17db) try count 1/1 (plugin=None)
2024-01-25 14:56:05.395 [INF] LOOP: 896ec1 Payment (896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a) try count 1/3 (plugin=Low/High)
2024-01-25 14:56:05.410 [INF] LOOP: 896ec1 Failed prepayment: rpc error: code = Unknown desc = payment is in transition
2024-01-25 14:56:05.410 [INF] LOOP: 896ec1 Canceling swap: 1 payment failed: FAILURE_REASON_NONE, 0 attempts
2024-01-25 14:56:05.860 [INF] LOOP: 896ec1 Wait for server pulling off-chain payment(s)
2024-01-25 14:56:05.860 [INF] LOOP: 896ec1 Swap payment failed: rpc error: code = Unknown desc = payment is in transition
2024-01-25 14:56:05.860 [INF] LOOP: 896ec1 Swap completed: FailOffchainPayments (final cost: server 0 BTC, onchain 0 BTC, offchain 0 BTC)
2024-01-25 14:56:05.867 [INF] LOOP: 896ec1 Loop out swap state: FailOffchainPayments

EDIT: highlighted the logs

[levmi]

Thank you for reporting the issue, we're going to look into this and get back to you shortly with recommended next steps!

[MButcho]

Update: The pending HTLCs were resolved, back on my node, yet the fee was paid and not returned. Possible to reimburse?

[levmi]

What was the fee amount? At what point did you restart?

[MButcho]

I restarted after like ~90 mins. You can see it in logs above. 30k sats was sent as a “prepayment”:
2024-01-25 13:37:44.649 [INF] LOOP: 896ec1 Sending prepayment lnbc300u1pjmy4scpp56tmpca0dkvhtsg8e7qaygvjug2dytextylle3689qal0rw07zldsdq2wpex2urp0ycqzxgxq97zvuqrzjqgwf02g2gy0l9vgdc25wxt0z72wjlfyagxlmk54ag9hyvrdsw37smapyqqqqqqqq2qqqqqqqqqqqqqqq9qsp5d6243690c6t73hdxv9ffsahvxfk5mu88hn99zzx4k28cjlt0s4hq9qyyssqnx4k50g7tdrfz4gan5f0dgz7z2q8d8lwl38z0sv0ftnndtqzxaq99dwphk23lsxa60qeed8jppty07chul6fydntyexv5slcpr3790sp908jxg

[MButcho]

What was the fee amount? At what point did you restart?

Any update?

[levmi]

Sorry for the delay! I think it'll be best to bring this to the Loop repo to investigate this further. Could you file an issue sharing the info shared initially in this repo? https://github.com/lightninglabs/loop/issues/new/choose

[guggero]

I transferred the issue to the correct repository. @MButcho do you have any additional logs since those posted two weeks ago?

[hieblmi]

Hi @MButcho, just to clarify, the loop out did obviously not succeed, only the prepayment seems to have been paid?

Also could you post the log lines after?

2024-01-25 13:38:14.647 [ERR] LOOP: 896ec1 swap subscription error: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: INTERNAL_ERROR```

[MButcho]

Hi @MButcho, just to clarify, the loop out did obviously not succeed, only the prepayment seems to have been paid?

Also could you post the log lines after?

2024-01-25 13:38:14.647 [ERR] LOOP: 896ec1 swap subscription error: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: INTERNAL_ERROR```

Correct, only the prepayment was sent, remaining funds HTLCs were resolved and were returned to node, prepayment was not. I don’t have the logs anymore, sadly 😏

[MButcho]

@levmi @guggero Any update guys?

[hieblmi]

taking another look.