Panic: "failed to update wallet"

[ChrisSchinnerl]

Current Behavior

ERROR data failed to persist data usage {"error": "transaction failed (attempt 30): failed to track ingress: failed to insert stat: database is locked"}
PANIC wallet failed to update wallet {"error": "transaction failed (attempt 30): failed to set last wallet change: database is locked", "changeID": "26ab4e3ea842d16cb13a5925ba44ead37c095133ac2abf73341798fd16d5a878", "height": 59961}
panic: failed to update wallet

goroutine 12017740 [running]:
go.uber.org/zap/zapcore.CheckWriteAction.OnWrite(0x80?, 0x3?, {0x3?, 0x0?, 0x0?})
go.uber.org/zap@v1.26.0/zapcore/entry.go:196 +0x78
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0x4000aa0000, {0x4001176480, 0x3, 0x3})
go.uber.org/zap@v1.26.0/zapcore/entry.go:262 +0x2c0
go.uber.org/zap.(*Logger).Panic(0x6cd142a83e4eab26?, {0xf96734?, 0x40?}, {0x4001176480, 0x3, 0x3})
go.uber.org/zap@v1.26.0/logger.go:284 +0x54
go.sia.tech/hostd/wallet.(*SingleAddressWallet).ProcessConsensusChange(_, {{0x26, 0xab, 0x4e, 0x3e, 0xa8, 0x42, 0xd1, 0x6c, 0xb1, ...}, ...})
go.sia.tech/hostd/wallet/wallet.go:749 +0x13e0
go.sia.tech/siad/modules/consensus.(*ConsensusSet).updateSubscribers(0x40000d6b00, {{0x0, 0x0, 0x0}, {0x40010d19e0, 0x1, 0x1}})
go.sia.tech/siad@v1.5.10-0.20230228235644-3059c0b930ca/modules/consensus/subscribe.go:136 +0x244
go.sia.tech/siad/modules/consensus.(*ConsensusSet).managedAcceptBlocks(0x40000d6b00, {0x40005b1c78, 0x1, 0x1})
go.sia.tech/siad@v1.5.10-0.20230228235644-3059c0b930ca/modules/consensus/accept.go:318 +0x5bc
go.sia.tech/siad/modules/consensus.(*ConsensusSet).threadedRPCRelayHeader.func5.(*ConsensusSet).managedReceiveBlock.func1({0x1a5bd18?, 0x400223ab20})
go.sia.tech/siad@v1.5.10-0.20230228235644-3059c0b930ca/modules/consensus/synchronize.go:535 +0xfc
go.sia.tech/siad/modules/gateway.(*Gateway).managedRPC(0x400054c900, {0x400017c2d0, 0x13}, {0xf851c2, 0x7}, 0x400058ac30)
go.sia.tech/siad@v1.5.10-0.20230228235644-3059c0b930ca/modules/gateway/rpc.go:70 +0x26c
go.sia.tech/siad/modules/gateway.(*Gateway).RPC(0x400054c900, {0x400017c2d0, 0x13}, {0xf851c2, 0x7}, 0x40001b0460?)
go.sia.tech/siad@v1.5.10-0.20230228235644-3059c0b930ca/modules/gateway/rpc.go:83 +0xb0
go.sia.tech/siad/modules/consensus.(*ConsensusSet).threadedRPCRelayHeader.func5()
go.sia.tech/siad@v1.5.10-0.20230228235644-3059c0b930ca/modules/consensus/synchronize.go:465 +0x15c
created by go.sia.tech/siad/modules/consensus.(*ConsensusSet).threadedRPCRelayHeader in goroutine 12017738
go.sia.tech/siad@v1.5.10-0.20230228235644-3059c0b930ca/modules/consensus/synchronize.go:463 +0x39c

Expected Behavior

No panic

Steps to Reproduce

Version

0b73159

What operating system did the problem occur on (e.g. Ubuntu 22.04, macOS 12.0, Windows 11)?

Anything else?

Reported by newOne on Discord

[n8maninger]

This should improve when we reduce the number of active database writers in #264. The panic behavior is correct since it effectively couldn't apply a consensus change after 20 transaction attempts.