lnd v0.10.0-beta

This marks the first major release in the v0.10.0 series! This release contains several new features including support for sender-side MPP payment splitting, PSBT channel funding, and much more. Notably, this release also includes a new experimental commitment type (anchor commitments). As usual, this release also contains a number of optimizations and bug fixes.

Database Migrations

This release contains a single migration which modifies the structure of the on-disk representation of payments for the sake of the new MPP payment splitting feature. This migration should look something like this upon initial start up:

2020-04-15 18:19:18.113 [INF] LTND: Version: 0.10.0-beta. commit=v0.10.0-beta, build=development, logging=default
2020-04-15 18:19:18.376 [INF] LTND: Opening the main database, this might take a few minutes..
2020-04-15 18:19:18.380 [INF] CHDB: Checking for schema update: latest_version=13, db_version=12
2020-04-15 18:19:18.381 [INF] CHDB: Performing database schema migration
2020-04-15 18:19:18.381 [INF] CHDB: Applying migration #13
2020-04-15 18:19:18.381 [INF] CHDB: Migrating payments to mpp structure
2020-03-15 18:19:18.391 [INF] LTND: Database now open (time_to_open=15.651683ms)

Verifying the Release

In order to verify the release, you'll need to have gpg or gpg2 installed on your system. Once you've obtained a copy (and hopefully verified that as well), you'll first need to import the keys that have signed this release if you haven't done so already:

curl https://keybase.io/roasbeef/pgp_keys.asc | gpg --import

Once you have the required PGP keys, you can verify the release (assuming manifest-v0.9.2-beta.txt and manifest-v0.10.0-beta.txt.sig are in the current directory) with:

gpg --verify manifest-v0.10.0-beta.txt.sig

You should see the following if the verification was successful:

gpg: assuming signed data in 'manifest-v0.10.0-beta.txt'
gpg: Signature made Wed Apr 29 11:02:38 2020 PDT
gpg:                using RSA key 4AB7F8DA6FAEBB3B70B1F903BC13F65E2DC84465
gpg: Good signature from "Olaoluwa Osuntokun <laolu32@gmail.com>" [ultimate]

That will verify the signature of the manifest file, which ensures integrity and authenticity of the archive you've downloaded locally containing the binaries. Next, depending on your operating system, you should then re-compute the sha256 hash of the archive with shasum -a 256 <filename>, compare it with the corresponding one in the manifest file, and ensure they match exactly.

Verifying the Release Binaries

Our release binaries are fully reproducible. Third parties are able to verify that the release binaries were produced properly without having to trust the release manager(s). See our reproducible builds guide for how this can be achieved. The release binaries are compiled with go1.13.10, which is required by verifiers to arrive at the same ones. They include the following build tags: autopilotrpc, signrpc, walletrpc, chainrpc, invoicesrpc, routerrpc, and watchtowerrpc. Note that these are already included in the release script, so they do not need to be provided.

The make release command can be used to ensure one rebuilds with all the same flags used for the release. If one wishes to build for only a single platform, then make release sys=<OS-ARCH> tag=<tag> can be used.

Finally, you can also verify the tag itself with the following command:

git verify-tag v0.10.0-beta

Building the Contained Release

Users are able to rebuild the target release themselves without having to fetch any of the dependencies. In order to do so, assuming that vendor.tar.gz and lnd-source-v0.10.0-beta.tar.gz are in the current directory, follow these steps:

tar -xvzf vendor.tar.gz
tar -xvzf lnd-source-v0.10.0-beta.tar.gz
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.10.0-beta" ./cmd/lnd
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.10.0-beta" ./cmd/lncli

The -mod=vendor flag tells the go build command that it doesn't need to fetch the dependencies, and instead, they're all enclosed in the local vendor directory.

Additionally, it's now possible to use the enclosed release.sh script to bundle a release for a specific system like so:

make release sys="linux-arm64 darwin-amd64"

⚡️⚡️⚡️ OK, now to the rest of the release notes! ⚡️⚡️⚡️

Release Notes

Architectural Changes

As is today, the default database for lnd is bbolt, an embedded key-value store based on a B+Tree. In the future, we'd like to enable users to be able to swap out bbolt for any arbitrary strongly consistent database. In this release, we've made a step towards this goal by introducing a new KV-store database abstraction. This new interface allows any database implementation which can adhere to the interface to be dropped into lnd.

In the upcoming versions of lnd, the first new database we plan to support is etcd, a strongly consistent distributed key-value store which is used as the basis of projects such as Kubernetes. etcd actually uses bbolt underneath the hood, so it won't be a drastic departure. Once this issue to further abstract the initialization of lnd is complete, then users will be able to "build their own lnd", attaching a database implementation at runtime which adhere to the the primary interface.

Wallet Bug Fixes

A bug was fixed in btcwallet that would at times (non-deterministically) result in change addresses not being picked up doing a full wallet recovery instance. A bug related to map iteration order would at times cause a change address to be placed a "non-default scope" (where we store all the keys for contracts: multi-sig keys, etc), rather than the main "default scope". The end result is that due to this bug, it's possible that prior instances of full wallet recovery (seed import) missed some funds on chain. If you've ever recovered an existing lnd wallet, then ceased using it, we strongly recommend that you attempt the seed import process again, as there may be funds that weren't picked up the first time around.

Payments

Multi-Path Payments Sending Support

Receiving multi-path payments was already possible since version 0.9. Today we are pleased to announce that this release completes the circle by adding support for multi-path payment sending.

Preparatory changes for the send functionality that were implemented during the development process are database support for multiple htlcs per payment and a payment loop that is aware of partial sends.

The splitting algorithm that is employed is a variation of "divide and conquer". First, lnd will try to complete the payment in a single shot. If that is not possible - because channel capacities aren't sufficient or other constraints like the fee limit are hit - an attempt is made for half the payment amount. After that htlc has been launched, a second path is searched for to pay the remaining amount (the other half). If further failures happen, amounts are repeatedly split until a minimum amount limit is reached or the payment succeeds.

We opted for this relatively simple strategy to begin with. It is expected to address various basic liquidity problems that arose in the past, but it is unlikely to always execute payments in a way that is optimal. The field of multi-path planning is still mostly uncharted and contains lots of challenges. Therefore we hope to collect valuable feedback with this release and use that to drive further development.

Multi-path payments can only be initiated through the routerrpc.SendPayment rpc. Payments will be single-shot by default unless the new parameter max_shards is set to a value greater than one. max_shards controls the maximum number of paths that will be used for the payment.

The following lncli command can be used to make a multi-path payment:

lncli payinvoice --max_shards 5 <payreq>

Additionally, the payment limit has now be removed for calls to the new MPP-enabled payment API. This allows users to now send payments that fully utilize all their outgoing payment bandwidth.

Payment tracking

Improvements have been made to provide more fine-grained information about the status of payments.

A list of per-payment htlcs was already added in a prior release. This has now been extended with the exact failure reason for failed htlcs.

Furthermore the payment status stream that is returned by routerrpc.SendPayment and routerrpc.TrackPayment now also contains intermediate state updates. An example of this is the launch of an additional htlc. This will result in a state update on the stream.

The proto message on these update streams has been aligned to match the main rpc Payment message. This is a breaking change in the sub-server API.

Invoices

Lifted Invoice Limit

As part of the mpp changes, the invoice limit has been lifted from 4.2 million satoshis. With this change, receivers will now be able to create invoices that can be used in tandem with mpp at the sender, to fully utilize the channel bandwidth at both the sender and receiver to complete payments.

Channel Commitments & Funding

PSBT Funding

It is now possible to open channels by using a PSBT. This allows users to fund multiple channels in one on-chain transaction and/or open channels directly from a hardware wallet without the need for a transaction to lnd's wallet first. An example of how to use this new feature in combination with a bitcoind node can be found here.

NOTE: The normal funding dialog between the initiator's node and the remote node is paused after the multisig keys for the funding transaction have been exchanged. It is therefore possible that the remote node will not wait indefinitely for the flow to continue and cancel the funding reservation after some time. lnd nodes will cancel after 10 minutes of inactivity for example.

Future versions of lnd will expand the PSBT capabilities in order to allow generalized usage of the transaction packet format for all transaction crafting within the daemon.

Anchor commitment format

An new commitment format that includes anchor outputs has been added. Anchor outputs allow users to increase the fee of their commitment transaction after it has been published by making use of CPFP. The anchor output is force-swept in a child transaction that may be negatively yielding. It allows timely confirmation even if the miner fee of the commitment tx itself is insufficient.

To do so, the BumpFee rpc can be used to increase the fee of the total package. For channels in the waiting_close_channels stage, the following lncli command can be issued. Note that lnd must be built with the walletrpc build tag for this command to be available.

lncli wallet bumpclosefee 2b66cf489298b5b4a9305127413e4124e385588178e4b0fc4cec2d28c277b17d:0 --sat_per_byte 15

The specified fee rate is the fee rate of just the child transaction. It needs to be chosen such that the fee rate of the total package including the commitment transaction is bumped to the desired level.

To open a payment channel with anchors, both sides of the channel need to run lnd with --protocol.anchors. All newly opened channels between those peers will then automatically be of the anchor type.

To facilitate identification, the channel type is now exposed on ListChannels and PendingChannels.

Be aware that anchor channels are experimental. They don't yet have the reliability track record of the regular commitment format and therefore carry more risk. The specification hasn't been finalized yet, so there is also still a chance of breaking changes in future releases. In that case, lnd will likely require anchor channels to be closed before upgrading.

RPC Changes

Because lncli now uses the jsonpb JSON marshaler to print RPC responses, the json_name hints in the proto files are no longer necessary and have been removed.

The proto files can now be formatted with the make rpc-format command, if the clang-format binary is installed. More importantly, Travis CI now makes sure the proto files were compiled using the correct version of the protobuf compiler.

routerrpc is no longer conditionally compiled and from now on available in any build. lncli is also switched over to use the payment rpcs of routerrpc. The asynchronous SendPayment call on the main rpc has been deprecated.

The --reversed flag for the lncli listinvoices command has been removed in favor of a new flag: --paginate-forwards in order to make the functionality easier to use.

The ListPayments endpoint now supports pagination. The exposed API is similar to the existing pagination support for ForwardingHistory. On the command-line, users can set the index_offset and max_payments values to seek around within their past payment history.

bitcoind Support

A new config option has been added to allow the user to select which fee estimation modifier (the confidence parameter) we use when requesting fee estimates from bitcoind. users can now select from CONSERVATIVE and ECONOMICAL using the --bitcoind.estimatemode config setting.

Documentation

An Operational Safety Guidelines document has been added that advises users on how to safely operate a mainnet lnd node.

Tor

We now support authenticating to tor via the HASHEDPASSWORD authentication mechanism. This allow users to delpoy lnd with tor enabled, in configruation where lnd may not have access to the filesystem that the tor daemon uses.

Watchtowers

Watchtowers now have support for automatically setting up a Tor hidden service. This supports both v2 and v3 hidden services and can either be specified on the command line or via config file. Instructions on how to do this can be found here.

Testing+Security

lnd has a new fuzz package which contains fuzzing harnesses for the lnwire, wtwire, and brontide packages. The harnesses for lnwire and wtwire check that (de)serialization works as expected. The brontide harnesses test the validity of various states of the initial brontide handshake. Documentation on how to get started with fuzzing lnd can be found here.

Miscellaneous

To make automated unlocking of lnd nodes easier, the lncli unlock now accepts the password from the standard input, if the --stdin flag is passed.
NOTE: This is considered to be unsafe/dangerous if the password is located in a file that can be read by another user. The --stdin flag should only be used in combination with some sort of password manager or secrets vault.

Bug fixes

• The channel.backup file is now also updated while a channel is still pending. Previously a channel was only added to the SCB file once it was fully confirmed on-chain.

• The JSON marshaler used by the REST gateway by default omitted "falsey" values (values that evaluate to false in JSON/JavaScript, like 0, "", false). This was especially confusing/incorrect with enums. because the whole field was omitted if the first enum option was set (because that defaulted to 0 in the underlying gRPC transport). This behavior was fixed in the REST marshaler.

• A bug that would at times cause persistently high CPU usage with certain configurations using bitcoind has been resolved.

• Channel updates received from the remote party are now stored on disk until we've produced a signature that includes these updates. Previously the updates were only stored in memory and dropped across restarts. This could lead to the production of an invalid signature and channel force closure.

• A current limitation of the Lightning protocol is that it is possible for the channel initiator to reach a balance that is so close to the reserve, that it is no longer possible to send or receive htlcs with a non-dust value. The internal available channel balance calculation has been refined to take this possibility into account and report a lower balance if necessary to prevent the channel from reaching a degraded state.

• Node-level failures are now penalized harder in mission control. This prevents routes that are unlikely to succeed from being explored and thereby speeds up the payment process.

• An erroroneous log message has been nixed. Previously, this message would be printed upon each successful payment and caused some confusion for users when nothing was actually wrong.

The full list of changes since v0.9.2-beta can be found here:

• v0.9.2-beta...v0.10.0-beta

Contributors (Alphabetical Order)

Andras Banki-Horvath
bitromortac
Carla Kirk-Cohen
Carsten Otto
chokoboko
Conner Fromknecht
Craig Hammell
Danny Paz
Ediz Turcan
elkanatovey
ErikEk
Eugene Siegel
fguisso
Hampus Sjöberg
Johan T. Halseth
John Newbery
Joost Jager
MrManPew
Olaoluwa Osuntokun
Oliver Gugger
Roei Erez
Tomas Carnecky
Wilmer Paulino
Yaacov Akiba Slama
yancy
yyforyongyu