-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: migration("re-indexing"), backfilling and diasgnostics tooling for the ChainIndexer
#12450
base: feat/msg-eth-tx-index
Are you sure you want to change the base?
feat: migration("re-indexing"), backfilling and diasgnostics tooling for the ChainIndexer
#12450
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions
chainindex/api.go
Outdated
return &types.IndexValidation{ | ||
TipsetKey: ts.Key().String(), | ||
Height: uint64(ts.Height()), | ||
Backfilled: true, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fill out the other fields here using SQL queries.
@rvagg Would be great to have a first round of review here when you get the time. |
ChainIndexer
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions
ChainIndexer
ChainIndexer
ChainIndexer
ChainIndexer
ChainIndexer
ChainIndexer
Co-authored-by: Rod Vagg <rod@vagg.org>
NonRevertedEventsCount uint64 | ||
Backfilled bool | ||
IndexedMessagesCount uint64 | ||
IndexedEventsCount uint64 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we rename this to IndexedNonRevertedMsgCount
and IndexedNonRevertedEventsCount
to be precise?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@akaladarshi That is an implementation detail that will confuse users. They're only asking for the indexed entries at a specific epoch here and so ofcourse expect only non-reverted tipsets back.
* feat: add lotus-shed command for backfilling chain indexer * feat: add lotus-shed command for inspecting the chain indexer * feat: use single lotus-shed command to inspect and backfill * fix: remove the unused queries * small changes * add change log
@BigLep This is now ready for review. Working on a user guide for this tooling as discussed offline. |
@rvagg This is now in a good shape and has been tested on a calibnet node. Please take a look when you get the time. |
_, _ = fmt.Fprintf(cctx.App.Writer, "✓ Epoch %d (%s)\n", epoch, string(jsonData)) | ||
} | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add a log for completion as well.
I think we can also print how many epochs were success out of total epoch or something like that.
…coin-project/lotus into feat/implement-index-validation-api
|
||
## Introduction | ||
|
||
We're shipping a new Indexer implementation in Lotus (`ChainIndexer`) to index Filecoin chain state such as tipsets, messages, events and ETH transactions for reliable and faster RPC responses. The `ChainIndexer` replaces the existing `MsgIndex`, `EthTxIndex` and `EventIndex` implementations in Lotus which suffer from a multitude of known problems documented [here](https://github.com/filecoin-project/lotus/issues/12293). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should probably mention which version we are shipping this is.
**Note: The following config options no longer exist in Lotus and have been removed in favor of the `ChainIndexer` config options explained above:** | ||
|
||
```toml | ||
[Fevm] | ||
EthTxHashMappingLifetimeDays = 0 | ||
|
||
[Events] | ||
DisableHistoricFilterAPI = false | ||
DatabasePath = "" | ||
``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe change it the Note to call out that these legacy configs can be removed when upgrading to the new version:
**Note: The following config options no longer exist in Lotus and have been removed in favor of the ChainIndexer
config options explained above. They can be removed when upgrading to the Lotus vX.XX.X version"
|
||
Migrating to the new `ChainIndexer` involves several steps to ensure a smooth transition. Here's a guide to help you through the process: | ||
|
||
1. **Backup Existing Index Databases** |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe be extra crisp here that the backup should be done while the node is stopped
**Note: This command can only be run when the Lotus daemon is already running with the `ChainIndexer` enabled as it depends on the `ChainValidateIndex` RPC API described above.** | ||
|
||
The `lotus-shed chainindex validate-backfill` command is a tool for validating and optionally backfilling the chain index over a range of epochs. It wraps the `ChainValidateIndex` API to efficiently process multiple epochs. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let us provide some basic on resource usage here, some potential questions from users I foresee is:
- How long does it take to backfill XX epochs.
- How much CPU / RAM does it require (i.e do we need to shift traffic to other nodes when running the backfill)
4. **Restart Lotus Node** | ||
- Restart your Lotus node with the new configuration. | ||
- The `ChainIndexer` will begin indexing **real-time chain state changes** immediately. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could be good to add some migration time estimates here when moving to the ChainIndexer?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Or to be clearer, it now sound a little bit like:
- Create a backup
- Remove Old Index
- Update Config
- Restart Lotus Node
And you should be all good. But for both snapshot synced / archival node operators it will take quite some time to do the backfilling?
## Migration Guide | ||
|
||
Migrating to the new `ChainIndexer` involves several steps to ensure a smooth transition. Here's a guide to help you through the process: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Time estimates for the migration for Snapshot synced and archival would also be good to add at the top level here.
ChainIndexer Migration and Diagnostics Tooling
This PR implements the "migration" (really re-indexing / backfilling), and diagnostics tooling for the
ChainIndexer
implemented in PR #12450, and is part of the work for #12453. This tooling takes the form of both RPC APIs on the daemon andlotus-shed
CLI commands.Re-indexing Process
The re-indexing tool enables clients to index their entire existing ChainState in the
ChainIndexer
. This process is necessary due to the removal of the existingMsgIndex
,EthTxIndex
, andEventIndex
from Lotus.Why Re-index Instead of Migrate?
We've chosen to re-index rather than migrate data from existing indices for two primary reasons:
Instead, we're re-indexing the
Chainstore
/Chainstate
on the node into theChainIndexer
. This ensures that all re-indexed entries have gone through the indexing logic of the newChainIndexer
and that the Index is in sync/reflects the actual contents of theChainstore
/Chainstate
post re-indexing.Diagnostics Tooling
This PR introduces diagnostic tools for detecting corrupt Index entries at specific epochs or epoch ranges.
While this PR implements functionality for optionally backfilling missing Index entries, it does not yet include the capability to "repair" corrupted Indexed entries. The repair functionality will be introduced in a subsequent PR. This approach allows us to first gather and analyze user reports, helping us understand the types and causes of corrupted Indexed entries(and if all they exist in the new
ChainIndexer
) before implementing repair mechanisms.Core API
The fundamental building block for this tooling is the following RPC API:
This API has the following features:
Chainstore
state and the Indexed entries (tipset messages/events)lotus-shed
CLI toolingThe
lotus-shed
CLI tooling for both re-indexing/backfilling and diagnostics can then invoke this RPC API over epoch ranges. The correspondinglotus-shed backfill index [from, to]
andlotus-shed inspect index [from, to]
can then backfill/inspect/diagnose the Index for the given epoch ranges.TODO
lotus-shed
commands