Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Proposal: Index node #113

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Proposal: Index node #113

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions proposals/113-index-node.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Indexer Node

Authors: @willscott

## What is the problem this project solves?
_Describe the status quo, including any relevant context on the problem you're seeing that this project should solve. Who is the user you're solving for, and why do they care about this problem? Wherever possible, include pain points or problems that you've seen users experience to help motivate why solving this problem works towards top-line objectives._

Currently, there isn't a way when a client asks for a cid - as is the current expectation in IPFS
to locate a miner and a piece that contains that cid within filecoin storage.

While the longer term answer for this problem is the full 'retrieval market', it would be good to have a shorter term answer.

## Impact
_What goals/OKRs are being addressed (for w3dt, a specific program, etc.)? Why is this project important? What do we get with this project that we can't get without it?_

This project aims to impact the goals of:
- low latency retrieval from filecoin

These are core goals that the Bedrock program aims to address.

## The idea
_Describe the proposed project solution, at a very high level. Stay at the level of the high-level requirements. Diagrams and interface descriptions can be useful, if you have any that help clarify and explain the idea._

As described in the [indexer node](https://www.notion.so/protocollabs/Indexer-Node-Design-4fb94471b6be4352b6849dc9b9527825?d=aa69bf93-0984-4070-8ced-b18207dc7734) design document we propose a logically centralized index of cids.
Internally it will be implemented by a pool of nodes partitioning the cid space by prefix.
Each node would keep a cache in memory of requested CIDs, and a sorted index on disk of the full list of CIDs.

## Success/acceptance criteria (optional)
_How do we know we're done with this project? How do we know we're successful? This field is OPTIONAL for the first draft of an MPP. Sometimes this field needs to be filled out once we have more detail on the shape of the actual solution._

Initial success/acceptance criteria:
- We can confidently provide a scale of cids we believe this design scales to for sub 10ms query lookups.
- we can we build an index and can answer queries against the current scale of stored filecoin data.
- we have a library implementation of the index that can also be used within a sharded store within miners.


## Detailed plans (optional)
_Link to more detailed project plans, e.g. product requirements documents (PRDs) and technical design docs, once they have been created for this project._

Described in the (currently private) [design doc](https://www.notion.so/protocollabs/Indexer-Node-Design-4fb94471b6be4352b6849dc9b9527825?d=aa69bf93-0984-4070-8ced-b18207dc7734)

## Program (optional)
_If this project is part of a program, please indicate the relevant program here._

This project is part of the golden path. It most directly affects **Bedrock** program OKRs.