StreamingFast Firehose node instrumentation

[AngeloCa]

Rationale

Why should this feature exist?

This feature will improve Arbitrum data availability.

Firehose is a new technology developed by StreamingFast working with The Graph Foundation. The product provides previously unseen capabilities and speeds for indexing blockchain data using a files-based and streaming-first approach.

Firehose is responsible for extracting data from blockchain nodes in a highly efficient manner. Firehose also consumes, processes, and streams blockchain data to consumers of nodes running Firehose-enabled, instrumented blockchain client software.

What are the use cases?

From a OffchainLabs blockchain core development team’s perspective:

• Immediate support for Substreams, and its high-speed parallel processing engine.
• Immediate integration into The Graph, through Substreams-based Subgraphs.
• Immediate integration into the latest graph node which can feed directly from the Firehose.
• Immediate support for dozens of sinks (think loading to PostgreSQL, MongoDB, CDC replication, Kafka, Flink, writing to buckets for ingestion by BigQuery, AWS Redshift, Clickhouse, etc.).
• Activates a community of developers that already know how to index blockchain data, and are going to be happy to discover your blockchain’s data model and activity.

From Arbitrum users’ perspective:

• The capability to quickly index and sift through your favorite blockchain’s history.
• Stream its output with low latency.
• Hook it to any data systems you have, trading bots, cross-chain bridges, databases.
• All of it in an extremely reliable way.
• Benefit from the anti-fragility brought by The Graph network.
• Offer an army of indexers to service your network.

Implementation

Do you have ideas regarding the implementation of this feature?

Read StreamingFast Docs to integrate new chains.