Status: Draft
The CAR format (Content Addressable aRchives) can be used to store content addressable objects in the form of IPLD block data as a sequence of bytes; typically in a file with a .car filename extension.
The CAR format is intended as a serialized representation of any IPLD DAG (graph) as the concatenation of its blocks, plus a header that describes the graphs in the file (via root CIDs). The requirement for the blocks in a CAR to form coherent DAGs is not strict, so the CAR format may also be used to store arbitrary IPLD blocks.
In addition to the binary block data, storage overhead for the CAR format consists of:
- A header block encoded as DAG-CBOR containing the format version and an array of root CIDs
- A CID for each block preceding its binary data
- A compressed integer prefixing each block (including the header block) indicating the total length of that block, including the length of the encoded CID
This diagram shows how IPLD blocks, their root CID, and a header combine to form a CAR.
The name Certified ARchive has also previously been used to refer to the CAR format.
The CAR format comprises a sequence of length-prefixed IPLD block data, where the first block in the CAR is the Header encoded as CBOR, and the remaining blocks form the Data component of the CAR and are each additionally prefixed with their CIDs. The length prefix of each block in a CAR is encoded as a "varint"—an unsigned LEB128 integer. This integer specifies the number of remaining bytes for that block entry—excluding the bytes used to encode the integer, but including the CID for non-header blocks.
|--------- Header --------| |---------------------------------- Data -----------------------------------|
[ varint | dag-cbor block ] [ varint | CID | block ] [ varint | CID | block ] [ varint | CID | block ] …
The first bytes of the CAR format hold a varint, this unsigned integer specifies the number of bytes beyond the varint itself that contain the Header block. This Header block is a byte array DAG-CBOR (CBOR with tag 42 for CIDs) encoded object holding the version number and array of roots. As an IPLD Schema:
type CarHeader struct {
version Int
roots [&Any]
}
- The
versionis always a value of1. Future iterations of this specification may make use ofversionto introduce variations of the format. - The
rootsarray must contain one or more CIDs, each of which should be present somewhere in the remainder of the CAR.
(Caveats: see Number of roots and Root CID block existence under Unresolved Issues.)
Immediately following the Header block, one or more IPLD blocks are concatenated to form the Data section of the CAR format. (Caveat: see Zero blocks under Unresolved Issues.) Each block is encoded into a Section by the concatenation of the following values:
- Length in bytes of the combined CID and data in this Section, encoded as a varint
- CID of the block in this Section, encoded in the raw byte form of the CID
- Binary data of the block
Each Section begins with a varint representation of an unsigned integer indicating the number of bytes containing the remainder of the section.
Following the Length, the CID of the block is included in raw byte form. A decoder reading a Section must decode the CID according to CID byte encoding rules, which don't provide a stable length. See https://github.com/multiformats/cid for details on the encoding of a CID. CIDv0 and CIDv1 are currently supported. (Caveat: see CID version under Unresolved Issues.)
CID byte decoding summary
See the CID specification for full details.
A CIDv0 is indicated by a first byte of 0x12 followed by 0x20 which specifies a 32-byte (0x20) length SHA2-256 (0x12) digest.
Failure to find 0x12, 0x20 indicates a CIDv1 which is decoded by reading:
- Version as an unsigned varint (should be
1) - Codec as an unsigned varint (valid according to the multicodec table)
- The raw bytes of a multihash
Reading the multihash requires a partial decode in order to determine the length:
| hash function code (varint) | digest size (varint) | digest |
The first two bytes of a multihash are varints, where the second varint is an unsigned integer indicating the length of the remaining portion of the multihash. Therefore, a manual decode requires two varint reads and then copying the bytes of those varints in addition to the number of bytes indicated by the second varint into a byte array.
The remainder of a Section, after length-prefix and CID, comprises the raw byte data of the IPLD block. The encoded block may be any IPLD block format as specified by the codec in the CID. Typical codecs will be DAG-PB, DAG-CBOR or RAW.
Deterministic CAR creation is not covered by this specification. However, deterministic generation of a CAR from a given graph is possible and is relied upon by certain uses of the format, most notably, Filecoin.
Additional rules for the generation of the CAR format may be applied in order to ensure that the same CAR is always generated from the same data. The burden of this determinism is primarily placed on selectors whereby a given selector applied to a given graph will always yield blocks in the same order regardless of implementation.
Care regarding the ordering of the roots array in the Header, as well as consideration for CID version (see below) and avoidance of duplicate blocks (see below) may also be required for strict determinism.
All such considerations are deferred to the user of the CAR format and should be documented there as this specification does not inherently support determinism.
Some considerations regarding performance:
- Streaming: the CAR format is ideal for dumping blocks via streaming reads as the Header can be loaded first and minimal state is required for ongoing parsing.
- Individual block reads: as the CAR format contains no index information, reads require either a partial scan to discover the location of a required block or an external index must be maintained and referenced for a seek and partial read of that data. See below regarding indexing.
- DAG traversal: without an external index, traversal of a DAG specified by a "root" CID is not possible without dumping all blocks into a more convenient data store or by partial scans to find each block as required, which will likely be too inefficient to be practical.
- Modification: CARs may be appended after initial write as there is no constraint in the Header regarding total length. Care must be taken in appending if a CAR is intended to contain coherent DAG data.
The roots specified by the Header of a CAR must appear somewhere in its Data section, however there is no requirement that the roots define entire DAGs, nor that all blocks in a CAR must be part of DAGs described by the root CIDs in the Header. Therefore, the roots must not be used alone to determine or differentiate the contents of a CAR.
The CAR format contains no internal means, beyond the IPLD block formats and their CIDs, to verify or differentiate contents. Where such a requirement exists, this must be performed externally, such as creating a digest of the entire CAR.
The CAR format contains no internal indexing, any indexing must be stored externally to a CAR. However, such indexing is possible and makes seeking reads practical. An index storing byte offset (of Section start or block data start) and length (of Section or block data), keyed by CID, will enable a single block read by seeking to the offset and reading the block data. The format of any such index is not specified here and is left up to CAR format parsing implementations.
The CAR format contains no specified means of padding to achieve specific total archive sizes or internal byte offset alignment of block data. Because it is not a requirement that each block be part of a coherent DAG under one of the roots of the CAR, dummy block entries may be used to achieve padding. Such padding should also account for the size of the length-prefix varint and the CID for a section. All sections must be valid and dummy entries should still decode to valid IPLD blocks.
https://github.com/ipfs/go-car
As used in Filecoin for genesis block sharing. Supports creation via a DAG walk from a datastore:
WriteCar(ctx context.Context, ds format.DAGService, roots []cid.Cid, w io.Writer) (error)And writing to a data store via Put(block) operations:
LoadCar(s Store, r io.Reader) (*CarHeader, error)https://github.com/rvagg/js-datastore-car
Wraps in Datastore interface with various modes for reading and writing to support different use-cases—including streaming reading and writing:
async CarDatastore.readBuffer(buffer)
async CarDatastore.readFileComplete(file)
async CarDatastore.readStreamComplete(stream)
async CarDatastore.readStreaming(stream)
async CarDatastore.writeStream(stream)Also supports an indexer() that parses a file or stream and yields block index data including CID, offset and length, in addition to a readRaw() to read individual blocks according to their index data.
Regarding the roots property of the Header block:
- The current Go implementation assumes at least one CID when creating a CAR
- The current Go implementation requires at least one CID when reading a CAR
- The current JavaScript implementation allows for zero or more roots
- Current usage of the CAR format in Filecoin requires exactly one CID
It is unresolved how the roots array should be constrained, but it is recommended that only a single root CID be used in this version of the CAR format.
It is unresolved whether a valid CAR must contain at least one block or whether the empty CAR is a valid format and should be accepted by encoders and decoders.
It is unresolved whether an implementation must verify that a CID present in the roots array of the Header also appears as a block in the archive. While it is expected that this would be the case, it is unresolved whether encoders and decoders must validate the existence of root blocks in the archive.
It is unresolved whether both CID versions 0 and 1 format are valid in the roots array and at the start of each block Section. Current implementations do not check CID version in the roots array, and both CUD versions are also acceptable in each block Section. Discussions on this specification have suggested limiting CIDs used throughout the format (not within blocks) to CIDv1—requiring conversion if an encoder is provided with a CIDv0 and requiring readers of a CAR to ensure CIDv1 is the only available block key.
The possibility of duplicate blocks in a single CAR (such as for padding) is currently not specified.