Migration to CIDv1 (default base32) #247

kyledrake · 2018-07-12T13:27:38Z

Lowercase identifiers are required in order to work with legacy URI, web browser security (sub)origins and the experimental Protocol Handler API. So we are moving to finish support for CIDv1 across the ecosystem, and use it as default (while still supporting CIDv0/Multihash).

There are some complications related to this switch, such as refactoring the blockstore to use multihashes instead of cid identifiers. A number of temporary solutions were proposed that would have made the change simpler with the current design, however it was decided that since this is going to be a significant update, we should focus on a full CIDv1 migration with one identifier change.

This is the parent issue for all issues related to migration to CIDv1 with default base32 (type 'b'). This issue closes when CIDv1b32 is enabled by default for new versions of go-ipfs, js-ipfs, and ipfs-companion.

@kevina @magik6k @lidel @whyrusleeping @diasdavid @alanshaw @olizilla @kyledrake @lgierth @mikeal

Berlin Jul 2018 developer meeting notes:
ipfs-inactive/developer-meetings#39

Initial proposal:
ipfs/kubo#4143

🐦 Bird's-eye view (WIP by @lidel)

Located here: https://gist.github.com/lidel/ceef96f60f30308e3c07be95182c4c88 (most likely out-of-date!)

Kubuxu · 2018-07-15T12:19:14Z

cc myself

daviddias · 2018-08-12T10:17:21Z

@kyledrake can you give an update on the status of this migration in the next IPFS All Hands?

kyledrake · 2018-08-12T19:17:22Z

I will be unavailable for the call unfortunately, I'm moving tomorrow. @lidel do you think you could give a quick update? It doesn't need to be complicated, just a quick summary of progress based on the referenced issues on 337.

kevina · 2018-08-12T19:30:07Z

@kyledrake @lidel @diasdavid I am the most qualified to give this update for go-ipfs. I will put something in the notes.

lidel · 2018-11-02T11:59:03Z

js-ipfs v0.33.0 and go-ipfs v0.4.18 shipped with some relevant changes:

This release makes some significant progress towards solving this issue by introducing two features:

(1) [..] ipfs cid base32 command for converting CID to a case intensive encoding required by domain names. This command converts a CID to version 1 and encodes it using base32.

(2) A hack to allow locally looking up blocks associated with a CIDv0 CID using the equivalent CIDv1 CID (or the reverse). This hack will eventually be replaced with a multihash indexed blockstore, which is agnostic to both the CID version and multicodec content type.

– Release Notes for go-ipfs v0.4.18

This PR adds a jsipfs cid command with tools for formatting, converting and discovering properties of CIDs. All the docs are here: https://github.com/ipfs-shipyard/js-cid-tool
– ipfs/js-ipfs#1560

The newly added ipfs cid subcommand (1) makes conversion super easy:

# cidv0(b58) → cidv1(b32)
$ ipfs cid base32 QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR
bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi

# cidv1(b58) → cidv1(b32)
$ ipfs cid base32 zb2rhk6GMPQF3hfzwXTaNYFLKomMeC6UXdUt6jZKPpeVirLtV
bafkreigh2akiscaildcqabsyg3dfr6chu3fgpregiymsck7e7aqa4s52zy

daviddias · 2018-12-03T18:36:23Z

@kyledrake can you compile a status update on this whole endeavour? How far are we from shipping it? Will there be a big launch?

kyledrake · 2018-12-06T20:10:17Z

@daviddias The referenced issues here show the current state of things pretty well. We're making good progress on better base32 support and are definitely at a point where we want to decide if we're comfortable with planning for a switch.

I would like an opinion from @alanshaw and @kevina on whether they think they're at a point where we would be comfortable migrating to base32, and if there's any blockers or issues on the technical side of things that would make them uncomfortable with switching until resolving. Then we can move to setting dates for transition, getting out announcements, etc.

daviddias · 2019-01-13T16:10:50Z

Is there a timeline for when to expect this work to be finalized? What's the plan for the launch? How will the users be informed? Any caveats to the migration?

kyledrake · 2019-01-16T08:57:14Z

@daviddias We're getting closer to the finish line, so now is a good time to start discussing a rough timeline/plan.

1) Release go-ipfs, js-ipfs, ipfs-companion versions with optional cidv1-base32 support

js-ipfs is going to get there with the 0.34 release (ipfs/js-ipfs#1721), and @alanshaw has set 0.36 as the version with default cidv1-base32 as a rough plan (ipfs/js-ipfs#1721 (comment)), which I'm OK with. Absolutely incredible work by the js-ipfs team here!

go-ipfs has a pull request pending (ipfs/kubo#5789) that will bring go-ipfs up to speed. I'm really excited to see this merged and finally put my hshca hack to rest. @kevina has done a ton of work on this (and had to re-do a lot of that work for various reasons) so kudos to him and everyone else at go-ipfs that has put a lot of good work into this.

I'm not sure what the timeline looks like for getting this in go-ipfs yet, @kevina @Stebalien is this something you foresee getting merged into master soon?

I'll let @lidel give us an update on ipfs-companion, it will be better than mine. Tracker issue for ipfs-companion is ipfs/ipfs-companion#527

Dates: No hard dates for these releases, but I would like to see them released within the next month if it's not too inconvenient. Most of the work is done here.

Step 3 happens after we get releases of js-ipfs, go-ipfs, ipfs-companion which have the (optional, with arguments) support. Step 2 can be done in parallel:

2) Prepare the gateway for subdomains (*.dweb.link)

This will also be a good point to encourage users to start using the subdomains instead of the ipfs.io gateway, *.ipfs.dweb.link and *.ipns.dweb.link (cc @eefahy @mburns), getting content/apps on their own security origin. But we'll need to make sure whatever we choose works well with ipfs-companion (we good here @lidel?)

The main missing pieces right now:

The *.ipfs.dweb.link and *.ipns.dweb.link need SSL certificates (https://test.ipfs.dweb.link returns a bad cert)
A default redirect to HTTPS from HTTP
An HSTS preload header so we can get them added to the browser lists.

Dates: would be nice to see this done within the next month as well.

3) Blog post with new cidv1-base32 information

We will write a blog post indicating that we will be switching to cidv1-base32, why we are making this change, and how users can start using it now with optional arguments. This will get people more comfortable working with it, show them the new gateway URLs they can (really should!) use, and give them time to prepare.

We absolutely need to release both go-ipfs and js-ipfs (also ipfs-companion?) at the same time with the new default support, so a set date will eventually be needed.

Dates: I will publish this very soon after the optional support releases are made. 2-4 days to complete.

4) Update documentation to reflect use of cidv1-base32

When we release with the new defaults, we are going to have a lot of documentation across the organization that references the old output. We will need to prepare the documentation to reflect these changes when we do the release, particularly the docs portal.

Dates: Work will begin soon after optional support releases are made. Rough, hardly educated guess is 2 weeks of work.

5) Release go-ipfs, js-ipfs, ipfs-companion, documentation versions with cidv1-base32 support

We release everything at the same time, and add to a blog post announcing switch to cidv1-base32 and re-hashing a lot of what was stated in the first blog post.

Dates:
I'm not sure about a hard date for final release, it really depends on how comfortable we are with the current progress, I don't want to rush anybody. To throw out a few random dates: Spring Equinox is on March 20th and Summer Solstice is June 21st. I'm leaning towards a date closer to Solstice, as it would give us more room to fix any lingering issues and get some user testing in. But if we're feeling comfortable with progress, we can change to an earlier date. I think we'll have a better feel for things once Step 1 is finished.

Let's call this "proposed rough plan". Feel free to add any suggestions/missing info.

kyledrake · 2019-01-17T13:03:04Z

Adding a missing piece from my writeup:

go-ipfs 0.4.18 shipped with some ipfs cid changes for cidv1-base32. It gives us some clever ways to use it with the currently released version using pipes / additional API calls. For example:

$ echo "hello world" | ipfs add -q | ipfs cid base32
bafybeicg2rebjoofv4kbyovkw7af3rpiitvnl6i7ckcywaq6xjcxnc2mby
$ ipfs cat bafybeicg2rebjoofv4kbyovkw7af3rpiitvnl6i7ckcywaq6xjcxnc2mby
hello world
$ curl http://bafybeicg2rebjoofv4kbyovkw7af3rpiitvnl6i7ckcywaq6xjcxnc2mby.ipfs.dweb.link
hello world

So if used this way, you could technically start using cidv1-base32 with go-ipfs already (for a lot of common uses).

More info from the blog post: https://blog.ipfs.io/53-go-ipfs-0-4-18/#cidv1-base32-migration

lidel · 2019-01-23T13:11:55Z

ipfs-companion detects <cidv1b32>.ipfs.foo.tld and displays page actions:

When js-ipfs/go-ipfs backends switch their defaults to cidv1b32, ipfs-companion will support it automatically.

Ad. preparing gateway for subdomains, there is an open question about how ipfs-companion should handle redirects of CID-in-Subdomain and DNSLink websites: ipfs/ipfs-companion#667 ← would appreciate feedback on proposed solution.

kyledrake · 2019-02-18T15:19:03Z

Heads up that @lidel is now the coordinator for this work.

lidel · 2019-03-03T18:43:18Z

🏆 `dweb.link` is available over HTTPS!

Demo: https://bafkreigh2akiscaildcqabsyg3dfr6chu3fgpregiymsck7e7aqa4s52zy.ipfs.dweb.link

(thanks @scout!)

ℹ️ Plan to support CID-in-subdomain via HTTP Proxy

http://<cidv1b32>.ipfs.localhost

To provide Origin-based isolation for websites loaded from local gateway we are planning to use a special domain names in .localhost space. In the long term, we could move to a dedicated Special-Use Domain Name .ipfs.dweb.

We don't want to mess with people's DNS servers, so the cross-platform support for this relies on the idea of on-the-fly domain support via go-ipfs as HTTP proxy (proposed in ipfs/kubo#5982, ipfs/js-ipfs#1877) + ipfs-companion setting go-ipfs a proxy for specific requests.

If proxied hostname is customizable, the same HTTP Proxy mode could be used on public gateways such as dweb.link: ipfs/infra#81 (comment)

ℹ️ WIP in js-ipfs: change defaults for adding content to cidv1b32

List of PRs in ipfs/js-ipfs#1440 (comment)

ℹ️ WIP in go-ipfs: switching DHT (& datastores) to raw multihashes

Some relevant go-ipfs updates were discussed at Golang Core Dev Team Weekly Sync 2019-02-11.
See Base32/CIDv1 section or TL;DR:

Q: What's the priority on this Base32/CIDv1 list?
A: Most important next milestone is storing/fetching [raw] multihashes:
(Issue on DHT, WIP datastore changes)

lidel · 2019-05-10T11:21:55Z

PSA: Switching CIDv1 to Base32

We are in the process of switching text representation of CIDv1 from Base58btc to Base32.
Details in ipfs/kubo#6220 and ipfs/js-ipfs#1995

Note: This is only about changing the default text representation of CIDv1.
CIDv0 in Base58btc is still the default. CIDv1 remains an opt-in.

lidel · 2019-05-27T23:18:17Z

PSA: Adding support for CIDv1 in IPNS

We are adding a new multicodec to unlock PeerID/KeyID-in-CIDv1 in subdomains:

http://<keyid-in-cidvb32>.ipns.localhost

More details in multiformats/multicodec#130 & ipfs/kubo#5287

There is the issue that ipfs URLs are currently case-sensitive with base58 hashes, but this has been recognized as a problem: https://github.com/ipfs/ipfs/issues/337 QUrl also converts addresses to lowercase. So we can't currently load base58 hashes into QUrl; consequently switched load(QUrl) to load(QString) for now. Maybe it can be reverted when they switch to base32 hashes.

lidel · 2019-06-19T17:46:02Z

Cool update: Cloudflare officially supports CIDv1B32 in subdomains:
https://blog.cloudflare.com/continuing-to-improve-our-ipfs-gateway/#subdomain-based-gateway

Example: https://bafybeiemxf5abjwjbikoz4mc3a3dla6ual3jsgpdr4cjr3oz3evfyavhwq.cf-ipfs.com/wiki/

lidel · 2019-08-23T09:29:28Z

PSA: Opened RFC (change to libp2p SPEC) to switch default text representation of PeerIDs to CID and make Base32 the default mutlibase: libp2p/specs#209
This upstream change is part of effort to unlock use of libp2p-keys in subdomains (ipfs/kubo#5287).

hsanjuan · 2020-03-27T18:37:54Z

I am transferring this issue to ipfs/specs as we want to clean up this repo.

Progress in Go (the missing part) can be followed here: ipfs/kubo#4143

lidel · 2020-04-16T16:00:19Z

📣 PSA: go-ipfs 0.5.0 will bring native support for PeerIDs as CIDv1 & CID-in-subdomain

TL;DR Support for PeerIDs as CIDv1 (ipfs/kubo#5287) and native support for CID-in-subdomain (AKA subdomain gateways) landed in go-ipfs (ipfs/kubo#6096) and will ship with go-ipfs 0.5.0 (ipfs/kubo#7109)

HTTP Gateway

When subdomain gateway is used as a path one, go-ipfs will automatically convert case-sensitive identifiers to base32 and redirect to a subdomain URL. For example, given a subdomain gateway enabled on localhost:8080:

http://localhost:8080/ipfs/QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn → HTTP 301 → http://bafybeiczsscdsbs7ffqz55asqdf3smv6klcw3gofszvwlyarci47bgf354.ipfs.localhost:8080

When /ipns/ path is used, gateway will convert to CIDv1 with libp2p-key multicodec.

Instructions how to run a subdomain gateway can be found in go-ipfs/docs/config.md:

CLI / RPC API

Command line and RPC API provide handy error message when an invalid multicodec is used in CIDv1:

# resolve Base58 PeerID
$ ipfs name resolve -r /ipns/QmY9DNm4UJxaTBB9FrhNkXQ75GaK43D8KhZtpJjmw8P7Sz
/ipfs/bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi

# convert PeerID to CIDv1 (with invalid multicoded, to illustrate next step)
$ ipfs cid base32 QmY9DNm4UJxaTBB9FrhNkXQ75GaK43D8KhZtpJjmw8P7Sz
bafybeieru3k6wvdzsrsjcx33jund7ebawoqlyc7jpitmn7wd4cwveunolm

# resolve CIDv1 with dag-pb multicodec
$ ipfs name resolve -r /ipns/bafybeieru3k6wvdzsrsjcx33jund7ebawoqlyc7jpitmn7wd4cwveunolm
Error: peer ID represented as CIDv1 require libp2p-key multicodec: retry with /ipns/bafzbeieru3k6wvdzsrsjcx33jund7ebawoqlyc7jpitmn7wd4cwveunolm

# resolve CIDv1 with libp2p-key multicodec suggested in previous step
$ ipfs name resolve -r /ipns/bafzbeieru3k6wvdzsrsjcx33jund7ebawoqlyc7jpitmn7wd4cwveunolm
/ipfs/bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi

lidel · 2021-12-02T15:07:16Z

🕸️ 🕷️ 🎩 brushing off the dust

DONE: CIDv0 creation was removed from go-ipfs 0.10 DAG API
- tldr: ipfs dag put produces CIDv1 with dag-cbor codec by default, but one can tweak the put behavior via --input-codec and --store-codec options.
IN PROGRESS: Switch to raw multihashes in datastores is planned for go-ipfs 0.12, after we are done with that we can start switching the defaults in the rest of IPFS ecosystem

lidel · 2022-04-08T00:03:49Z

go-ipfs 0.12 shipped the blockstore migration from full CID to Multihash keys.

Finally, we can do the thing! 🙀

Started poking at go-ipfs to produce CIDv1 by default – tracking remaining work in ipfs/kubo#4143 (comment)

kyledrake self-assigned this Jul 12, 2018

lidel assigned lidel and unassigned kyledrake Feb 18, 2019

lidel unassigned lidel May 27, 2019

daviddias pinned this issue Oct 8, 2019

jessicaschilling unpinned this issue Mar 27, 2020

hsanjuan transferred this issue from ipfs/ipfs Mar 27, 2020

lidel mentioned this issue Apr 28, 2020

Companion support for Base32 CIDv1 ipfs/ipfs-companion#527

Closed

3 tasks

aschmahmann mentioned this issue Jul 8, 2020

Use newer PeerID as CID representation libp2p/go-libp2p#976

Open

OR13 mentioned this issue Dec 1, 2020

Spec vs Verify CAS file URI decentralized-identity/sidetree#927

Closed

b5 mentioned this issue Sep 28, 2021

base58 bitcoin encoding is considered legacy w3c-ccg/did-method-key#21

Open

BigLep added this to the Best Effort Track milestone Apr 7, 2022

lidel mentioned this issue Apr 7, 2022

test: ipfs add with CIDv1 as default ipfs/kubo#8185

Draft

7 tasks

lidel mentioned this issue Apr 8, 2022

Make base32 CIDv1 the default for go-ipfs ipfs/kubo#4143

Open

sergiimk mentioned this issue Apr 17, 2022

Consider going back to base16 hash encoding open-data-fabric/open-data-fabric#23

Closed

b5 mentioned this issue Feb 17, 2023

Use CIDv1 for all dags constructed by iroh, output string representations in base32 CIDv1 n0-computer/beetle#137

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migration to CIDv1 (default base32) #247

Migration to CIDv1 (default base32) #247

kyledrake commented Jul 12, 2018 •

edited by lidel

Loading

Kubuxu commented Jul 15, 2018

daviddias commented Aug 12, 2018

kyledrake commented Aug 12, 2018

kevina commented Aug 12, 2018 •

edited

Loading

lidel commented Nov 2, 2018 •

edited

Loading

daviddias commented Dec 3, 2018

kyledrake commented Dec 6, 2018

daviddias commented Jan 13, 2019

kyledrake commented Jan 16, 2019

kyledrake commented Jan 17, 2019 •

edited

Loading

lidel commented Jan 23, 2019

kyledrake commented Feb 18, 2019

lidel commented Mar 3, 2019 •

edited

Loading

lidel commented May 10, 2019

lidel commented May 27, 2019

lidel commented Jun 19, 2019

lidel commented Aug 23, 2019

hsanjuan commented Mar 27, 2020

lidel commented Apr 16, 2020 •

edited

Loading

lidel commented Dec 2, 2021 •

edited

Loading

lidel commented Apr 8, 2022 •

edited

Loading

Migration to CIDv1 (default base32) #247

Migration to CIDv1 (default base32) #247

Comments

kyledrake commented Jul 12, 2018 • edited by lidel Loading

🐦 Bird's-eye view (WIP by @lidel)

Kubuxu commented Jul 15, 2018

daviddias commented Aug 12, 2018

kyledrake commented Aug 12, 2018

kevina commented Aug 12, 2018 • edited Loading

lidel commented Nov 2, 2018 • edited Loading

daviddias commented Dec 3, 2018

kyledrake commented Dec 6, 2018

daviddias commented Jan 13, 2019

kyledrake commented Jan 16, 2019

1) Release go-ipfs, js-ipfs, ipfs-companion versions with optional cidv1-base32 support

2) Prepare the gateway for subdomains (*.dweb.link)

3) Blog post with new cidv1-base32 information

4) Update documentation to reflect use of cidv1-base32

5) Release go-ipfs, js-ipfs, ipfs-companion, documentation versions with cidv1-base32 support

kyledrake commented Jan 17, 2019 • edited Loading

lidel commented Jan 23, 2019

kyledrake commented Feb 18, 2019

lidel commented Mar 3, 2019 • edited Loading

🏆 dweb.link is available over HTTPS!

ℹ️ Plan to support CID-in-subdomain via HTTP Proxy

ℹ️ WIP in js-ipfs: change defaults for adding content to cidv1b32

ℹ️ WIP in go-ipfs: switching DHT (& datastores) to raw multihashes

lidel commented May 10, 2019

PSA: Switching CIDv1 to Base32

lidel commented May 27, 2019

PSA: Adding support for CIDv1 in IPNS

lidel commented Jun 19, 2019

lidel commented Aug 23, 2019

hsanjuan commented Mar 27, 2020

lidel commented Apr 16, 2020 • edited Loading

📣 PSA: go-ipfs 0.5.0 will bring native support for PeerIDs as CIDv1 & CID-in-subdomain

HTTP Gateway

CLI / RPC API

lidel commented Dec 2, 2021 • edited Loading

lidel commented Apr 8, 2022 • edited Loading

Finally, we can do the thing! 🙀

kyledrake commented Jul 12, 2018 •

edited by lidel

Loading

kevina commented Aug 12, 2018 •

edited

Loading

lidel commented Nov 2, 2018 •

edited

Loading

kyledrake commented Jan 17, 2019 •

edited

Loading

lidel commented Mar 3, 2019 •

edited

Loading

🏆 `dweb.link` is available over HTTPS!

lidel commented Apr 16, 2020 •

edited

Loading

lidel commented Dec 2, 2021 •

edited

Loading

lidel commented Apr 8, 2022 •

edited

Loading