fix: replace node buffers with uint8arrays #117

achingbrain · 2020-07-30T09:42:42Z

Relaxes input from requiring node Buffers to being Uint8Arrays.

This also means that the .buffer (now .byte) and .prefix properties have changed to be Uint8Arrays.

Depends on:

BREAKING CHANGES:

node Buffers have been replaced with Uint8Arrays
the .buffer property has been renamed to .bytes and is now a Uint8Array
the .prefix property is now a Uint8Array

vmx

Thanks the changes look good!

Would you mind updating the README to not use Buffer, to indicate the Uint8Arrays are the way to go? I think using the global TextEncoder/Decoder is fine in a README example.

vmx · 2020-07-30T11:16:20Z

The CI failure is real, though you could also use https://github.com/Gozala/web-encoding/ instead of ipfs-utils.

achingbrain · 2020-07-30T11:27:00Z

you could also use Gozala/web-encoding instead of ipfs-utils.

I will if using ipfs-utils would prevent this from getting merged, but ipfs-utils already has the TextEncoder browser/node polyfill in it so it seems redundant.

Plus once ipfs/js-ipfs-utils#55 is merged I can remove some of the duplicated utility functions here.

Gozala

This is awesome! I just would like to use this opportunity to rename .buffer property:

So it better reflects what it containts
So that code expecting older CID breaks early rather than later.
So we could in the future make CIDs ArrayBufferViews in which case we'd need buffer property to point to the underlying ArrayBuffer instead.

Gozala · 2020-07-30T21:50:57Z

README.md

@@ -142,19 +158,19 @@ Property containing the CID version integer.

 #### cid.multihash

-Property containing the multihash buffer.
+Property containing the multihash `Uint8Array`.

 #### cid.multibaseName

 Property containing the default base to use when calling `.toString`

 #### cid.buffer


I think we should rename this property, so changes are more obvious and name isn't misleading.

It makes me wince a little, because it's going to be very disruptive, but I've renamed it to .bytes.

src/index.d.ts

src/index.js

test/index.spec.js

Gozala · 2020-07-30T22:08:58Z

I will if using ipfs-utils would prevent this from getting merged, but ipfs-utils already has the TextEncoder browser/node polyfill in it so it seems redundant.

Plus once ipfs/js-ipfs-utils#55 is merged I can remove some of the duplicated utility functions here.

For what it's worth we have discussed option of using ipfs-utils vs creating a web-encodings library and chose later to avoid pulling all the extra stuff that api-utils has. I'm fine with either option.

achingbrain · 2020-07-31T13:28:08Z

I've pulled out ipfs-utils in favour of uint8arrays which uses web-encoding internally.

Relaxes input from requiring node `Buffer`s to being `Uint8Arrays`. This also means that the `.buffer` (now `.byte`) and `.prefix` properties have changed to be `Uint8Array`s. BREAKING CHANGES: - node `Buffer`s have been replaced with `Uint8Array`s - the `.buffer` property has been renamed to `.bytes` and is now a `Uint8Array` - the `.prefix` property is now a `Uint8Array`

vmx · 2020-08-03T13:07:52Z

@mikeal @rvagg I'd really like to have your approval on that change. Once merged I want to release this as 1.0.0.

mikeal · 2020-08-03T16:58:36Z

So, +1 in general, great to see this happening.

One concern I have taking a breaking change is just wasting the opportunity to do a bit more. Knowing that we have a more substantial CID change coming in the future it would be great to get some code into this change that will make that transition a bit easier.

For instance, we could add the .code property to CID and we could allow for instantiation using integers for the codec rather than just strings. This would allow people to migrate to using these newer forms even with the old API and should make the future transition to js-multiformats a bit easier.

Basically, we know that we need to move to using integers rather than strings for codec identifiers, it would be nice if we had a transitional release that accepted both so people could upgrade gradually or at least have the opportunity to do some of that migration when they migrate off of Buffer.

Depends on: - [ ] multiformats/js-cid#117 BREAKING CHANGE: - `util.serialize` now returns a Uint8Array

achingbrain · 2020-08-03T17:28:52Z

One concern I have taking a breaking change is just wasting the opportunity to do a bit more

I can see the temptation but I really think to deliver this change in a reasonable timeframe we should limit the scope to what's currently in this PR.

Ref: #117 (comment)

rvagg

I'm +1 on this but also +1 on Mikeal's ask, so see #118 for that piece of the puzzle which would be nice to slot in prior to a release of this if we can get agreement on that addition.

Ref: #117 (comment)

This module now accepts Uint8Arrays as well as node Buffers and returns Uint8Arrays. Internally it converts non-Buffers into Buffers because the ethereum libs require that. BREAKING CHANGES: - `util.serialize` returns a `Uint8Array` - `util.cid` returns `CID`s with a breaking API change - see multiformats/js-cid#117 for changes

returns Uint8Arrays. Internally it converts non-Buffers into Buffers because the ethereum libs require that. BREAKING CHANGES: - `util.serialize` returns a `Uint8Array` - `util.cid` returns `CID`s with a breaking API change - see multiformats/js-cid#117 for changes

This module now accepts Uint8Arrays as well as node Buffers and returns Uint8Arrays. Internally it converts non-Buffers into Buffers because the ethereum libs require that. BREAKING CHANGES: - `util.serialize` returns a `Uint8Array` - `util.cid` returns `CID`s with a breaking API change - see multiformats/js-cid#117 for changes

BREAKING CHANGES: - `util.cid` returns `CID`s with a breaking API change - see multiformats/js-cid#117 for changes

Removes use of node Buffer in favour of Uint8Arrays and updates all dependencies to versions that do the same. BREAKING CHANGES: - CIDs returned from `put*` methods have breaking API changes See see multiformats/js-cid#117 for changes

This module now accepts Uint8Arrays as well as node Buffers and returns Uint8Arrays. Internally it converts non-Buffers into Buffers because the ethereum libs require that. BREAKING CHANGES: - `util.serialize` returns a `Uint8Array` - `util.cid` returns `CID`s with a breaking API change - see multiformats/js-cid#117 for changes

BREAKING CHANGES: - `util.cid` returns `CID`s with a breaking API change - see multiformats/js-cid#117 for changes

Accepts and emits `Uint8Array`s instead of node `Buffer`s but converts them internally because `bitcoinjs-lib` only understands `Buffer`s. BREAKING CHANGE: - `util.serialize` returns `Uint8Array`s - `util.cid` returns `CID`s with a breaking API change - see multiformats/js-cid#117 for changes

This module now accepts Uint8Arrays as well as node Buffers and returns Uint8Arrays. Internally it converts non-Buffers into Buffers because the ethereum libs require that. BREAKING CHANGES: - `util.serialize` returns a `Uint8Array` - `util.cid` returns `CID`s with a breaking API change - see multiformats/js-cid#117 for changes

Gozala · 2020-08-04T16:53:53Z

For what it's worth I would also like to slot in asCID into 1.0. That said, I do not think we should block this pull request on other things we want in 1.0. We should instead consider having a meeting to make some decision about how to best roll out all the breaking changes.

Removes use of node Buffer in favour of Uint8Arrays and updates all dependencies to versions that do the same. BREAKING CHANGES: - CIDs returned from `put*` methods have breaking API changes See see multiformats/js-cid#117 for changes

Follow up to #66 that removes node Buffers from the tests and uses version of protons that returns uint8arrays. BREAKING CHANGES: - All use of node Buffers have been replaced with Uint8Arrays - CIDs exported by this module have breaking API changes, see: multiformats/js-cid#117

achingbrain requested review from Gozala, vmx and hugomrdias July 30, 2020 09:42

vmx reviewed Jul 30, 2020

View reviewed changes

This was referenced Jul 30, 2020

fix: replace node Buffers with Uint8Arrays ipld/js-ipld-dag-pb#187

Merged

fix: replace node buffers with uint8arrays ipfs/js-ipfs-repo-migrations#25

Merged

fix: swap node buffers for uint8arrays ipfs/js-ipfs-repo#249

Merged

Gozala suggested changes Jul 30, 2020

View reviewed changes

achingbrain requested review from vmx and Gozala July 31, 2020 13:28

vmx approved these changes Jul 31, 2020

View reviewed changes

vmx requested review from rvagg and mikeal July 31, 2020 14:01

Gozala approved these changes Jul 31, 2020

View reviewed changes

achingbrain changed the title ~~fix: support uint8arrays~~ fix: replace node buffers with uint8arrays Jul 31, 2020

achingbrain force-pushed the fix/support-uint8arrays branch 2 times, most recently from af9dea1 to 5c00d18 Compare August 3, 2020 12:18

achingbrain force-pushed the fix/support-uint8arrays branch from 5c00d18 to cb0eba1 Compare August 3, 2020 12:55

vmx approved these changes Aug 3, 2020

View reviewed changes

chore: handle uint8array multihashes

cecfb53

achingbrain mentioned this pull request Aug 3, 2020

fix: replace node buffers with uint8arrays ipld/js-ipld-dag-cbor#134

Merged

1 task

achingbrain added a commit to ipld/js-ipld-raw that referenced this pull request Aug 3, 2020

fix: replace node buffers with uint8arrays

aba179c

Depends on: - [ ] multiformats/js-cid#117 BREAKING CHANGE: - `util.serialize` now returns a Uint8Array

achingbrain mentioned this pull request Aug 3, 2020

fix: replace node buffers with uint8arrays ipld/js-ipld-raw#49

Merged

1 task

Gozala mentioned this pull request Aug 3, 2020

Array buffer view multiformats/js-multiformats#29

Merged

rvagg added a commit that referenced this pull request Aug 4, 2020

feat: expose codec code and allow construction by code

c043bc5

Ref: #117 (comment)

rvagg mentioned this pull request Aug 4, 2020

feat: expose codec code and allow construction by code #118

Merged

rvagg approved these changes Aug 4, 2020

View reviewed changes

vmx merged commit a7ae250 into master Aug 4, 2020

vmx deleted the fix/support-uint8arrays branch August 4, 2020 11:49

rvagg added a commit that referenced this pull request Aug 4, 2020

feat: expose codec code and allow construction by code

845b2df

Ref: #117 (comment)

vmx pushed a commit that referenced this pull request Aug 4, 2020

feat: expose codec code and allow construction by code

59469b6

Ref: #117 (comment)

achingbrain mentioned this pull request Aug 4, 2020

fix: replace node buffers with uint8arrays ipld/js-ipld-ethereum#69

Merged

achingbrain mentioned this pull request Aug 4, 2020

fix: replace node buffers with uint8arrays ipld/js-ipld-git#70

Merged

achingbrain added a commit to ipld/js-ipld-zcash that referenced this pull request Aug 4, 2020

fix: update to latest cids and uint8array compatible deps

d86f0bb

BREAKING CHANGES: - `util.cid` returns `CID`s with a breaking API change - see multiformats/js-cid#117 for changes

achingbrain mentioned this pull request Aug 4, 2020

fix: update to latest cids and uint8array compatible deps ipld/js-ipld-zcash#60

Merged

achingbrain mentioned this pull request Aug 4, 2020

fix: update all deps to versions that return uint8arrays ipld/js-ipld#286

Merged

7 tasks

vmx pushed a commit to ipld/js-ipld-zcash that referenced this pull request Aug 4, 2020

fix: update to latest cids and uint8array compatible deps

c613e57

BREAKING CHANGES: - `util.cid` returns `CID`s with a breaking API change - see multiformats/js-cid#117 for changes

achingbrain mentioned this pull request Aug 10, 2020

Replace node Buffers with Uint8Arrays ipfs/js-ipfs#3220

Closed

64 tasks

Gozala mentioned this pull request Sep 9, 2020

TypeError: Cannot read property 'length' of undefined orbitdb/orbitdb#839

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: replace node buffers with uint8arrays #117

fix: replace node buffers with uint8arrays #117

achingbrain commented Jul 30, 2020 •

edited

Loading

vmx left a comment

vmx commented Jul 30, 2020

achingbrain commented Jul 30, 2020 •

edited

Loading

Gozala left a comment

Gozala Jul 30, 2020

achingbrain Jul 31, 2020

Gozala commented Jul 30, 2020

achingbrain commented Jul 31, 2020

vmx commented Aug 3, 2020

mikeal commented Aug 3, 2020

achingbrain commented Aug 3, 2020

rvagg left a comment

Gozala commented Aug 4, 2020

fix: replace node buffers with uint8arrays #117

fix: replace node buffers with uint8arrays #117

Conversation

achingbrain commented Jul 30, 2020 • edited Loading

vmx left a comment

Choose a reason for hiding this comment

vmx commented Jul 30, 2020

achingbrain commented Jul 30, 2020 • edited Loading

Gozala left a comment

Choose a reason for hiding this comment

Gozala Jul 30, 2020

Choose a reason for hiding this comment

achingbrain Jul 31, 2020

Choose a reason for hiding this comment

Gozala commented Jul 30, 2020

achingbrain commented Jul 31, 2020

vmx commented Aug 3, 2020

mikeal commented Aug 3, 2020

achingbrain commented Aug 3, 2020

rvagg left a comment

Choose a reason for hiding this comment

Gozala commented Aug 4, 2020

achingbrain commented Jul 30, 2020 •

edited

Loading

achingbrain commented Jul 30, 2020 •

edited

Loading