feat: mfs implementation #1360

achingbrain · 2018-05-18T19:04:26Z

~~Not ready for merging yet~~ but I'd love some feedback on the approach. I've placed the mfs commands behind a flag named --enable-mfs-experiment - if you enable this it will cause the ipfs files subcommands to be the same as the go implementation.

Mr0grog · 2018-05-18T19:50:14Z

Minor note — would you please document the new EXPERIMENTAL flag along with the others in the README?

alanshaw

You're right, we should really be reading config from an existing config file and overriding with --enable-XXX-experiment or env vars when set.

I'm ok with loading MFS functions up like this from a separate module. Was your plan to leave it like this or eventually merge the ipfs-mfs repo with this one when it's all done?

alanshaw · 2018-05-29T12:04:46Z

src/cli/bin.js

+
+            return commandObject
+          }
+        })


Would you mind explaining what's going on here?

It's preventing any of the existing ipfs files <command> commands from loading if enableMfs is truthy.

alanshaw · 2018-05-29T12:06:29Z

test/cli/file.js


 describe('file ls', () => runOnAndOff((thing) => {
  let ipfs

  before(function () {
    this.timeout(50 * 1000)
    ipfs = thing.ipfs
-    return ipfs('files add -r test/fixtures/test-data/recursive-get-dir')
+    return ipfs('add -r test/fixtures/test-data/recursive-get-dir')


So is this suggesting that add, get and cat would no longer be available under the files namespace?

Yes, this is consistent with go-ipfs

Let's think more about this...

I would be totally down to move forward with ipfs/specs#98 in JS land (both js-ipfs and js-ipfs-api). In fact, I believe we are reaching a time where we need to start pushing forward better and polished APIs, given that we live in a different ecosystem and we need to make things that work for our user needs.

An advantage to ourselves is that we have a SPEC and tests, making this transition smooth and more pleasant.

achingbrain · 2018-05-29T13:52:27Z

Was your plan to leave it like this or eventually merge the ipfs-mfs repo with this one

@diasdavid's preference was for a separate module so there are no plans to merge it into this one AFAIK.

daviddias · 2018-06-01T11:04:08Z

@achingbrain can we drop the experimental flag? This should work with go-ipfs right off the bat.

achingbrain · 2018-06-01T12:09:10Z

Sure, that’d certainly make things a bit more straightforward.

…

On 1 Jun 2018, at 13:04, David Dias ***@***.***> wrote: @achingbrain can we drop the experimental flag? This should work with go-ipfs right off the bat. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

We are going all-in with mfs and not having it behind a flag: ipfs/js-ipfs#1360 (comment)

lidel · 2018-07-03T18:39:11Z

May be too specific, but is there an interop test that checks if files.write produces the same CIDs in js-ipfs and go-ipfs?
We will be changing default CID encoding at some point and such tests would help us coordinate.

achingbrain · 2018-07-04T13:35:16Z

@diasdavid @lidel the interop tests are here: ipfs/interop#25 and there are a whole bunch that assert on CIDs generated by go & js.

alanshaw

This is incredible, thanks for your patients @achingbrain

alanshaw · 2018-07-04T14:10:14Z

src/cli/bin.js

+          } else if (arg.error && arg.error.message) {
+            print(arg.error.message)
+          } else {
+            print('Unknown error, please re-run the command with DEBUG=ipfs:cli to see debug output')


It would be great to have some consistency here - the rest of the cli debug prefixes are cli:* and elsewhere in the code we have jsipfs:*. One for a separate PR 😉

I'd love it if we could standardise on one prefix for all ipfs related DEBUG values. Just doing ipfs:* instead of ipfs:*,ipfsd-ctl:*,ipfs-api:*,etc is my dream.

alanshaw · 2018-07-04T14:12:46Z

src/cli/bin.js

+            print('Unknown error, please re-run the command with DEBUG=ipfs:cli to see debug output')
+          }
+
+          process.exit(1)


We always called cleanup before even if there was an error. I think this needs to be added here too.

We should probably register a process exit/error handler to do the cleanup instead, but we'd need a synchronous way of closing the repo which I don't think exists yet.

alanshaw · 2018-07-04T14:18:56Z

src/cli/commands/dag/get.js

@@ -41,7 +41,7 @@ module.exports = {
      // * reads as 'agree in'
      if (node._json) {
        delete node._json.multihash
-        node._json.data = '0x' + node._json.data.toString('hex')
+        node._json.data = node._json.data.toString('base64')


Is this change meant to be in this PR? Why not also for the if test below?

Probably could be separate. go prints out the data as base64 encoded, we print it out as hex.

alanshaw · 2018-07-04T14:21:36Z

src/core/index.js

+    const mfs = components.mfs(this)
+
+    Object.keys(mfs).forEach(key => {
+      if (mfs.hasOwnProperty(key)) {


Minor: I believe Object.keys iterates over own properties so I think this check is redundant, unless I'm missing something?

From my understanding Object.keys returns all properties, including inherited ones whereas hasOwnProperty will only return true if it's a non-inherited property. Happy to change this if you prefer it the other way.

alanshaw · 2018-07-04T14:42:58Z

test/browser.js

+if (isWebWorker) {
+  // https://github.com/Joris-van-der-Wel/karma-mocha-webworker/issues/4
+  global.MFS_DISABLE_CONCURRENCY = true
+}


Since there is a requirement to either set this variable or do some setup on the main thread (for webworkers) we should mention it and link to it from the js-ipfs README so that it's easily discoverable and we don't get issues opened about it here.

Also a separate PR to add a examples for using MFS in webworkers with/without concurrency is needed and would be very 🚀 🎸 ✨ 💖

I've added a paragraph to the README, will create an example in a separate PR.

What does MFS with concurrency mean?

nvm. Read the paragraph.

alanshaw · 2018-07-04T14:45:23Z

test/core/interface.spec.js


  tests.key(CommonFactory.create({
    spawnOptions: {
      args: ['--pass ipfs-is-awesome-software'],
-      initOptions: { bits: 512 }
+      initOptions: { bits: 1024 }


This is for faster boot - defaultCommonFactory uses 512 also, is this change needed?

I ran the tests against the go daemon and it refused to start up with 512 and said it should be 1024 as a minimum. Can change it back if you prefer.

Yep in js-ipfs-api it's set to 1024 it tests against go-ipfs, but these tests only run against js-ipfs and there's no restriction in place currently.

There probably should be, for the sake of interop though, right?

Yes good point, lets not slow down the tests unnecessarily until that PR comes in though. I'll create an issue for adding it so we can track it.

daviddias · 2018-07-05T09:49:40Z

README.md

@@ -135,6 +136,20 @@ You can also load it using a `<script>` using the [unpkg](https://unpkg.com) CDN
 ```
 Inserting one of the above lines will make an `Ipfs` object available in the global namespace.

+### Use with Web Workers
+
+The `ipfs.files.*` [MFS](https://github.com/ipfs/interface-ipfs-core/blob/master/SPEC/FILES.md#mutable-file-system) commands use LevelDB to store the CID that corresponds to the current root of the file system.  LevelDB has no concurrency control so a read/write lock at the application level is necessary.  With single-process and [Node.js cluster](https://nodejs.org/api/cluster.html) based apps this is completely transparent but [Web Workers](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API) require a little extra setup.  Use the [`observable-webworkers`](https://www.npmjs.com/package/observable-webworkers) module to allow communication with any Web Workers your application creates:


I see. This is so that if multiple WebWorkers are spawn, the store is shared?

Note that this is valid for anything that IPFS stores really, in the browser it is IndexedDB for everything.

Yes, if multiple threads mutate the MFS graph at the same time they'll result in different hashes for the root of the graph creating a last-update-wins race condition. There needs to be some sort of gatekeeper to resolve this.

Multiple threads, each with its IPFS Node or just one IPFS node?

I've flipped this round so that you only need to do the config dance if you've spawned a node on the main process then forked (e.g. node clusters) or spawned nodes on the UI thread and web workers (e.g. browsers) all using the same repo - which hopefully shouldn't be very common.

I'm not sure we should encourage this in the browser at least so I've left it undocumented for the time being but we can discuss this during the locking session at the dev meetup next week.

License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io>

achingbrain · 2018-07-05T21:44:42Z

ghost assigned achingbrain May 18, 2018

ghost added the status/in-progress In progress label May 18, 2018

alanshaw changed the title ~~feat: mfs implementation~~ [WIP] feat: mfs implementation May 21, 2018

achingbrain force-pushed the add-mfs branch 2 times, most recently from a0fd5b9 to 259aa82 Compare May 26, 2018 07:07

alanshaw reviewed May 29, 2018

View reviewed changes

achingbrain added a commit to ipfs/js-ipfsd-ctl that referenced this pull request Jun 4, 2018

chore: remove experimental mfs flag

75fc836

We are going all-in with mfs and not having it behind a flag: ipfs/js-ipfs#1360 (comment)

achingbrain mentioned this pull request Jun 4, 2018

chore: remove experimental mfs flag ipfs/js-ipfsd-ctl#265

Merged

vasco-santos pushed a commit to ipfs/js-ipfsd-ctl that referenced this pull request Jun 6, 2018

chore: remove experimental mfs flag (#265)

9dcd5b2

We are going all-in with mfs and not having it behind a flag: ipfs/js-ipfs#1360 (comment)

achingbrain force-pushed the add-mfs branch 11 times, most recently from 2e73d97 to 4fed9c0 Compare June 14, 2018 12:48

Mr0grog mentioned this pull request Jun 18, 2018

Using IPFS with electron (or muon) ipfs/notes#256

Open

achingbrain force-pushed the add-mfs branch 4 times, most recently from 8c6a548 to 4406bb7 Compare June 20, 2018 09:18

achingbrain force-pushed the add-mfs branch from 4406bb7 to 34943e4 Compare June 27, 2018 16:38

achingbrain force-pushed the add-mfs branch 4 times, most recently from aec7c8a to a1d2a01 Compare July 3, 2018 14:11

achingbrain force-pushed the add-mfs branch 2 times, most recently from 94228dd to bd4d34b Compare July 4, 2018 13:29

achingbrain mentioned this pull request Jul 4, 2018

test: add mfs interop tests ipfs/interop#25

Merged

alanshaw suggested changes Jul 4, 2018

View reviewed changes

alanshaw mentioned this pull request Jul 5, 2018

⚡️ v0.30.0 RELEASE 🚀 #1375

Closed

23 tasks

achingbrain force-pushed the add-mfs branch from bd4d34b to 971a0d3 Compare July 5, 2018 09:34

daviddias reviewed Jul 5, 2018

View reviewed changes

achingbrain force-pushed the add-mfs branch 4 times, most recently from 2f11def to 3933a3a Compare July 5, 2018 11:39

daviddias mentioned this pull request Jul 5, 2018

locking between 2 browser tabs #1039

Closed

achingbrain force-pushed the add-mfs branch 2 times, most recently from e9bd96e to 95f99eb Compare July 5, 2018 14:54

feat: add mfs implementation

89cb6ea

achingbrain force-pushed the add-mfs branch from 95f99eb to 89cb6ea Compare July 5, 2018 15:40

fix: increase timeout for unadulterated data test (#1425)

9e75541

License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io>

ghost assigned alanshaw Jul 5, 2018

alanshaw approved these changes Jul 5, 2018

View reviewed changes

alanshaw merged commit 871d24e into master Jul 5, 2018

ghost removed the status/in-progress In progress label Jul 5, 2018

alanshaw deleted the add-mfs branch July 5, 2018 21:43

daviddias mentioned this pull request Mar 10, 2020

Files API - How can we make it better for users #2883

Closed

feat: mfs implementation #1360

feat: mfs implementation #1360

Conversation

achingbrain commented May 18, 2018 • edited Loading

Mr0grog commented May 18, 2018

alanshaw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

achingbrain May 29, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

achingbrain commented May 29, 2018

daviddias commented Jun 1, 2018

achingbrain commented Jun 1, 2018 via email

lidel commented Jul 3, 2018

achingbrain commented Jul 4, 2018

alanshaw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

achingbrain commented Jul 5, 2018

achingbrain commented May 18, 2018 •

edited

Loading

achingbrain May 29, 2018 •

edited

Loading