feat: compatibility with go-libp2p-mdns #80

alanshaw · 2019-04-08T15:14:54Z

This PR adds a compatibility class that allows a js-libp2p node to find a go-libp2p node (and vice versa) over MDNS. I do not know when go-libp2p plans to land the long awaited new MDNS implementation but until then this will allow interop between js and go.

It's implemented as a separate class so the two differing implementations do not get confused.

Fun fact, it uses 2 dgram servers. 1 is the multicast listener, which simply responds to queries. The other is a dgram server that exists for 5 seconds at a time, sends a multicast query, waits for unicast responses, stops and then starts up again on a different random port.

I've verified this is working correctly by running a go-ipfs and js-ipfs node with no boostrap nodes (and no other discovery methods) and verifying they find each other.

TODO:

Add tests!

Some tips if you want to try this out:

After you've run ipfs init, remember to remove the bootstrap nodes from the config file (~/.ipfs/config) of each node before you start up
Use ipfs log level mdns debug for some go-ipfs mdns logs
You can use the following script (after npm linking this branch) to start a js-ipfs node with no bootstrap nodes and no discovery modules other than MDNS:

const IPFS = require('ipfs')
const MDNS = require('libp2p-mdns')
const TCP = require('libp2p-tcp')

const ipfs = new IPFS({
  repo: '/tmp/ipfs-mdns',
  config: {
    Bootstrap: []
  },
  libp2p: {
    modules: {
      peerDiscovery: [MDNS],
      transport: [TCP]
    }
  }
})

ipfs.on('ready', async () => {
  console.log('ipfs is ready')
  console.log('My Peer ID:', (await ipfs.id()).id)
  setInterval(async () => {
    const peers = await ipfs.swarm.peers()
    console.log(peers.length, 'peers:')
    peers.forEach(p => console.log(p.peer.toB58String()))
  }, 10000)
})

dirkmc

I left a couple of comments about event listeners, otherwise looks good 👍

Would it make sense to branch of the async/await version of this package?

dirkmc · 2019-04-08T21:35:37Z

src/compat/querier.js

+      // Create a querier that queries multicast but gets responses unicast
+      const querier = MDNS({ multicast: false, interface: '0.0.0.0', port: 0 })
+
+      querier.on('response', this._onResponse)


I think having this event listener will prevent the querier objects from being garbage collected
(looking at the source code for mdns, it doesn't seem like destroy() removes all event listeners)

dirkmc · 2019-04-08T21:47:41Z

src/compat/responder.js

+  }
+
+  stop (callback) {
+    this._mdns.destroy(callback)


this._mdns.removeListener('query', this._onQuery)

vasco-santos

Great work Alan! 🥇

I think that it is a good call to have this in a separate class!

Regarding @dirkmc 's point for branching out from the async await branch, according to the table in ipfs/js-ipfs#1670 it is a P3+, and should have a lot of dependencies to get merged first. However, the async migration PR will have to be updated with this. Looking at the code, it does not seem a lot of work. What we can also do is changing this PR's codebase (compat) to use async await from the beginning, which would ease the other PR to get merged afterward. What do you guys think?

Also, do not forget to provide documentation for the new classes added.

vasco-santos · 2019-04-09T09:49:35Z

src/compat/querier.js

+      return { stop: callback => querier.destroy(callback) }
+    }, this._options.queryPeriod)
+
+    setImmediate(() => callback())


Use process.nextTick instead

vasco-santos · 2019-04-09T09:52:56Z

src/index.js

+    if (this._goMdns) {
+      this._goMdns.start(callback)
+    } else {
+      setImmediate(() => callback())


process.nextTick

vasco-santos · 2019-04-09T09:55:21Z

src/compat/responder.js

+  start (callback) {
+    this._mdns = MDNS()
+    this._mdns.on('query', this._onQuery)
+    setImmediate(() => callback())


process.nextTick

src/compat/responder.js

vasco-santos · 2019-04-09T09:56:31Z

src/compat/responder.js

+      data: peerServiceTagLocal
+    })
+
+    // Only announce TCP multiaddrs for now


Can you have as an action item of merging this PR to create an issue for tracking the announce of other transports?

alanshaw · 2019-04-09T10:19:29Z

What we can also do is changing this PR's codebase (compat) to use async await from the beginning, which would ease the other PR to get merged afterward. What do you guys think?

I don't think we should mix the two styles in the code base:

IMHO we should only mix promises and callbacks when we have to deal with an external library we have no control over. We already have async as a dependency so we'd not be saving any bundle bytes by using builtins. I think it sends the wrong message to contributors. We have a awesome endeavour where we're trying to coordinate all of this at the same time and adding Promises here piecemeal is problematic for a number of reasons:

It's super inconsistent with the rest of the libp2p code (everything is callbacks, not even a dual promise/callback api)

Sets the precedent for other contributors to do the same thing, when actually we want contributions to either focus on fixing problems or focus on the endeavour

Finally, without going into specifics, mixing callbacks and Promises is fraught with double callback danger, perhaps not in this specific case, but it can be super hard to debug when it happens!

from libp2p/js-libp2p-switch#310 (comment)

So either we merge as is or rebase off of the async/await branch.

jacobheun · 2019-04-09T10:33:16Z

My main concern here relates to #71, in regards to polling mdns. Currently this is configured to run the go query every 5 seconds by default. The normal query is set to run every 10s by default. Running 3 queries every 10 seconds is abusive to mdns and devices using it.

I think this might be a good opportunity for us to look at disabling polling by default, and allowing an interval to be set, instead of defaulting to polling. As new devices join, we should discover them when they broadcast.

alanshaw · 2019-04-09T14:01:30Z

My main concern here relates to #71, in regards to polling mdns. Currently this is configured to run the go query every 5 seconds by default. The normal query is set to run every 10s by default. Running 3 queries every 10 seconds is abusive to mdns and devices using it.

Ok, there are some options:

We replace the current implementation with the go compatible implementation
- (+) Smaller code base, compatible with go-ipfs
- (-) We poll every 5 seconds (but we could just increase that to 10s so we're no worse than we are already)
- (-) We break MDNS discovery of older js-ipfs nodes running on the network (although this is already broken for other reasons https://github.com/libp2p/js-libp2p-switch/issues/326)
We turn off polling (but not responding) for Go MDNS compat
- (+) Not introducing more MDNS polling
- (+) go-ipfs nodes still poll so we'll send responses and go-ipfs nodes will probably connect to us (i.e. go-ipfs can discover js-ipfs but not the other way round)
- (-) If go-ipfs nodes stop polling they'll never discover js-ipfs nodes

I think this might be a good opportunity for us to look at disabling polling by default, and allowing an interval to be set, instead of defaulting to polling. As new devices join, we should discover them when they broadcast.

Simply sending a multicast go-ipfs compatible "response" on an interval might work because I think go-ipfs accepts responses multicast as well as unicast. It's similar to (2) above, except that go-ipfs nodes will discover us when we send a response in an interval, not as a response to their query.

I'll check it out and report back.

alanshaw · 2019-04-09T14:30:05Z

It works! #81

jacobheun · 2019-04-09T16:30:27Z

So, #81 doesn't do a query at all which is why we won't be able to get go nodes. It should still do a query, it just shouldn't need to be run on an interval.

Anytime a node starts, it should listen for MDNS queries and then immediately query.
That same query will be received by the started node, which will respond to it. So we've discovered any existing peers, and told them about ourselves.

If we start another node 10 minutes later and it does the same thing. The first node should be made aware of the new node, and vice versa.

For every node we have, we should only need that many queries, as long as everyone responds. There is the problem of sleeping nodes or network changes, but since our discovery doesn't work all that consistently right now, I think that could be addressed in a followup release or temporarily just increasing the interval.

alanshaw · 2019-04-10T15:12:43Z

@jacobheun I've altered this PR to increase the interval between MDNS queries to 1 min.

When GoMulticastDNS starts it creates 2 MDNS servers:

A Responder which listens for multicast queries and responds DIRECTLY to them i.e. a unicast response
A Querier which listens on a RANDOM port for UNICAST responses. It immediately sends a multicast query and waits for 5s for responses. After 5s have elapsed it shuts down and repeats after 1 minute

This follows the implementation of go-libp2p-mdns except in go the interval is 0 seconds not 1 minute.

The JS implementation is untouched.

Can you please confirm that's cool and I'll get on with adding tests.

jacobheun · 2019-04-10T15:37:08Z

@alanshaw that sounds reasonable to me.

I'll look at trying to get libp2p/specs#80 at least moved into Draft so those proper implementations can get unblocked. I believe rust-libp2p is already using that spec.

This PR adds a compatibility class that allows a js-libp2p node to find a go-libp2p node (and vice versa) over MDNS. It's implemented as a separate class so the two differing implementations do not get confused. I've verified this is working correctly by running a go-ipfs and js-ipfs node with no boostrap nodes (and no other discovery methods) and verifying they find each other. TODO: * [ ] Add tests! Some tips if you want to try this out: * After you've run `ipfs init`, remember to remove the bootstrap nodes from the config file (`~/.ipfs/config`) of each node before you start up * Use `ipfs log level mdns debug` for some go-ipfs mdns logs * You can use the following script (after `npm link`ing this branch) to start a js-ipfs node with no bootstrap nodes and no discovery modules other than MDNS: ```js const IPFS = require('ipfs') const MDNS = require('libp2p-mdns') const TCP = require('libp2p-tcp') const ipfs = new IPFS({ repo: '/tmp/ipfs-mdns', config: { Bootstrap: [] }, libp2p: { modules: { peerDiscovery: [MDNS], transport: [TCP] } } }) ipfs.on('ready', async () => { console.log('ipfs is ready') console.log('My Peer ID:', (await ipfs.id()).id) setInterval(async () => { const peers = await ipfs.swarm.peers() console.log(peers.length, 'peers:') peers.forEach(p => console.log(p.peer.toB58String())) }, 10000) }) ``` License: MIT Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>

License: MIT Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>

Co-Authored-By: alanshaw <alan.shaw@protocol.ai>

License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io>

License: MIT Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>

License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io>

License: MIT Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>

jacobheun

Other than adding the compat option to the readme, this looks good.

jacobheun · 2019-05-02T11:06:03Z

src/index.js

@@ -18,10 +21,18 @@ class MulticastDNS extends EventEmitter {
    this.port = options.port || 5353
    this.peerInfo = options.peerInfo
    this._queryInterval = null
+    this._onPeer = this._onPeer.bind(this)
+
+    if (options.compat !== false) {


Needs to be added to the readme.

License: MIT Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>

vasco-santos

LGTM!

jacobheun

Looks good. Just FYI, the mdns spec was merged https://github.com/libp2p/specs/blob/master/discovery/mdns.md. We can work with the go team to try and coordinate some future releases to the new version.

ghost assigned alanshaw Apr 8, 2019

ghost added the status/in-progress In progress label Apr 8, 2019

alanshaw requested review from jacobheun, vasco-santos, dirkmc and achingbrain April 8, 2019 15:18

dirkmc suggested changes Apr 8, 2019

View reviewed changes

vasco-santos suggested changes Apr 9, 2019

View reviewed changes

alanshaw mentioned this pull request Apr 11, 2019

Cannot retrieve content via the DHT ipfs/js-ipfs#1984

Closed

alanshaw marked this pull request as ready for review April 15, 2019 15:34

alanshaw and others added 13 commits April 15, 2019 16:37

chore: appease linter

0320312

License: MIT Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>

fix: move async to dependencies

62562b8

License: MIT Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>

fix: typo in comment

2483aba

Co-Authored-By: alanshaw <alan.shaw@protocol.ai>

refactor: pr feedback

655b4d7

License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io>

fix: respond directly to querier

7f6ac76

License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io>

fix: reemit the peer event from GoMulticastDNS

324bbf7

License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io>

refactor: add interval between queries

05ec9f0

License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io>

chore: appease linter

31369eb

License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io>

fix: existing tests

3ff7250

License: MIT Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>

fix: option name

0b39814

License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io>

fix: use async/nextTick

2b01368

License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io>

test: add tests for querier and main compat class

3aca133

License: MIT Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>

alanshaw added 2 commits April 15, 2019 16:38

chore: remove .only

15416c3

License: MIT Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>

test: add responder tests

37fbfdd

License: MIT Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>

alanshaw force-pushed the feat/go-libp2p-mdns-compat branch from dc964c1 to 37fbfdd Compare April 15, 2019 15:38

fix: increase timeout for query on interval test

a3683e7

License: MIT Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>

alanshaw requested a review from vasco-santos April 16, 2019 13:44

lidel mentioned this pull request Apr 22, 2019

Embedded JS-IPFS in Brave (experiment) ipfs/ipfs-companion#716

Closed

52 tasks

jacobheun suggested changes May 2, 2019

View reviewed changes

lidel mentioned this pull request May 8, 2019

In-browser discovery of local-network IPFS nodes ipfs/in-web-browsers#45

Open

docs: document new compat option

f8f9830

License: MIT Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>

alanshaw requested a review from jacobheun May 9, 2019 08:12

alanshaw mentioned this pull request May 9, 2019

⚡️ v0.36.0 RELEASE 🚀 ipfs/js-ipfs#2024

Closed

44 tasks

vasco-santos approved these changes May 9, 2019

View reviewed changes

jacobheun approved these changes May 9, 2019

View reviewed changes

jacobheun merged commit c6d1d49 into master May 9, 2019

jacobheun deleted the feat/go-libp2p-mdns-compat branch May 9, 2019 10:10

ghost removed the status/in-progress In progress label May 9, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: compatibility with go-libp2p-mdns #80

feat: compatibility with go-libp2p-mdns #80

alanshaw commented Apr 8, 2019 •

edited

Loading

dirkmc left a comment

dirkmc Apr 8, 2019

dirkmc Apr 8, 2019

vasco-santos left a comment

vasco-santos Apr 9, 2019

vasco-santos Apr 9, 2019

vasco-santos Apr 9, 2019

vasco-santos Apr 9, 2019

alanshaw commented Apr 9, 2019

jacobheun commented Apr 9, 2019

alanshaw commented Apr 9, 2019

alanshaw commented Apr 9, 2019

jacobheun commented Apr 9, 2019

alanshaw commented Apr 10, 2019

jacobheun commented Apr 10, 2019

jacobheun left a comment

jacobheun May 2, 2019

vasco-santos left a comment

jacobheun left a comment

feat: compatibility with go-libp2p-mdns #80

feat: compatibility with go-libp2p-mdns #80

Conversation

alanshaw commented Apr 8, 2019 • edited Loading

dirkmc left a comment

Choose a reason for hiding this comment

dirkmc Apr 8, 2019

Choose a reason for hiding this comment

dirkmc Apr 8, 2019

Choose a reason for hiding this comment

vasco-santos left a comment

Choose a reason for hiding this comment

vasco-santos Apr 9, 2019

Choose a reason for hiding this comment

vasco-santos Apr 9, 2019

Choose a reason for hiding this comment

vasco-santos Apr 9, 2019

Choose a reason for hiding this comment

vasco-santos Apr 9, 2019

Choose a reason for hiding this comment

alanshaw commented Apr 9, 2019

jacobheun commented Apr 9, 2019

alanshaw commented Apr 9, 2019

alanshaw commented Apr 9, 2019

jacobheun commented Apr 9, 2019

alanshaw commented Apr 10, 2019

jacobheun commented Apr 10, 2019

jacobheun left a comment

Choose a reason for hiding this comment

jacobheun May 2, 2019

Choose a reason for hiding this comment

vasco-santos left a comment

Choose a reason for hiding this comment

jacobheun left a comment

Choose a reason for hiding this comment

alanshaw commented Apr 8, 2019 •

edited

Loading