Skip to content
This repository has been archived by the owner on Feb 8, 2023. It is now read-only.

OKRs - 2019 Q1 JS Core WG #799

Merged
merged 6 commits into from
Feb 5, 2019
Merged

OKRs - 2019 Q1 JS Core WG #799

merged 6 commits into from
Feb 5, 2019

Conversation

alanshaw
Copy link
Member

@alanshaw alanshaw commented Dec 11, 2018

refs #792

cc @hugomrdias @achingbrain @daviddias

refs #792

- [ ] [Score 2018 Q4 OKRs](https://docs.google.com/spreadsheets/d/139lROP7-Ee4M4S7A_IO4iIgSgugYm7dct620LYnalII/edit#gid=274358435)
- [ ] [Async Retrospective](https://docs.google.com/document/d/1z6_eTW_8pI5VUj5UMq4K_Gpofofd7ENwzLdEBCuNULI/edit)
- [ ] 2019 Q1 OKRs Open Planning (this thread)
- [ ] Move 2019 Q1 OKRs to [2019 Q1 OKRs Spreadsheet](https://docs.google.com/spreadsheets/d/1BtOfd7s9oYO5iKsIorCpsm4QuQoIsoZzSz7GItE-9ys/edit#gid=274358435)

License: MIT
Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>
@ghost ghost assigned alanshaw Dec 11, 2018
@ghost ghost added the status/in-progress In progress label Dec 11, 2018
@daviddias daviddias changed the title 2019 Q1 JS IPFS WG OKRs OKRs - 2019 Q1 JS IPFS WG Dec 11, 2018
@daviddias daviddias changed the title OKRs - 2019 Q1 JS IPFS WG OKRs - 2019 Q1 JS Core WG Dec 12, 2018
@vasco-santos
Copy link
Member

vasco-santos commented Dec 17, 2018

IPNS has feature parity with go

I would like to do some improvements in IPNS in the next quarter.

At first, DDC WG would really benefit from having a streaming api for IPNS, as go-ipfs recently added. Context from @dirkmc comment: ipfs/js-ipfs#1725 (comment)

I also added an OKR in libp2p land for having the stream available in the DHT, which is a requirement for implementing this here.

go-ipfs also had some recent improvements, such as some stuff in the configuration file and experimental parameters that we still lack in JS land.

Service worker gateway v2

We have been discussing the next version of the service-worker-gateway for a long time. Maybe someone can have a hand on this during Q1. There are a panoply of use cases that will benefit from this.

References:

ipfs/service-worker-gateway#30
https://github.com/ipfs/developer-meetings/blob/master/DEEP_DIVES/service-worker.md

@daviddias
Copy link
Member

@lidel I believe that the In Web Browsers WG is taking over the Service Worker Gateway project, is that correct? Possibly ipfsd-ctl too?

@alanshaw
Copy link
Member Author

  • 13 weeks in the quarter: 65 days (assuming 5 day work week)
  • Planned holidays: 54 days
  • Probable hack week/conference: 49 days
  • I aim to spend 90% of that time in JS IPFS WG: 44 days
  • 60% of that is ~26 days or ~207 hours

@achingbrain
Copy link
Member

Carried over from Q4 2018

  • A repo migration tool exists and JS IPFS can automatically upgrade older repos
  • Continuous deployment for js projects
    • Dust needs to settle on which CI tool we are using
    • Resolve how we can store deploy key type secrets without them being vulnerable to exfiltration
    • Start with js-ipfs-unixfs-* and js-ipfs-mfs
    • Expand to js-ipld-* and js-libp2p-*

New for Q1 2019

@alanshaw
Copy link
Member Author

Things to finish

  • Base32 CIDv1 migration is complete

This is basically 2/3 complete. We have the CID tool, CID version agnostic get and CID base option. The final step is to take the plunge and make the switch to CIDv1 and base32 encoding by default.

For additional bonus points we could migrate the repo to key on multihashes instead of CIDs. However this might depend on there being a tool to migrate a repo (or we could do it ad hoc on access) and has consequences for pinning that would need to be worked out.

  • A JS IPFS daemon is one (or more!) of the IPFS gateway nodes

I'm not super excited to work on this but I feel as though this is really necessary towards being production ready.

  • A website is published to visually track a set of basic performance profiles of JS IPFS releases over historical versions

This basically already exists, but needs to be hooked up to CI and more benchmarks added before we can really announce it.

  • Test quality is improved, duplicates removed and coverage increased by at least 5%

I really just need to have the time to get stuck into this...

New things I want to tackle

  • Drive async iterators endeavour - aim for proposal/PR to at least half of all repos in scope

  • Pubsub in the browser - dependency on switch to fetch API, enable cancelable requests

  • Fix up libdweb discovery and transport modules - include by default in companion

@hugomrdias
Copy link
Member

Things to finish:

  • merge bundle size PRs (publish guidelines to help keep bundle size small)
  • CI/CD setup for our repos
  • async iterators endeavour multihashing/async, mplex, etc.

New Things:

  • do a second pass for bundle size ( this is crypto packages only )
  • service worker gateway v2
  • better browser datastores (with support for web streams and arraybuffers instead of node streams/buffer)

@lidel
Copy link
Member

lidel commented Dec 18, 2018

@lidel I believe that the In Web Browsers WG is taking over the Service Worker Gateway project, is that correct? Possibly ipfsd-ctl too?

@daviddias Depends on resources available to WB WG in Q1. If it comes with time commitment from someone, then I'd be happy to add KR related to Service Worker to Web Browsers WG OKRS at #804

Update: had a call with @alanshaw and @hugomrdias, and we will be moving Service Worker and other browser-specific KRs to In Web Browser WG (#804) to make OKRs more clear/aligned with WG missions.

@alanshaw
Copy link
Member Author

  1. @achingbrain please do Q4 OKR scoring

  2. @hugomrdias please do async retro

  3. @achingbrain @vasco-santos I've taken a stab at the Q1 OKRs - I could do with some suggestions for grouping/objective naming...

@alanshaw
Copy link
Member Author

These OKRs have a big ask from infra 🙏 - we'd like a JS IPFS node to be added as one of the nodes that make up the IPFS gateway cluster.

I envisage this to be a process of slow introduction:

  1. Initially I want to get JS IPFS running on a macine or vm provisioned in a similar way to the existing nodes, in order for it to be easy to add as a gateway node when the time comes. I suspect this will require some pointers to existing docs or access to a repo where the provisioning scripts are kept so that I can learn how it fits together and replicate it for a JS IPFS node.

  2. Run the machine for a suitable period to assess it's stability. Test it's ability to process requests and handle load. I'll need some information from infra about the load that it would be expected to handle so that a suitable stress test can be devised.

  3. Use the machine or set or machines as the default gateway in a tool like companion (@lidel will this be possible? - perhaps in the beta channel?) to further test in real world scenarios.

  4. Slow introduction to the gateway cluster - can we introduce it and slowly ramp up the load as it proves it's worth?

cc @eefahy @ipfs/infra

Copy link
Member

@vasco-santos vasco-santos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

Left a small suggestion.

OKR/JS_CORE.md Outdated

* p0 - Base32 encoded version 1 CIDs are the default for new content added to IPFS @alanshaw
* p0 - IPNS has a streaming API @vasco-santos
* p1 - IPNS supports publishing parameters `ttl`, `dhtt` and `dhtrc` @vasco-santos
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As the dht parameters are for resolve, I would put it in the following way:

Suggested change
* p1 - IPNS supports publishing parameters `ttl`, `dhtt` and `dhtrc` @vasco-santos
* p1 - IPNS supports `ttl`, `dhtt` and `dhtrc` parameters @vasco-santos

@alanshaw
Copy link
Member Author

Ok, I've reworded https://github.com/ipfs/team-mgmt/blob/7bdd947db5cf95a95fce0745891c8ff2178d8091/OKR/JS_CORE.md

@vasco-santos @achingbrain @daviddias can I get a review please 🙏

@achingbrain
Copy link
Member

A complete implementation of IPFS in JavaScript

We should probably sort out the missing parts of the API if this is our headline task - wiring in the DAG API to the http client, etc.

@alanshaw
Copy link
Member Author

A more complete implementation of IPFS in JavaScript

??? I'm open to suggestions - also what is the "etc."?

@achingbrain
Copy link
Member

what is the "etc."?

Dunno, the DAG API was the first thing that came to mind.

I think a complete implementation is what we should be shooting for, js-ipfs can't be taken seriously without it - just need to put it on the OKRs list. Am happy to take it on, but will have to drop some other stuff.

@alanshaw
Copy link
Member Author

Ok so in terms of the HTTP API, JS IPFS does not implement the following endpoints:

  • /bitswap/ledger
  • /bitswap/reprovide
  • /bootstrap/add/default
  • /bootstrap/list
  • /bootstrap/rm/all
  • /commands
  • /config/profile/apply
  • /dag/put
  • /dag/resolve
  • /dht/*
  • /diag/*
  • /files/chcid
  • /filestore/*
  • /log/*
  • /mount
  • /object/diff
  • /p2p/*
  • /refs/*
  • /repo/fsck
  • /repo/gc
  • /repo/verify
  • /swarm/filters/*
  • /tar/*
  • /update

That would be a LOT to do in the quarter! There's a subset of that API that I'd consider "good enough" for 99% of people using JS IPFS to call it "complete". I'd say the most important ones to implement might be /dag/put, /dag/resolve, /repo/gc, /refs and /swarm/filters/* (/dht is basically done).

/dag* should be relatively easy to replicate what Go IPFS is doing, /repo/gc will be the big one, /refs should be easy and /swarm/filters/* might be tricky and a foray into libp2p land :D

Note: I've listed the HTTP API here but it makes sense to also implement the CLI for these at the same time.

@achingbrain do you want to drop the homebrew OKR in favour of implementing those (or a different set)?

Either way, we're not going to be fully complete this quarter but as an aspirational objective do you think it works? You never know we might have time to implement everything 😉.

@achingbrain
Copy link
Member

😨 There's a lot to do!

Yes, lower priority items in favour of fixing up js-ipfs sounds sensible.

@vasco-santos
Copy link
Member

I also agree with the API higher priority!

@alanshaw
Copy link
Member Author

I have refactored - approve @vasco-santos @achingbrain?

Copy link
Member

@vasco-santos vasco-santos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚢

@olizilla
Copy link
Member

The GUI team needs an observable / event-emitter style api for reporting what work IPFS is doing and what is being waited on for a given get request, so we can communicate to the user work being done on their behalf by a network of peers instead of just a "loading pls wait!"

It looks like your list is pretty full for Q1. Is that something we could do some ground work for in Q1?


### Inviting for developers and contributors

* p1 - Continuous deployment drives `js-ipfs-unixfs-*` and `js-ipfs-mfs` [@achingbrain](https://github.com/achingbrain)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What work does this KR translate into?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once we settle on a new CI this'll involve:

  1. Figuring out a strategy to use e.g. solving questions like:
    • does publish a prerelease or a release?
    • how does CI know what version number to increment?
    • when does a release happen? push to master?
  2. Configuring the CI to enact the release ensuring secrets are handled correctly
  3. Testing the setup beforehand on a test module

* p0 - A JS IPFS daemon is one (or more!) of the IPFS gateway nodes [@alanshaw](https://github.com/alanshaw)
* p2 - Perf benchmarking site is launched and 10 different benchmarking scenarios exist [@alanshaw](https://github.com/alanshaw)
* p0 - IPFS is a transport for [npm/tink](https://github.com/npm/tink) [@achingbrain](https://github.com/achingbrain)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: This objective should contain a KR on "js-ipfs can find (Peer Routing) any peer and dial to it (direct or relay) if this Peer is online."

Potentially adding a job that keeps trying to dial to many different nodes in the network, just to be sure that the connection always happens.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potentially adding a job that keeps trying to dial to many different nodes in the network, just to be sure that the connection always happens.

This is to mitigate against a failed connection due to network error or your laptop goes offline or something? I'm surprised this isn't already implemented!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alanshaw I meant a job in our infra that keeps checking that JS can make quality connections everywhere as the network grows. But yeah, your proposal (if not done yet) is sound too.

@alanshaw
Copy link
Member Author

Is that something we could do some ground work for in Q1?

Absolutely. I think giving users feedback when IPFS "goes away" searching for something would be really valuable. Some suggestions for groundwork tasks that could usefully be done:

  • A proposal on interface-ipfs-core for this feature
  • Engage Go IPFS, and all libp2p teams to check they are on board with adding the feature
  • Check it is possible (how would it work over HTTP?) and would not be too invasive
  • Some dream code™️ for what the API would look like...on the JS side and I guess you could get someone from the Go team to imagineer how it could work in go

@alanshaw alanshaw mentioned this pull request Dec 19, 2018
4 tasks
@alanshaw
Copy link
Member Author

@achingbrain do you approve these OKRs?

@jbenet
Copy link
Member

jbenet commented Dec 19, 2018

can't see retro doc-- not sure what perms should be (public or js-ipfs core dev team only)

@daviddias
Copy link
Member

@jbenet the current default is WG only and invitees on share request.

### A complete implementation of IPFS in JavaScript

* p0 - Base32 encoded version 1 CIDs are the default for new content added to IPFS [@alanshaw](https://github.com/alanshaw)
* p0 - IPNS has a streaming API [@vasco-santos](https://github.com/vasco-santos)
Copy link
Member

@lidel lidel Dec 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 as this will enable ipfs/ipfs-companion#643 with js-ipfs :)

@lidel
Copy link
Member

lidel commented Dec 23, 2018

3. Use the machine or set or machines as the default gateway in a tool like companion (@lidel will this be possible? - perhaps in the beta channel?) to further test in real world scenarios.

If infra sets up a dedicated subdomain that will only route to JS-based gateways (eg. js.gateway.ipfs.io & go.gateway.ipfs.io), then we can use it as the default public gateway in ipfs-companion / ipfs-desktop (for all beta users or just n% of users/requests).

@daviddias
Copy link
Member

Hellloooo helloo JS Core WG!

I'll be your concierge for finalizing your OKRs for Q1 2019! Please read the fantastic tutorial that @momack2 prepared on how you can do a simple Roadmap timeline exercise with your team to identify and reach consensus in which are the most important priorities, you can find the instructions at ipfs/roadmap#17

I'm available to answer all your questions, take part on the timeline exercise and review your iterations.

I understand that 2/3 of JS Core - IPFS are out this week. Please ack when you all get to this.

@daviddias daviddias added the P1 High: Likely tackled by core team if no one steps up label Jan 15, 2019
@momack2
Copy link
Contributor

momack2 commented Feb 5, 2019

Hi JS Core WG! Your OKRs look really great and already updated with the Q1 OKR sheet (and they all have owners! 🙌) - can we merge this PR now and freeze OKRs for the quarter?

@alanshaw alanshaw merged commit 02c0c13 into master Feb 5, 2019
@ghost ghost removed the status/in-progress In progress label Feb 5, 2019
@daviddias daviddias deleted the 2019-q1-okrs-js-ipfs-wg branch February 6, 2019 13:40
@daviddias
Copy link
Member

@alanshaw this looks great! Thanks for getting the OKRs finalized.

Just one tiny tiddy little bit. Can you remove the OKRs from https://github.com/ipfs/team-mgmt/blob/master/OKR/JS_CORE.md and just point to the spreadsheet as a single source of truth?

@alanshaw
Copy link
Member Author

alanshaw commented Feb 6, 2019

sorry about that - #862

@momack2 momack2 mentioned this pull request Mar 18, 2019
6 tasks
@momack2 momack2 mentioned this pull request Jul 15, 2019
13 tasks
@momack2 momack2 mentioned this pull request Jan 8, 2020
7 tasks
@daviddias daviddias mentioned this pull request Apr 8, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
P1 High: Likely tackled by core team if no one steps up
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants