feat: add ability to use existing config during init #6489

djdv · 2019-07-09T22:46:13Z

Related to:
#6262

This patch adds
ipfs daemon --init --init-config-file=${PATH}
and
ipfs init --config-file=${PATH}
allowing users to specify existing configuration files instead of (re)generating them.

Does not deal with merging of existing configs and/or profiles (yet?).

Stebalien

Good start. We can do merging in a followup.

cmd/ipfs/daemon.go

Stebalien · 2019-07-10T00:30:41Z

cmd/ipfs/init.go

 		profile, _ := req.Options[profileOptionName].(string)

+		if cfgLocation != "" {
+			if profile != "" {
+				return errInitConfigArgs


Any reason to reject this?

Probably not.
I was thinking we needed to merge profiles rather than use the config transformers. I'll update this.

Okay so this works now.

set IPFS_PATH=Z:\ipfs-dev-tmp1 ipfs init set IPFS_PATH=Z:\ipfs-dev-tmp2 ipfs daemon --init --init-config-file=Z:\ipfs-dev-tmp1\config --init-profile=badgerds,randomports diff Z:\ipfs-dev-tmp1 Z:\ipfs-dev-tmp2

Only in Z:\ipfs-dev-tmp2: badgerds Only in Z:\ipfs-dev-tmp1: blocks diff "Z:\\ipfs-dev-tmp1/config" "Z:\\ipfs-dev-tmp2/config" 11,34c11,18 < "mounts": [ < { < "child": { < "path": "blocks", < "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2", < "sync": true, < "type": "flatfs" < }, < "mountpoint": "/blocks", < "prefix": "flatfs.datastore", < "type": "measure" < }, < { < "child": { < "compression": "none", < "path": "datastore", < "type": "levelds" < }, < "mountpoint": "/", < "prefix": "leveldb.datastore", < "type": "measure" < } < ], < "type": "mount" --- > "child": { > "path": "badgerds", > "syncWrites": true, > "truncate": true, > "type": "badgerds" > }, > "prefix": "badger.datastore", > "type": "measure" 41,42c25,26 < "/ip4/0.0.0.0/tcp/4001", < "/ip6/::/tcp/4001" --- > "/ip4/0.0.0.0/tcp/16936", > "/ip6/::/tcp/16936" Only in Z:\ipfs-dev-tmp1: datastore diff "Z:\\ipfs-dev-tmp1/datastore_spec" "Z:\\ipfs-dev-tmp2/datastore_spec" 1c1 < {"mounts":[{"mountpoint":"/blocks","path":"blocks","shardFunc":"/repo/flatfs/shard/v1/next-to-last/2","type":"flatfs"},{"mountpoint":"/","path":"datastore","type":"levelds"}],"type":"mount"} \ No newline at end of file --- > {"path":"badgerds","type":"badgerds"} \ No newline at end of file Common subdirectories: Z:\ipfs-dev-tmp1/keystore and Z:\ipfs-dev-tmp2/keystore

I'd say allow both (but we can stick to transformers for now). That is:

Take the initial config, if supplied.

Merge it with the default config (later).

Apply transformers (if specified).

djdv · 2019-07-10T14:53:23Z

Rewrote the first commit to be more accurate

-               cmds.StringOption(configOptionName, "Use supplied config instead of generating one"),
+               cmds.StringOption(configOptionName, "Use supplied config as a base instead of the default"),

Edit:
now with the correct commit...

Stebalien · 2019-07-10T17:30:50Z

cmd/ipfs/init.go

-		var profiles []string
-		if profile != "" {
-			profiles = strings.Split(profile, ",")
+		if cfgLocation != "" {


Hm, it looks like we currently accept one argument for the "initial config". We should:

Make sure we aren't passed both.

Consider deprecating that?

Or maybe we don't need this flag on init (we could only support passing the config as an argument on init?). @hugomrdias, what's the state of javascript here?

Stebalien · 2019-07-10T17:31:49Z

cmd/ipfs/init.go

+				return err
+			}
+
+			return doInit(os.Stdout, cctx.ConfigRoot, false, nBitsForKeypairDefault, profiles, conf)


Why not use the doInit call below? This one drops the empty flag and passes the wrong nBitsForKeypair option (although I guess that doesn't do anything right now).

cmd/ipfs/daemon.go

Stebalien · 2019-07-12T02:51:24Z

Status: Blocked on ipfs init /path/to/config versus ipfs init --config /path/to/config. We currently do the former, this patch does both. I'd prefer to not have two ways to do the same thing if possible.

cc @hugomrdias?

lanzafame · 2019-07-12T03:14:49Z

What does ipfs init /path/to/config do?

Stebalien · 2019-07-12T03:46:05Z

@lanzafame at the moment, it initializes with an existing config.

djdv · 2019-07-12T04:26:39Z

@Stebalien

ipfs init --config /path/to/config

Whether this was intended or not, I have to be pedantic here since it's relevant.

ipfs init --config (global flag) is not the same as ipfs init --config-file or ipfs daemon --init --init-config-file (subcommand specific flags).

Calling that results in this

>set IPFS_PATH=Z:\itmp-1
>ipfs init
initializing IPFS node at Z:\itmp-1
...
>set IPFS_PATH=Z:\itmp-2
>ipfs init --config=Z:\itmp-1\config
initializing IPFS node at Z:\itmp-1\config
Error: unexpected error while checking writeablility of repo root: open Z:\itmp-1\config\test: The system cannot find the path specified.

It's also a bit of a hasty blunder on my part to have forgotten the existing --config flag as well as the <default-config> handling. There's a lot of redundancy happening in this patch.

If we can, I'm not opposed to using ipfs init --config to act the same as ipfs init --config-file.
But this may be confusing if we handle that global flag in a unique way inside of init than we do in other other commands.

Explicitly pointing out that init handles config via stdin or a file path argument where --config-file only handles file paths.

Stebalien · 2019-07-12T04:45:34Z

Oh dear. --config appear to be equivalent to IPFS_PATH=... ipfs. That's really confusing and we should make that sane (i.e., make --config point to an out-of-repo config). But that's a separate issue.

Yeah, we need to make sure js/go agree here or it's going to be really confusing.

hugomrdias · 2019-07-12T15:50:08Z

in js:

we don't have the global --config flag
ipfs init accepts a json string input
- ipfs init "{}"
ipfs daemon doesn't support init or config
we do have IPFS_PATH for the repo path

making ipfs [cmd] --config point to an out-of-repo config will make ipfs init /path/config equal to ipfs init --config /path/config right ?
If the above is the way to go we don't need anything else, but that is probably a bigger change that we don't want right now.

so maybe,

ipfs init /path/config (no changes at all to master)
ipfs daemon --init --init-config /path/config

makes sense ?

On js side i'll code whatever we decide here no blockers there.

djdv · 2019-07-17T03:11:00Z

Went ahead with the change.
Removed ipfs init --config-file flag (in favour of existing file+stdin handling via <default-config>).
ipfs daemon --init --init-config-file became ipfs daemon --init --init-config
Cleanup in init.go remains in patch.

hugomrdias · 2019-07-22T09:46:51Z

@Stebalien can we unblock this ?

Stebalien

Ok, code LGTM (sorry for the delay).

@djdv could you write a quick sharness test for this?

hugomrdias · 2019-08-26T10:41:02Z

@djdv anything i can do to help with this ?

djdv · 2019-08-26T15:07:10Z

@hugomrdias
Would you mind looking at the sharness tests? I'm not sure what's breaking things here.

This PR aligns `ipfs init` and `ipfs daemon` with go-ipfs. ipfs/kubo#6489 `ipfs init` changed to accept a file path as an argument `ipfs daemon` changed to support `--init` and `--init-config` options. Now we can do `ipfs daemon --init --init-config /path/to/custom-config` refs: ipfs/js-ipfsd-ctl#303

djdv mentioned this pull request Jul 9, 2019

Separate identity values from configuration file #6262

Open

Stebalien requested changes Jul 10, 2019

View reviewed changes

djdv force-pushed the feat/init-with-config-file branch 2 times, most recently from 73615a8 to 69893e3 Compare July 10, 2019 14:51

djdv force-pushed the feat/init-with-config-file branch from 69893e3 to 4bb2df4 Compare July 10, 2019 15:02

Stebalien reviewed Jul 10, 2019

View reviewed changes

djdv force-pushed the feat/init-with-config-file branch 3 times, most recently from 915e376 to 7b3e818 Compare July 10, 2019 21:42

djdv force-pushed the feat/init-with-config-file branch from 7b3e818 to 8c9fab9 Compare July 17, 2019 03:01

hugomrdias mentioned this pull request Jul 18, 2019

Improve overall state of js-ipfs tests ipfs/js-ipfs#2276

Closed

42 tasks

Stebalien requested changes Aug 5, 2019

View reviewed changes

Stebalien added the need_tests label Aug 5, 2019

djdv force-pushed the feat/init-with-config-file branch from 0755391 to c85a98f Compare August 15, 2019 14:23

feat: add ability to use existing config during init

995e674

djdv force-pushed the feat/init-with-config-file branch from c85a98f to 21b8aba Compare August 26, 2019 01:37

djdv force-pushed the feat/init-with-config-file branch 3 times, most recently from 56746d0 to 1a89933 Compare August 26, 2019 17:49

djdv force-pushed the feat/init-with-config-file branch 3 times, most recently from f12f59d to 6996faa Compare August 26, 2019 21:16

add init config tests

e7e7000

djdv force-pushed the feat/init-with-config-file branch from 6996faa to e7e7000 Compare August 28, 2019 23:00

Stebalien approved these changes Aug 28, 2019

View reviewed changes

Stebalien merged commit d2a1ce3 into ipfs:master Aug 28, 2019

hugomrdias mentioned this pull request Sep 5, 2019

feat: allow daemon to init and start in a single cmd ipfs/js-ipfs#2428

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add ability to use existing config during init #6489

feat: add ability to use existing config during init #6489

djdv commented Jul 9, 2019 •

edited

Loading

Stebalien left a comment

Stebalien Jul 10, 2019

djdv Jul 10, 2019

djdv Jul 10, 2019 •

edited

Loading

Stebalien Jul 10, 2019

djdv commented Jul 10, 2019 •

edited

Loading

Stebalien Jul 10, 2019

Stebalien Jul 10, 2019

Stebalien commented Jul 12, 2019

lanzafame commented Jul 12, 2019

Stebalien commented Jul 12, 2019

djdv commented Jul 12, 2019 •

edited

Loading

Stebalien commented Jul 12, 2019

hugomrdias commented Jul 12, 2019

djdv commented Jul 17, 2019

hugomrdias commented Jul 22, 2019

Stebalien left a comment

hugomrdias commented Aug 26, 2019

djdv commented Aug 26, 2019

feat: add ability to use existing config during init #6489

feat: add ability to use existing config during init #6489

Conversation

djdv commented Jul 9, 2019 • edited Loading

Stebalien left a comment

Choose a reason for hiding this comment

Stebalien Jul 10, 2019

Choose a reason for hiding this comment

djdv Jul 10, 2019

Choose a reason for hiding this comment

djdv Jul 10, 2019 • edited Loading

Choose a reason for hiding this comment

Stebalien Jul 10, 2019

Choose a reason for hiding this comment

djdv commented Jul 10, 2019 • edited Loading

Stebalien Jul 10, 2019

Choose a reason for hiding this comment

Stebalien Jul 10, 2019

Choose a reason for hiding this comment

Stebalien commented Jul 12, 2019

lanzafame commented Jul 12, 2019

Stebalien commented Jul 12, 2019

djdv commented Jul 12, 2019 • edited Loading

Stebalien commented Jul 12, 2019

hugomrdias commented Jul 12, 2019

djdv commented Jul 17, 2019

hugomrdias commented Jul 22, 2019

Stebalien left a comment

Choose a reason for hiding this comment

hugomrdias commented Aug 26, 2019

djdv commented Aug 26, 2019

djdv commented Jul 9, 2019 •

edited

Loading

djdv Jul 10, 2019 •

edited

Loading

djdv commented Jul 10, 2019 •

edited

Loading

djdv commented Jul 12, 2019 •

edited

Loading