Support a style of npm shrinkwrap

[itsjamie]

glide pin would change to directly generating a glide-pin.yaml file.
glide install would check if a glide-pin.yaml file existed, and if it did would install using that instead of glide.yaml
glide up(date) would reference glide.yaml and would update the underlying repositories, hopefully having the semantic version support discussed in #52.

This can be achieved currently using the global -y option, but I think having in-tool support would make adoption easier. Especially when recursing through other packages that might have glide inside them, there is no option for a user to choose which YAML file to use, so in the case of someone rolling this themselves, they would require their default yaml file be the pinned version.

Thanks for any consideration.

[technosophos]

I've been playing with Elixir this weekend, and I like the fact that their tool has support for dev, test, and prod "environments".

I wonder if we could borrow a fragment of that, yet basically keep the same idea as is presented here.

For example, we could say that by default, Glide is in dev mode, but provide a global flag for setting modes. glide -m prod would explicitly be prod mode, while glide -m dev would explicitly be dev mode.

Each mode would by default first look for glide-MODE.yaml (or some similar pattern TBD), and then fall back to glide.yaml.

Then glide pin -m prod would generate glide-prod.yaml. While glide up -m prod would update based on the glide-prod.yaml file (if it exists).

If we were to go this way, what really are the modes? Ruby and Elixir break down into prod, dev, and test. Those seem reasonable to me... though I'm not really sure what the use case for test would be.

[itsjamie]

Rather than allowing any mode, I would limit it to either "pinned" or not.

I think this is a case where less is actually more. If I could think of another use-case where I would want to have multiple pinned dependencies for my codebase based on the environment then I would get this.. but whatever your environments look like for your product, you should really only ever have two states ongoing at any point in your codebase.

Something bound for prod, which might hit many environments on it's way to prod. Such as... integration -> qa -> staging -> prod and then the current dev build. Anything bound for prod in our use-case would be pinned the whole way, and the dependencies version shouldn't change based on what environment they are in (though their configuration might).

Does that make sense?

[mattfarina]

@itsjamie thanks for bringing this up.

While I have my opinion on workflow and i'm sure @technosophos does as well, I'm not sure Glide should enforce just one. Instead I'd like it to enable developers. Sometimes there cases, even edge cases, that you may need something like this for.

@technosophos I kinda like your suggestion. I'm trying to come up with some real world use cases it satisfies. Any pointers on Ruby and Elixir I could look at?

For test, maybe you have a mocked monitoring library or some other tool pulled instead of the real thing. I'm not saying this is ideal but may provide some reason for it.

[itsjamie]

@mattfarina I understand the want to give more options.

I just ask that you think about the case of recursive glide up's. If the default for pinned dependencies isn't tool defined and is instead driven by the -m flag, I would think there would have to be an interactive mode to list each possible glide-mode.yaml file when processing the sub-dependency.

[mattfarina]

@itsjamie great point on the recursive case!

[technosophos]

I actually lost sleep over this last night. I find the idea to be really compelling, but I'm worried that if we don't do it just right it will become a huge headache.

That said, I like the idea of tying pinned yaml files to VCS tags (e.g. having a v1.2.3.yaml file for v1.2.3), but that might be excessive.

Maybe we could enumerate some use cases and that would give us a little more clarity on how we should build this (and what the problems are to solve).

For me, CI/CD tooling is a really big deal here, since that was my original target for the pin logic. So I'll start with a few use cases:

• As a CI/CD dev, I want to capture an exact specification for dependencies so that I can reproduce builds
• As a CI/CD dev, I want to not have to guess which glide file is the right one for a build
• As a CI/CD dev, I do not want to have to worry about recursive dependency resolution; that should be handled by someone more familiar with the code

[itsjamie]

Right.

So currently, the way that I implemented this was having two glide.yaml files. One called the default glide.yaml. This is the pinned dependencies, that will be automatically used. Second, I have one called glide-update.yaml, which has the standard refs that I want to reference (generally branches).

Updating the dependencies is interesting...
glide -y glide-update.yaml up && glide -y glide-update.yaml pin > glide.yaml

I just hope to make it slightly easier for the end user, otherwise I'll probably end up throwing that command inside a makefile.

[technosophos]

That is a case I think is probably common. I'll have to think about it.

I think you can omit the >, since IIRC the pin command takes a filename, and defaults to STDOUT if no filename is given.

[itsjamie]

The only reason I brought it up, is that as a user of third party packages, I have to rely on them providing glide.yaml as their pinned dependencies currently.

By making the pin filename tool specific, when doing updates tool can look for that pinned file, and if it doesn't exist it could even raise a warning saying that the build will not be reproducible.

This would be the ideal 👍.

[technosophos]

The more I think about the workflow above, the more I like it. It's a brilliant way of keeping a dev environment up to date, but still unsurprising -- and still keeping the CI/CD workflow repeatable. Once we have semver support finished, it'll be even better.

[itsjamie]

I'll volunteer to send a PR that does this if you want.

glide pin - creates a glide-pinned.yaml
glide install - utilizes glide-pinned.yaml instead of the default glide.yaml if present

[technosophos]

@mattfarina any objections?

[mattfarina]

No objections

[technosophos]

A new idea has been floating around, and might solve this problem a different way. I'd really like @itsjamie 's input.

What if we change the workflow to allow two commands: glide pin and glide unpin, and we changed the YAML format to add a pin: field

glide pin would add the pin: COMMIT_ID to every dependency. glide unpin would remove the pin field.

So here's how it would work:

Commands that update and install (glide up, for example) would pull the pin: version if present, or else go back to the ref: version. That would mean we could have things like:

package: foo
ref: v1.2    # SEMVER constraint
pin: afe45  # Exact version

Developers would largely work in unpinned mode when updating dependencies, but then everything could be pinned at will. Since the same file is checked into VCS, VCS would have an exact record of what was pinned and when, but we wouldn't need extra files to do that. This would give both repeatable build artifacts and an easy path to updating, but using weaker references like branches and semver constraints.

Does that make sense? Think we could make that work?

/cc @kalbasit

[kalbasit]

Yep @technosophos that summarize what we talked about. Also possible glide repin which should glide unpin then update all dependencies (pull in master (or whichever branch ref reference) from upstream) and then glide pin again.

[itsjamie]

I think that all makes sense and sounds like it would work!

My only two cents would be that the commands support the standard package argument. So, typing just glide unpin wouldn't do anything you would need to glide unpin <pkg>, glide pin <pkg>. To do the whole project, it would be glide pin ./... from the root. Similarily, if glide repin was added, it would also require a package argument.

This would be to keep the standard convention of the Go toolchain, which would be important for adoption.

[technosophos]

I agree with @itsjamie that we should allow per-package pin/unpin/repin. I have mixed feelings about using ./... to mean "all packages" because it implies that we are working with path globs, which we're not. Or is there a situation in which we would?

[albrow]

I'm currently using glide in production and have some thoughts I would like to share on this discussion based on my experience with it so far. However, it's a little unclear to me what the goal is. Would someone mind explaining exactly what problem this issue aims to solve? @itsjamie originally mentioned npm shrinkwrap, which as I understand it allows developers to specify specific versions for their dependencies' dependencies. (Without shrinkwrap, npm would always install the newest version of each nested dependency that satisfies the constraints in their respective package.json. You don't have control over the package.json file in third-party dependencies, which can cause problems for CI tools and reproducible builds). Is that still the goal here? A real-world example would be extremely helpful.

[technosophos]

We're currently discussing how pinning, unpinning, and repinning should work. The idea is that instead of needing multiple files, we could store pin information in addition to ref information. So consider something like this:

package: github.com/technosophos/pika
import:
  - package: github.com/foo/bar
    ref: feature/whatever

Right now that means the package foo/bar is on a particular branch. If we ran glide pin, it would generate this:

package: github.com/technosophos/pika
import:
  - package: github.com/foo/bar
    ref: 17c2da2566839baf3631d71d62b8d689e31ad712

Now you have two choices:

• replace glide.yaml with this pinned version, in which case there is "no going back" to the old one, or
• start maintaining multiple files, like maybe a glide-pinned.yaml

What we've started talking about is tracking pin state inside of the glide.yaml file, but separately from the ref field. So running glide pin would change glide.yaml to this:

package: github.com/technosophos/pika
import:
  - package: github.com/foo/bar
    ref: feature/whatever
    pin: 17c2da2566839baf3631d71d62b8d689e31ad712

During update operations (install, up, etc.) glide would use pin fields as the definitive source. And now we have the opportunity to add some new features:

• glide unpin removes the pin field(s)
• glide repin removes the pin, updates to the latest, and then adds the pin back (glide unpin && glide up && glide pin)

The theory is that this workflow would improve a couple of different developer workflows:

• people like me, who prefer to only pin for releases, can easily do that by pinning right before a release, and unpinning afterward.
• others who prefer to keep dependencies static most of the time can easily leave things pinned, but at the start of a new development cycle (or whatever), they can quickly repin

So it's different in approach from npm shrinkwrap, but the idea is to make it easy to flip back and forth between explicit commit-based checkouts and more open "stay on head" workflows.

That's the idea, anyway.

[kalbasit]

+1 @technosophos that's exactly what I'd be looking for.

[technosophos]

This would be a breaking change to glide pin, since it would render glide pin FILENAME irrelevant. Further, I'd like to support glide pin github.com/foo/bar to allow selective pin (same with unpin and repin). That would make this command work the same as glide get and glide up.

Any objections? And advice on how we can do this without causing people major headaches?

[mattfarina]

@technosophos I think making breaking changes is OK right now. We're pre-release and the GO15VENDOREXPERIMENT is opt-in. It's early adopters who are using it. It would be better to make the change now than later. I'm for it.

[kalbasit]

@technosophos you can treat the argument as a possible filename (if it exists) and as a package if not. In the meantime you can start showing a warning starting now about it.

[itsjamie]

Sorry that I haven't followed up with a PR was out. I'll take a look at starting an implementation this coming week

[mattfarina]

Now that #156 has been merged and we have a lockfile I believe this is closed. If not please feel free to reopen.