glide get on a working repo gave me errors

[thockin]

I got my repo all building with glide, and then added a dep (to test and document).

glide get github.com/renstrom/dedent --update-vendored --strip-vcs --strip-vendor

This produced a lot of output and these errors:

[ERROR] Error scanning github.com/google/cadvisor/container/rkt: open /home/thockin/tmp/glide/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/container/rkt: no such file or directory

[ERROR] Error scanning github.com/docker/docker/pkg/term/windows: open /home/thockin/tmp/glide/src/k8s.io/kubernetes/vendor/github.com/docker/docker/pkg/term/windows: no such file or directory

glide then exited, leaving my tree a mess. I have new directories that I don't know why, I have a ton of modified files, and I have new deps listed in glide.lock (totally unrelated stuff). Note that the dep I imported has ZERO external imports.

Now I am stuck. I don't know why glide did what it did, and I don't know how to proceed. I can get my repo back to a sane state, but how do I import this one tiny dep?

Aside from wanting to know how to proceed, here's a chance for better messages explaining what's going on and why. Truthfully, I am not sure I understand why glide is doing so much work here - I didn't ask for --all-dependencies

[thockin]

I added a spew.Dump() before and after the call to installer.Update in the get path, and then diffed them. What I see is some new subpackages like "Godeps/_workspace/src/github.com/coreos/go-semver/semver" being slurped in.

It would make sense that, if a recursively vendored dep was newer than my vendored version, and that dep included other packages that I don't, that what I am experiencing might happen. Do we need to undo Godep rewriting before processing?

[thockin]

hmm, no, that's wrong. I did the same dump on update and I seethe same sorts of thing.

One thing I notice is that the "before" case for update has the Pin field set, but the get has Pin: "" for everything. That's the diff except for pointer values and the actual new package (from get).

[thockin]

@vishh this is the biggest blocker to using glide for kubernetes right now.

[mattfarina]

@thockin I need to get a bugfix release out in the next day or so. After that I'll look into this. We were talking about this issue earlier in the day. I'm going to look for the fastest and least complicated solution for this.

Can I assume https://github.com/thockin/kubernetes/tree/glide is where you're working on this? If I had a suggested change would you be open to a pull request there? Note, I am under the Google CLA already.

Right now the --strip-vendor flag should cause the godep rewrites to be undone if they are vendoring in the Godeps/_workspace directory. This will happen inside of Update.

[thockin]

That is the correct github tree, and if you have proposed changes I am all
ears - either PR me or email a patch or whatever. :)

Thanks.

On Tue, Apr 5, 2016 at 3:53 PM, Matt Farina notifications@github.com
wrote:

@thockin https://github.com/thockin I need to get a bugfix release out
in the next day or so. After that I'll look into this. We were talking
about this issue earlier in the day. I'm going to look for the fastest and
least complicated solution for this.

Can I assume https://github.com/thockin/kubernetes/tree/glide is where
you're working on this? If I had a suggested change would you be open to a
pull request there? Note, I am under the Google CLA already.

Right now the --strip-vendor flag should cause the godep rewrites to be
undone if they are vendoring in the Godeps/_workspace directory. This
will happen inside of Update.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#367 (comment)

[thockin]

Any news here? Given that we are going to embrace go-1.6, our change to Glide is even easier -- assuming we can understand/fix this issue (or document whatever I am doing incorrectly)

[sdboyer]

I'm not sure how difficult @mattfarina thinks this would be in the current architecture. Given that glide does a fair bit of operating in-place within the vendor directory, it might be tricky. So, this might be a bit easier to wait for vsolver on.

(Which oughtn't be that long. If progress continues as it has, maybe as early as next week!_

[thockin]

I don't actually know what "this" is. To my eyes, get just doesn't work,
and I am sort of baffled how something so fundamental to the tool is
"broken". That said, I understand that my usage pattern might be outside
of what people think of as normal (but then what do you think is "normal"?)

We can't commit to glide until we can get packages, I'm relatively
ambivalent about exact solutions

On Tue, Apr 12, 2016 at 11:43 PM, Sam Boyer notifications@github.com
wrote:

I'm not sure how difficult @mattfarina https://github.com/mattfarina
thinks this would be in the current architecture. Given that glide does a
fair bit of operating in-place within the vendor directory, it might be
tricky. So, this might be a bit easier to wait for vsolver on.

(Which oughtn't be that long. If progress continues as it has, maybe as
early as next week!_

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#367 (comment)

[sdboyer]

@thockin briefly, "this" is the problem of doing in-place updates in vendor, rather than assembling a new vendor somewhere else, and only moving it into place when we're sure it's complete and correct.

The in-place approach was originally taken as a performance optimization. When people aren't committing their vendor dirs, the harm is at least less obvious, because the whole vendor dir is just treated as disposable - rm -rf and try again. (to be clear, I don't like that, or think it's OK - one of many reasons I'm writing vsolver). As glide has only recently been aiming to support the committability of vendor via those --strip-vendor and --strip-vcs commands, there are things like this that appear to be right smack in the middle of an obvious workflow but that still have problems.

We're working...feverishly! (I just committed the vsolver code earlier today that writes out a complete and correct vendor dir that can then be moved in to place.)

[thockin]

Sounds great - though an out-of-place rebuild of vendor/ which gets swapped
into place also sounds OK to me (less debris upon failure).

I'm happy to sit on it for a bit, but I just wanted to be very clear that
get is a requirement for us :)

On Wed, Apr 13, 2016 at 10:38 AM, Sam Boyer notifications@github.com
wrote:

@thockin https://github.com/thockin briefly, "this" is the problem of
doing in-place updates in the vendor directory, rather than assembling a
new vendor directory somewhere else, and only moving it into place when
we're sure it's complete and correct.

The in-place approach was originally taken as a performance optimization.
When people aren't committing their vendor dirs, the harm is at least less
obvious, because the whole vendor dir is just treated as disposable - rm
-rf and try again. (to be clear, I don't like that, or think it's OK -
one of many reasons I'm writing vsolver). As glide has only recently been
aiming to support the committability of vendor via those --strip-vendor
and --strip-vcs commands, there are things like this that appear to be
right smack in the middle of an obvious workflow but that still have
problems.

We're working...feverishly! (I just committed the vsolver code earlier
today that writes out a complete and correct vendor dir
https://github.com/sdboyer/vsolver/blob/master/result.go#L22 that can
then be moved in to place.)

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#367 (comment)

[mattfarina]

@thockin There are a couple things we'll be doing.

1. vsolver is coming and with it we will be doing some changes to the way things are handled.
2. I'm going to work on get to make it a better experience no matter how solving improves. In support of that I've been working on lower level VCS functionality (because Go does more than git) to support what's next this week and hope to come back around to get and some other Glide work next week.

The work your doing on kubernetes also gave me an idea for a plugin (some of which makes sense to roll into Glide). You'll see more of me on this stuff next week.

Having working compatibility with 4 VCS can take more time than I'd like.

[thockin]

For your amusement - I got some pushback to do the move to vendor/ separately from the move to glide. I tried. I can't - we still have some deps that use path rewrites and godep isn't smart enough to un-rewrite itself. But Glide is. Or would be if get worked :)

[thockin]

I managed to convert our tree to a vendor dir, but now many people are asking "what about glide"? So what about it...

[sdboyer]

I believe this particular problem's essentially addressed now in the vsolver PR, #384, at least for glide install and glide update, both of which vsolver is now powering. Some caveats there, but it's fundamentally working.

They're really the hard part, and having them means the system fundamentally works - we can solve, and write out a vendor tree. Converting glide get to follow the same pattern is mostly trivial, now that those are working. There's some other, higher priorities first, but we'll get there soon.

Help is always welcome 😄

[mattfarina]

The get, update, and install commands all do their work outside the vendor/ directory now. If all of the work was successful it exports the packages and moves the export to the vendor/ location.

So, if there is a problem in the middle of resolving it exits without mangling the vendor/ directory. This brings about more safety.

I'm going to close this issue out now that the original problem are, I think, all fixed. If not please open new issues.