dep ensure fails on non-git repositories #144

ChrisHines · 2017-01-25T01:59:34Z

go version go1.7.3 windows/amd64

Steps to reproduce:

go get github.com/ChrisHines/dep-test
dep init
dep ensure

Expected to see package bitbucket.org/zombiezen/gopdf/pdf in the vendor folder.

Instead I got this error:

C:\Users\Chris\Go\src\github.com\ChrisHines\dep-test>dep init

C:\Users\Chris\Go\src\github.com\ChrisHines\dep-test>dep ensure
grouped write of manifest, lock and vendor: error while writing out vendor tree: error while exporting bitbucket.org/zombiezen/gopdf: destination already exists

Additional hints:

I first encountered this on a bigger (private) project. The error can also occur for packages hosted in Bazaar repos on launchpad.net. Hacking on the dep code to dump out additional info I traced the origin of the error to https://github.com/golang/dep/blob/master/vendor/github.com/sdboyer/gps/util.go#L88 which is called a few stack frames down from https://github.com/golang/dep/blob/master/vendor/github.com/sdboyer/gps/result.go#L51. In short WriteDepTree makes the destination directory, but the implementation of interface method source.exportVersionTo for non-git repos calls copyDir which requires the destination not to exist.

The text was updated successfully, but these errors were encountered:

sdboyer · 2017-01-25T02:08:33Z

Oh gooooood, thank you! We definitely need help on the non git(hub) tests. I cover [some of] the basics in gps' tests for bzr and hg, but it's much harder to find good cases in the wild to really put it through its paces.

Do you mind replicating this issue over on gps, so we can get it fixed there?

sdboyer · 2017-01-25T06:28:49Z

actually nvm, it's nbd - i'll fix it over there in the morning, then we can just close this out :)

The main fix here is avoiding creating an empty directory for the destination, as copyDir() doesn't like that. Instead, we create only up to the parent dir. The other bit is ensuring the source repos exist in the cache before attempting to export them, for both bzr and hg. Addresses golang/dep#144

sdboyer · 2017-01-26T06:08:22Z

Taking a bit longer than expected - windows problems.

sdboyer · 2017-02-07T01:07:24Z

Should be fixed now (actually, back when #205 was merged). Do you mind retrying the case that gave you trouble?

ChrisHines · 2017-02-07T06:09:54Z

It may be partially fixed, but it still fails during dep ensure. Below is a full transcript of a session on the same test repo starting from scratch using dep built from 466bb30:

C:\Users\Chris\Go\src\github.com\ChrisHines\dep-test>dep status
could not find project manifest.json, use dep init to initiate a manifest

C:\Users\Chris\Go\src\github.com\ChrisHines\dep-test>dep init -v
dep: Finding dependencies for "github.com/ChrisHines/dep-test"...
dep: Found 1 dependencies.
dep: Building dependency graph...
dep: Package "github.com/ChrisHines/dep-test", analyzing...
dep: Package "github.com/ChrisHines/dep-test" has import "bitbucket.org/zombiezen/gopdf/pdf", analyzing...
dep: Could not determine version for "bitbucket.org/zombiezen/gopdf", omitting from generated manifest
dep: Analyzing transitive imports...
dep: Analyzing "bitbucket.org/zombiezen/gopdf/pdf"...
dep: Solving...
Root project is "github.com/ChrisHines/dep-test"
 1 transitively valid internal packages
 1 external packages imported from 1 projects
✓ select (root)
| ? attempt bitbucket.org/zombiezen/gopdf with 1 pkgs; 6 versions to try
| | try bitbucket.org/zombiezen/gopdf@default
| ✓ select bitbucket.org/zombiezen/gopdf@default w/1 pkgs
✓ found solution with 1 packages from 1 projects

Solver wall times by segment:
     b-list-versions: 1.1513933s
     b-source-exists:  608.654ms
         b-list-pkgs: 568.7343ms
              b-gmal: 568.3437ms
         select-root:   1.0027ms
            new-atom:    988.5µs
  b-deduce-proj-root:         0s
             satisfy:         0s
         select-atom:         0s
               other:         0s

  TOTAL: 2.8991165s

dep: Writing manifest and lock files.

C:\Users\Chris\Go\src\github.com\ChrisHines\dep-test>dep ensure -v
Root project is "github.com/ChrisHines/dep-test"
 1 transitively valid internal packages
 1 external packages imported from 1 projects
✓ select (root)
| ? attempt bitbucket.org/zombiezen/gopdf with 1 pkgs; at least 1 versions to try
| | try bitbucket.org/zombiezen/gopdf@default
| ✓ select bitbucket.org/zombiezen/gopdf@default w/1 pkgs
✓ found solution with 1 packages from 1 projects

Solver wall times by segment:
              b-gmal: 565.2885ms
     b-source-exists: 564.3366ms
         b-list-pkgs: 548.3366ms
         select-root:   5.0151ms
            new-atom:   2.0051ms
  b-deduce-proj-root:         0s
               other:         0s
             satisfy:         0s
         select-atom:         0s

  TOTAL: 1.6849819s

grouped write of manifest, lock and vendor: error while writing out vendor tree: error while exporting bitbucket.org/zombiezen/gopdf: destination already exists

sdboyer · 2017-02-07T12:56:26Z

Great, thanks for checking in on it again - I've got a space in which I should be able to poke more seriously at windows testing now, so I should be able to get ahead of this better.

sdboyer · 2017-02-07T13:25:47Z

Oh, also - did you test hg repositories as well, or just bzr? If not, would you mind renaming this issue appropriately?

ChrisHines · 2017-02-08T02:25:48Z

I've seen the problem with both hg and bzr, but my test repo only imports one of them. I can track down a package of the other type of add it as a dependency to the test repo. I'll post back here when it is ready.

ChrisHines · 2017-02-10T01:35:51Z

I've updated my test repo. It now has two external dependencies, one is hg hosted, the other is bzr hosted. They are imported with _ imports, so there is no actual code needed. The minimal reproducer is:

package main

import (
	_ "bitbucket.org/zombiezen/gopdf/pdf"
	_ "launchpad.net/gocheck"
)

func main() {
}

sdboyer · 2017-04-02T01:46:46Z

OK so...weird. I finally actually tried to reproduce this problem (sorry), and found that I could not on either OSX or linux. And, my original fix for this includes a test that writes out from git, bzr, and hg, and it passes on windows.

The fact that it's ensure vs. init shouldn't really matter. But, could you verify that this is still a problem with the latest dep? If so, we can dig in more on replication.

ChrisHines · 2017-04-09T02:04:57Z

I just updated dep with go get -u -v github.com/golang/dep, which put me on commit 7f85e04.

The problem bug still occurs on the code I posted above.

go version go1.8 windows/amd64

C:\Users\Chris\Go\src\github.com\ChrisHines\dep-test>dep init -v
dep: Finding dependencies for "github.com/ChrisHines/dep-test"...
dep: Found 1 dependencies.
dep: Building dependency graph...
dep: Package "github.com/ChrisHines/dep-test", analyzing...
dep: Package "github.com/ChrisHines/dep-test" has import "bitbucket.org/zombiezen/gopdf/pdf", analyzing...
dep: Could not determine version for "bitbucket.org/zombiezen/gopdf", omitting from generated manifest
dep: Package "github.com/ChrisHines/dep-test" has import "launchpad.net/gocheck", analyzing...
dep: Could not determine version for "launchpad.net/gocheck", omitting from generated manifest
dep: Analyzing transitive imports...
dep: Analyzing "bitbucket.org/zombiezen/gopdf/pdf"...
dep: Analyzing "launchpad.net/gocheck"...
dep: Solving...
Root project is "github.com/ChrisHines/dep-test"
 1 transitively valid internal packages
 2 external packages imported from 2 projects
✓ select (root)
| ? attempt bitbucket.org/zombiezen/gopdf with 1 pkgs; 6 versions to try
| | try bitbucket.org/zombiezen/gopdf@default
| ✓ select bitbucket.org/zombiezen/gopdf@default w/1 pkgs
| | ? attempt launchpad.net/gocheck with 1 pkgs; 20 versions to try
| | | try launchpad.net/gocheck@(default)
| | ✓ select launchpad.net/gocheck@(default) w/1 pkgs
✓ found solution with 2 packages from 2 projects

Solver wall times by segment:
     b-list-versions: 14.8983669s
              b-gmal:  755.9688ms
         b-list-pkgs:  713.0233ms
         select-root:    3.5561ms
            new-atom:    2.4717ms
     b-source-exists:          0s
             satisfy:          0s
         select-atom:          0s
               other:          0s
  b-deduce-proj-root:          0s

  TOTAL: 16.3733868s

dep: Writing manifest and lock files.

C:\Users\Chris\Go\src\github.com\ChrisHines\dep-test>dep ensure -v
Root project is "github.com/ChrisHines/dep-test"
 1 transitively valid internal packages
 2 external packages imported from 2 projects
✓ select (root)
| ? attempt bitbucket.org/zombiezen/gopdf with 1 pkgs; at least 1 versions to try
| | try bitbucket.org/zombiezen/gopdf@default
| ✓ select bitbucket.org/zombiezen/gopdf@default w/1 pkgs
| | ? attempt launchpad.net/gocheck with 1 pkgs; at least 1 versions to try
| | | try launchpad.net/gocheck@(default)
| | ✓ select launchpad.net/gocheck@(default) w/1 pkgs
✓ found solution with 2 packages from 2 projects

Solver wall times by segment:
         b-list-pkgs: 774.0829ms
              b-gmal: 704.5915ms
     b-source-exists: 558.2251ms
            new-atom:     3.46ms
         select-root:   2.9827ms
               other:         0s
         select-atom:         0s
             satisfy:         0s
  b-deduce-proj-root:         0s

  TOTAL: 2.0433422s

grouped write of manifest, lock and vendor: error while writing out vendor tree: error while exporting bitbucket.org/zombiezen/gopdf: destination already exists

The main fix here is avoiding creating an empty directory for the destination, as copyDir() doesn't like that. Instead, we create only up to the parent dir. The other bit is ensuring the source repos exist in the cache before attempting to export them, for both bzr and hg. Addresses golang#144

ChrisHines · 2017-05-04T04:06:30Z

I'm starting to follow up on this issue. I added some debug prints, rebuilt, and ... nothing changed. Hmmm, I got bit by building with go install -v github.com/golang/dep rather than go install -v github.com/golang/dep/cmd/dep, so my dep binary wasn't getting rebuilt. It looks like that may have been the case on my most recent reports above as well.

So here is the current status, actually building 2c974c5:

C:\Users\Chris\Go\src\github.com\ChrisHines\dep-test>dep init -v
dep: Finding dependencies for "github.com/ChrisHines/dep-test"...
dep: Found 1 dependencies.
dep: Building dependency graph...
dep: Found import of "bitbucket.org/zombiezen/gopdf/pdf", analyzing...
dep: Could not determine version for "bitbucket.org/zombiezen/gopdf", omitting from generated manifest
dep: Found import of "launchpad.net/gocheck", analyzing...
dep: Could not determine version for "launchpad.net/gocheck", omitting from generated manifest
dep: Analyzing transitive imports...
dep: Analyzing "bitbucket.org/zombiezen/gopdf/pdf"...
dep: Analyzing "launchpad.net/gocheck"...
dep: Solving...
Cached bitbucket.org/zombiezen/gopdf
Cached launchpad.net/gocheck
Root project is "github.com/ChrisHines/dep-test"
 1 transitively valid internal packages
 2 external packages imported from 2 projects
(0)   ✓ select (root)
(1)     ? attempt bitbucket.org/zombiezen/gopdf with 1 pkgs; 0 versions to try
panic: version queue is empty, should not happen

goroutine 1 [running]:
github.com/golang/dep/gps.(*solver).findValidVersion(0xc0421a65a0, 0xc042047400, 0xc04216b100, 0x1, 0x1, 0x0, 0xc04216b100)
        C:/Users/Chris/Go/src/github.com/golang/dep/gps/solver.go:804 +0x601
github.com/golang/dep/gps.(*solver).createVersionQueue(0xc0421a65a0, 0xc04214a6a0, 0x1d, 0x0, 0x0, 0xc04216b100, 0x1, 0x1, 0x0, 0x0, ...)
        C:/Users/Chris/Go/src/github.com/golang/dep/gps/solver.go:791 +0xb03
github.com/golang/dep/gps.(*solver).solve(0xc0421a65a0, 0x0, 0x0, 0x1e)
        C:/Users/Chris/Go/src/github.com/golang/dep/gps/solver.go:408 +0x478
github.com/golang/dep/gps.(*solver).Solve(0xc0421a65a0, 0x34, 0xa6bd00, 0xab9608, 0xc0420fbf60)
        C:/Users/Chris/Go/src/github.com/golang/dep/gps/solver.go:359 +0xb2
main.(*initCommand).Run(0xc042129421, 0xc042112c00, 0xc042054290, 0x0, 0x0, 0x0, 0x0)
        C:/Users/Chris/Go/src/github.com/golang/dep/cmd/dep/init.go:166 +0xd6d
main.main()
        C:/Users/Chris/Go/src/github.com/golang/dep/cmd/dep/main.go:128 +0x6f9

peterbourgon added the bug label Jan 25, 2017

This was referenced Jan 26, 2017

Export non-git VCS correctly sdboyer/gps#161

Merged

Chase latest gps #205

Merged

ChrisHines mentioned this issue May 4, 2017

Store version map in cache on success (instead of failure) to retrieve it. #514

Merged

sdboyer closed this as completed in #514 May 5, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dep ensure fails on non-git repositories #144

dep ensure fails on non-git repositories #144

ChrisHines commented Jan 25, 2017

sdboyer commented Jan 25, 2017 •

edited

Loading

sdboyer commented Jan 25, 2017

sdboyer commented Jan 26, 2017

sdboyer commented Feb 7, 2017

ChrisHines commented Feb 7, 2017

sdboyer commented Feb 7, 2017

sdboyer commented Feb 7, 2017

ChrisHines commented Feb 8, 2017

ChrisHines commented Feb 10, 2017

sdboyer commented Apr 2, 2017

ChrisHines commented Apr 9, 2017

ChrisHines commented May 4, 2017

dep ensure fails on non-git repositories #144

dep ensure fails on non-git repositories #144

Comments

ChrisHines commented Jan 25, 2017

sdboyer commented Jan 25, 2017 • edited Loading

sdboyer commented Jan 25, 2017

sdboyer commented Jan 26, 2017

sdboyer commented Feb 7, 2017

ChrisHines commented Feb 7, 2017

sdboyer commented Feb 7, 2017

sdboyer commented Feb 7, 2017

ChrisHines commented Feb 8, 2017

ChrisHines commented Feb 10, 2017

sdboyer commented Apr 2, 2017

ChrisHines commented Apr 9, 2017

ChrisHines commented May 4, 2017

sdboyer commented Jan 25, 2017 •

edited

Loading