ensure does not fully populate vendor/

[gregory-m]

The bug sometimes happens on our ci machines like 1 per 20 builds.
dep doesn't install github.com/namsral/flag package.

Here some logs from our CI machines:

[16:56:25][Step 3/10] go get -u github.com/golang/dep/...
[16:56:29][Step 3/10] dep ensure -v
[16:56:29][Step 3/10] Root project is "github.com/wix-system/scheduler"
[16:56:29][Step 3/10]  15 transitively valid internal packages
[16:56:29][Step 3/10]  6 external packages imported from 6 projects
[16:56:29][Step 3/10] (0)   ��� select (root)
[16:56:29][Step 3/10] (1)	? attempt github.com/briandowns/spinner with 1 pkgs; at least 1 versions to try
[16:56:29][Step 3/10] (1)	    try github.com/briandowns/spinner@master
[16:56:30][Step 3/10] (1)	��� select github.com/briandowns/spinner@master w/1 pkgs
[16:56:30][Step 3/10] (2)	? attempt github.com/davecgh/go-spew with 1 pkgs; 3 versions to try
[16:56:30][Step 3/10] (2)	    try github.com/davecgh/go-spew@v1.1.0
[16:56:30][Step 3/10] (2)	��� select github.com/davecgh/go-spew@v1.1.0 w/1 pkgs
[16:56:30][Step 3/10] (3)	? attempt github.com/dustin/go-humanize with 1 pkgs; at least 1 versions to try
[16:56:30][Step 3/10] (3)	    try github.com/dustin/go-humanize@master
[16:56:30][Step 3/10] (3)	��� select github.com/dustin/go-humanize@master w/1 pkgs
[16:56:30][Step 3/10] (4)	? attempt github.com/fatih/color with 1 pkgs; at least 1 versions to try
[16:56:30][Step 3/10] (4)	    try github.com/fatih/color@master
[16:56:30][Step 3/10] (4)	��� select github.com/fatih/color@master w/1 pkgs
[16:56:30][Step 3/10] (5)	? attempt github.com/mattn/go-colorable with 1 pkgs; at least 1 versions to try
[16:56:30][Step 3/10] (5)	    try github.com/mattn/go-colorable@master
[16:56:30][Step 3/10] (5)	��� select github.com/mattn/go-colorable@master w/1 pkgs
[16:56:30][Step 3/10] (6)	? attempt github.com/mattn/go-isatty with 1 pkgs; at least 1 versions to try
[16:56:30][Step 3/10] (6)	    try github.com/mattn/go-isatty@master
[16:56:30][Step 3/10] (6)	��� select github.com/mattn/go-isatty@master w/1 pkgs
[16:56:30][Step 3/10] (7)	? attempt github.com/gregory-m/slack with 1 pkgs; at least 1 versions to try
[16:56:30][Step 3/10] (7)	    try github.com/gregory-m/slack@feature/update-with-attachments
[16:56:30][Step 3/10] (7)	��� select github.com/gregory-m/slack@feature/update-with-attachments w/1 pkgs
[16:56:30][Step 3/10] (8)	? attempt github.com/wix-private/go-api with 1 pkgs; at least 1 versions to try
[16:56:30][Step 3/10] (8)	    try github.com/wix-private/go-api@master
[16:56:30][Step 3/10] (8)	��� select github.com/wix-private/go-api@master w/1 pkgs
[16:56:30][Step 3/10] (9)	? attempt github.com/namsral/flag with 1 pkgs; 3 versions to try
[16:56:30][Step 3/10] (9)	    try github.com/namsral/flag@v1.7.4-pre
[16:56:30][Step 3/10] (10)  ���   github.com/namsral/flag@v1.7.4-pre not allowed by constraint master:
[16:56:30][Step 3/10] (10)      master from (root)
[16:56:30][Step 3/10] (9)	    try github.com/namsral/flag@v1.7.4-alpha
[16:56:30][Step 3/10] (9)	��� select github.com/namsral/flag@v1.7.4-alpha w/1 pkgs
[16:56:30][Step 3/10] (10)  ? attempt golang.org/x/sys with 1 pkgs; at least 1 versions to try
[16:56:30][Step 3/10] (10)      try golang.org/x/sys@master
[16:56:31][Step 3/10] (10)  ��� select golang.org/x/sys@master w/1 pkgs
[16:56:31][Step 3/10] (11)  ? attempt golang.org/x/net with 1 pkgs; at least 1 versions to try
[16:56:31][Step 3/10] (11)      try golang.org/x/net@master
[16:56:32][Step 3/10] (11)  ��� select golang.org/x/net@master w/1 pkgs
[16:56:32][Step 3/10]   ��� found solution with 11 packages from 11 projects
[16:56:32][Step 3/10] 
[16:56:32][Step 3/10] Solver wall times by segment:
[16:56:32][Step 3/10]          b-list-pkgs: 2.336814271s
[16:56:32][Step 3/10]      b-source-exists: 222.127134ms
[16:56:32][Step 3/10]               b-gmal: 154.849261ms
[16:56:32][Step 3/10]              satisfy:   1.781078ms
[16:56:32][Step 3/10]          select-atom:   1.427346ms
[16:56:32][Step 3/10]          select-root:   1.210623ms
[16:56:32][Step 3/10]             new-atom:    461.948��s
[16:56:32][Step 3/10]   b-deduce-proj-root:    141.055��s
[16:56:32][Step 3/10]            b-matches:     69.688��s
[16:56:32][Step 3/10]      b-list-versions:     57.037��s
[16:56:32][Step 3/10]                other:     45.847��s
[16:56:32][Step 3/10]           b-pair-rev:     24.712��s
[16:56:32][Step 3/10]       b-pair-version:      5.426��s
[16:56:32][Step 3/10] 
[16:56:32][Step 3/10]   TOTAL: 2.719015426s
[16:56:32][Step 3/10] 
[16:56:32][Step 3/10] Process exited with code 0

[16:56:32][Step 5/10] go test `go list ./... | grep -v '/vendor/'`
[16:56:33][Step 5/10] cmd/scheduler-server/main.go:11:2: cannot find package "github.com/namsral/flag" in any of:
[16:56:33][Step 5/10] 	/some/path/vendor/github.com/namsral/flag (vendor tree)
[16:56:33][Step 5/10] 	/some/other/path/src/github.com/namsral/flag (from $GOROOT)
[16:56:33][Step 5/10] 	/some/path/src/github.com/namsral/flag (from $GOPATH)
[16:56:33][Step 5/10] make: *** [test] Error 1
[16:56:33][Step 5/10] Makefile:5: recipe for target 'test' failed
[16:56:33][Step 5/10] Process exited with code 2
[16:56:33][Step 5/10] Step Test (Command Line) failed

And github.com/namsral/flag indeed not in vendor directory:
(in step 9 I do ls vendor/*/*)

[16:56:33][Step 9/10] vendor/github.com/briandowns:
[16:56:33][Step 9/10] spinner
[16:56:33][Step 9/10] 
[16:56:33][Step 9/10] vendor/github.com/davecgh:
[16:56:33][Step 9/10] go-spew
[16:56:33][Step 9/10] 
[16:56:33][Step 9/10] vendor/github.com/dustin:
[16:56:33][Step 9/10] go-humanize
[16:56:33][Step 9/10] 
[16:56:33][Step 9/10] vendor/github.com/fatih:
[16:56:33][Step 9/10] color
[16:56:33][Step 9/10] 
[16:56:33][Step 9/10] vendor/github.com/gregory-m:
[16:56:33][Step 9/10] slack
[16:56:33][Step 9/10] 
[16:56:33][Step 9/10] vendor/github.com/mattn:
[16:56:33][Step 9/10] go-colorable
[16:56:33][Step 9/10] go-isatty
[16:56:33][Step 9/10] 
[16:56:33][Step 9/10] vendor/github.com/wix-private:
[16:56:33][Step 9/10] go-api
[16:56:33][Step 9/10] 
[16:56:33][Step 9/10] vendor/golang.org/x:
[16:56:33][Step 9/10] net
[16:56:33][Step 9/10] sys
[16:56:33][Step 9/10] Process exited with code 0

Also I attach manifest.json and lock.json:
dep.zip

[ankitm123]

Hey @sdboyer, if it is fine with you, I would like to jump on this issue, and see if I can fix it.

[sdboyer]

@ankitm123 that'd be great!

[gregory-m]

@ankitm123 ping me if you need more info.

I'am unable to reproduce issue on my dev box (MacOS). Our CI agents runs inside docker containers on debian.

[ankitm123]

@gregory-m. I run ubuntu on my machine. I will see if I can reproduce it. I will let you know soon, if I need more info from you.

[gregory-m]

I made debug session, and its looks like docker storage driver issue and not dep.
I can't 100% confirm, and I will dig deeper tomorrow.

@ankitm123 If you can't reproduce it, I think its good Idea to stop trying to find not existing bug.
Sorry for wasting your time.

I will update issue tomorrow.

[ankitm123]

@gregory-m, no problem. I will wait for your update. In the meantime, I will dive a bit deeper into the source code.

[gregory-m]

Ok I found root case. I don't know if its bug or expected behavior.
dep ensure dons't update dependencies if vendor folder exists.

Consider flowing example:

⇢ $ cat manifest.json                                                                                                                                                                                
{
    "dependencies": {
        "github.com/gregory-m/test": {
            "branch": "master"
        }
    }
}
⇢ $ cat lock.json
{
    "memo": "2137cf9e0a427bf8a8f33b2a682aa7e3333f874febb163e1ff86900bbc90bb62",
    "projects": [
        {
            "name": "github.com/gregory-m/test",
            "branch": "master",
            "revision": "7e46f3e0412d1ae295eeb1d241c38e337c2f9363",
            "packages": [
                "."
            ]
        }
    ]
}

⇢ $ dep ensure
⇢ $ ls vendor/github.com/gregory-m/test
LICENSE  v2.go
⇢ $ rm -rf vendor/github.com/gregory-m/test
⇢ $ dep ensure
⇢ $ ls vendor/github.com/gregory-m/*
zsh: no matches found: vendor/github.com/gregory-m/*

Our CI agents checkout source code to the same place in file system and don't remove source code after build,
so if I add vendor to .gitignore, next build can run with vendor directory from previous build and dep ensure will not touch it.

I don't know if its expected behavior and I personally think this is confusing.
But if so I think you should document this behavior.

Thanks.

[sdboyer]

@gregory-m OH - yes, this is a known issue. Sorry, there are so many things floating around that my mind doesn't immediately go to the right thing.

There's a simple (less ideal) solution to this, and a more complicated (more ideal) one.

The simple solution is adding a flag to ensure - something like a -force-vendor, which unconditionally rebuilds the vendor dir from the lock.

We're a few degrees of yak away from the more complicated solution, because it involves #121. Ideally, we'd be able to inspect the vendor/ directory and just know that it's out of date, and repopulate it - no user intervention required. This should be the default behavior.

We have a mechanism for doing this already between manifest and lock - just by looking at the manifest + import set, we can determine if the lock still represents a valid solution. (That's what the memo field is for in your lock files). But we don't have an analogous capability for lock and vendor yet.

[gregory-m]

@sdboyer no problem. If you can, please add this info to Readme so new users will not fall on same issue again and again.

Good thing, I actually found bug with aufs on our CI kernel.

[sdboyer]

Unfortunately there's still a number of little gotchas like this; I'm not sure that trying to add them to the README would be that effective? But the issue is here now, and I'll update the title for a bit more discoverability.

[sdboyer]

this was fixed by #489, as we now always write out vendor/ on dep ensure in order to specifically avoid this confusing case (even though it's often wasteful)