vendor: add fake imports for glide's benefit

We've repeatedly encountered issues arising from the fact that entries
in `glide.yaml` are not treated as the, or even as additional, roots in
the calulation of the transitive dependency closure.

Previously in #11797, we attempted to clarify the situation by pruning
all derivable specs from `glide.yaml`, so that it would be clear where
the dependency roots were indeed coming from.

We however left vendored tools, and their transitive dependencies,
there, as they were not derivable from any imports (e.g. some are not
importable due to `package main`).

As we've already found though, simply adding a tool to `glide.yaml`, or
even `glide get`'ing them, doesn't work.

Thus, to reliably vendor tools and their dependencies, we need to
introduce artifical imports of them. We get around the `package main`
issue with by hiding the imports in a file that is build-tagged `+glide`
so we never attempt to build it.

---

The `glide up` used to test this picked up a test-only raft change,
etcd-io/etcd#6935, and:

- golang/tools@ae1141f...3d92dd6
- cockroachdb/cockroach-go@2c874f1...140a8c5
- etcd-io/etcd@cfd10b4...b713113
- distribution/distribution@67095fb...844b928
- google/go-github@171a931...43e7458
- opencontainers/runc@5974b4c...34f23cb

build/README.md

@@ -40,9 +40,12 @@ and checked out as a submodule at `./vendor`.
 ## Updating Dependencies
 This snapshot was built and is managed using `glide`.
 
-The [docs](https://github.com/Masterminds/glide) have detailed instructions, but in brief:
+The [docs](https://github.com/Masterminds/glide) have detailed instructgions, but in brief:
 * run `./scripts/glide.sh` in `cockroachdb/cockroach`.
 * add new dependencies with `./scripts/glide.sh get -s github.com/my/dependency`
+	- Note: if you are adding a non-import dependency (e.g. a binary tool to be used in development),
+		please add a dummy import to `build/tool_imports.go`. This is a workaround for an upstream issue:
+		https://github.com/Masterminds/glide/issues/690.
 * update dependencies to their latest version with `./scripts/glide.sh up`
   - to pin a dependency to a particular version, add a `version: ...` line to `glide.yaml`, then update.
   - to update a pinned dependency, change the version in `glide.yaml`, then update.

build/checkdeps.sh

@@ -14,7 +14,7 @@ echo "checking that 'vendor' matches manifest"
 
 echo "checking that all deps are in 'vendor''"
 
-missing=$(cat GLOCKFILE | cut -f2 -d' ' | grep -v '^./pkg/' | awk '{print} END {print "./pkg/..."}' \
+missing=$(sed -n 's,[[:space:]]*_[[:space:]]*"\(.*\)",\1,p' build/tool_imports.go | awk '{print} END {print "./pkg/..."}' \
   | xargs go list -f '{{ join .Deps "\n"}}{{"\n"}}{{ join .TestImports "\n"}}{{"\n"}}{{ join .XTestImports "\n"}}' \
   | grep -v '^github.com/cockroachdb/cockroach' \
   | sort | uniq \

build/tool_imports.go

@@ -0,0 +1,48 @@
+// Copyright 2016 The Cockroach Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+// implied. See the License for the specific language governing
+// permissions and limitations under the License.
+
+// +build glide
+
+package main
+
+// Our vendoring tool, glide, only considers imports as the set of roots used
+// to compute the transitive closure of dependencies it should resolve.
+// We depend on many tools which we don't actually import directly elsewhere –
+// e.g in a `go generate` or our Makefiles – but convincing `glide` to vendor
+// them, and their dependencies, is difficult: https://github.com/Masterminds/glide/issues/690
+//
+// Thus, instead of trying to use `glide get` or `glide.yaml` to document those
+// additional dependency roots, we add fake imports here for glide to find.
+// NB: Some of these are `package main` so they cannot actually be imported, so
+// this file is build-tagged +glide to prevent attempting to build it.
+
+import (
+	_ "github.com/client9/misspell/cmd/misspell"
+	_ "github.com/cockroachdb/c-protobuf/cmd/protoc"
+	_ "github.com/cockroachdb/crlfmt"
+	_ "github.com/cockroachdb/stress"
+	_ "github.com/golang/lint/golint"
+	_ "github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway"
+	_ "github.com/jteeuwen/go-bindata/go-bindata"
+	_ "github.com/kisielk/errcheck"
+	_ "github.com/kkaneda/returncheck"
+	_ "github.com/mattn/goveralls"
+	_ "github.com/mdempsky/unconvert"
+	_ "github.com/mibk/dupl"
+	_ "github.com/robfig/glock"
+	_ "github.com/wadey/gocovmerge"
+	_ "golang.org/x/tools/cmd/goimports"
+	_ "golang.org/x/tools/cmd/goyacc"
+	_ "golang.org/x/tools/cmd/stringer"
+)

glide.yaml

@@ -11,39 +11,10 @@ import:
   version: 7248742ae7127347a52ab9d215451c213b7b59da
 - package: github.com/gogo/protobuf
   version: ccdc7fbcb4cd13f34b76d7262805e58316faaaf4
-- package: github.com/grpc-ecosystem/grpc-gateway
-  subpackages:
-  - protoc-gen-grpc-gateway
 - package: golang.org/x/net
   version: 97edce0b2e423f6a8debb459af47f4a3cb4ff954
 - package: google.golang.org/grpc
   version: 79b7c349179cdd6efd8bac4a1ce7f01b98c16e9b
-- package: github.com/jteeuwen/go-bindata
-  subpackages:
-  - go-bindata
-- package: github.com/mattn/goveralls
-- package: github.com/client9/misspell
-  subpackages:
-  - cmd/misspell
-- package: github.com/cockroachdb/crlfmt
-- package: github.com/cockroachdb/stress
-- package: github.com/golang/lint
-  subpackages:
-  - golint
-- package: github.com/kisielk/errcheck
-- package: github.com/kkaneda/returncheck
-- package: github.com/mdempsky/unconvert
-- package: github.com/mibk/dupl
-- package: github.com/robfig/glock
-- package: github.com/wadey/gocovmerge
-- package: golang.org/x/tools
-  subpackages:
-  - cmd/goimports
-  - cmd/goyacc
-  - cmd/stringer
-- package: github.com/golang/glog
-- package: github.com/pborman/uuid
-- package: github.com/agtorre/gocolorize
 - package: google.golang.org/api
   version: 3ef9447cad8d725923ca024dab57786299cf745a
 - package: cloud.google.com/go