Workflow: local cache servers for remote packages

[zevdg]

Spinning this item out of #174

In general, this issue is to discus avoiding leftpad-type outages while using the dep tool. Historically in go, the advice has been to check in your vendor directory to ensure that you can always rebuild. Dep stores enough metadata to reproduce builds without needing to commit your vendor folder, however choosing to do this forces us to find a different way to ensure that remote resources needed to reproduce builds are accessable. Some languages do this with an immutability policy on a central repository, but 1) go does not have a central repository that can impose this kind of rule and 2) even if it did, it would be a single point of failure that could cause a catastrophic outage. Due to that, enterprise companies generally have some kind of local cache of all remote packages they use in their code bases (most likely a meta tool like nexus or artifactory although most languages have a language specific equivalent). The dep tool will need to support this workflow to some degree.

The good news is that it already does. Alternate sources provide for exactly this kind of functionality BUT the workflow given the existing feature set isn't great. You must specify an alternate source for EVERY external dependency (not just top level) used by your project. You can write tooling to do this for you, but even if you do, you're gonna end up with huge manifests that generally obscure any real, non-cache related constraints you want on your code base. The project seems to be moving towards minimal manifests and this is exactly the opposite. Furthermore, in many cases, companies in this situation would like some easily verifiable guarantee that that external requests are not being made. Forgetting to specify a required override to the local cache is too easy to do/overlook and relying on developers to remember to update the manifest every time any dependency is added (especially ones that aren't top level) is a massive footgun.

I suspect almost every enterprise company wants some form of "get all external packages from a cache and all internal packages directly from source". There are a few ways this could be done.

1. dep could support some limited form of fancy conditional logic. for example:

   "rules":{
       "*.intranet.mycorp.com": { 
           "negate": true
           "source": "cache.intranet.mycorp.com",
       }
   }

2. dep could allow an external script or tool to do arbitrary transformations. for example

   "rewriteCmd": "fancysourcerewriter -some args ${ImportPath}"

3. dep could support only a global alternate source proxy and make that proxy handle the more complex rules about which package requests should be routed where.

   "cacheServer": "cache.intranet.mycorp.com"

I'm currently leaning towards option 3 as it keeps dep from getting too magical and gives enterprises some centralized place to enforce guarantees, but I'm interested to hear from other people in this situation.

[zevdg]

Just noticed this seems to be a dupe of #248