Allow using source replacement with fallback to crates.io

[luser]

We're working on using source replacement in Firefox so that we can vendor our crate dependencies into mozilla-central. This works pretty well in practice, but we'd also like to enable developers to do "normal" Rust development without extra hassle. The current implementation includes adding a .cargo/config in the Firefox objdir to point at our vendored crates, and also running cargo build --frozen to ensure that we're not hitting the network. For our CI builds this is perfect. For local development it's not. We talked about adding a --enable-rust-developer-mode switch to turn those off, but it would be nice if we could do that by default in non-automation builds. However, if we don't do source replacement by default that means that Firefox builds will start requiring a network connection, which we have previously not required. Additionally, it seems silly to fetch crates from crates.io when they're sitting there in the source checkout.

It would be great to have a way to use source replacement but allow fallback--we should be able to point cargo at our vendored crates, but if someone adds a new dependency cargo should be able to fetch it from crates.io as usual (assuming we're not building with --frozen). This way the default build would simply use our vendored crates, but adding new crate dependencies would not require adding any special build options, and developers would just run the mach vendor command we'll be adding to vendor in the new dependencies for landing.

cc @alexcrichton @wycats

[alexcrichton]

This sounds like a good feature to have to me! My preference of how to do this would be a "multi source" along the lines of:

[source.foo]
# ...

[source.bar]
# ...

[source.baz]
any = ['foo', 'bar']

[source.crates-io]
replace-with = 'baz'

The definition of an "any" source would be to just check each source in order and return the first that has the package. Cargo would then also assume that all sources have the same checksum, but wouldn't verify it until you built a crate.

[luser]

For simplicity, could we use the same configuration for "developer mode" and "CI mode", and just have the latter build with --frozen?

[alexcrichton]

I believe that would work, yeah, at least with what I'm thinking

[LegNeato]

Similarly, when using a top-level shared vendor directory and top-level source replacement config and a developer wants to add a new dependency, there appears to be no way to easily vendor in the new dependency without blanking out the source replacement configuration, running cargo vendor, and then re-adding back the configuration. This is a pain, but perhaps I missed a better way?

How about something like cargo vendor sync or cargo vendor download that bypasses the vendor / source replacement config for just that command...which might cover this case too?

Of course, if someone is experimenting/developing they could add dependencies that ultimately aren't needed and bloat the repo...maybe a better way would be cargo vendor stage (looks in vendor dir, falls back to crates.io, but DOESN'T write it to the global vendor and instead writes it to a local vendored spot).

The developer experience might look like:

1. Add deps to Cargo.toml
2. Run cargo vendor stage. This downloads missing crates from crates.io and vendors somewhere temporary (or downloads to the global vendor directory with a file named .VENDOR-STAGED so that downstream tools can do smart things).
3. Iterate, perhaps adding or removing deps.
4. Run cargo vendor sync to download and save any deps in the vendor dir (or remove .VENDOR-STAGED in each crate from step Support tests #2).
5. Commit changes in version control

Not sure this is 100% the same as the original issue, feel free to tell me to file another issue.

[luser]

We do have the same issue in the Firefox build, but we sort of cheat around it because the .cargo/config file that specifies the source replacement is placed in the objdir, which is where we run cargo during the build, but mach vendor rust runs cargo vendor from the srcdir, so it doesn't use the source replacement.

[alexcrichton]

@LegNeato yeah those sound like real issues! Want to open a or some bugs over at cargo-vendor?

[LegNeato]

@luser yeah, I've been reading bugzilla about your setup 😄 . I think we need to support not having a meta buildsystem / script layer above making the end-to-end vendoring workflow in the shared vendor case...even if most big companies will merely call out via their existing tools. I'll file some new issues!

[alexcrichton]

This was discussed at the recent Mozilla all-hands (cc @froydnj @rillian) as an issue that'd help Gecko quite a bit. Instead of what I mentioned above though I think I'd probably advocate instead for something like:

[source.crates-io]
vendor-cache = 'path/to/vendor'

Or something along those. I don't think we should prematurely try to generalize this, and let's just stick to the workflow of "I've got vendored crates in my source tree which come from crates.io" and then Cargo can just "do the right thing"

[luser]

That sounds fine. As I mentioned earlier, if the same config would "just work" for our CI and our developer builds that would be great, and we would simply make CI builds use --frozen but not developer builds, so that forgetting to vendor something is still an error in CI, but cargo would be free to simply fetch the new crate for a local build.

[alexcrichton]

Yeah I don't think it'd be too hard to set up something like that, I think we could make it work!

[tbarusseau]

Hi, was this ever implemented? I can't find anything in the documentation: https://doc.rust-lang.org/cargo/reference/config.html

I'm trying to use cargo vendor as an alternative to the currently hack-ish solutions to only build the dependencies of a project (for caching purposes), and I guess cargo vendor could work?

But without a crates.io fallback this doesn't sound like a great idea.

[dmulder]

8 years on, and this is still an issue. I also need vendor'd crates for CI to work, but that breaks local development without a crates.io fallback.