Release channels take 2

[brson]

This RFC describes changes to the Rust release process, primarily the division of Rust's time-based releases into 'release channels', following the 'release train' model used by e.g. Firefox and Chrome; as well as 'feature staging', which enables the continued development of experimental language features and libraries APIs while providing strong stability guarantees in stable releases.

This revision includes a significant simplification by merging stability attributes with feature gates, which further enables some additional useful features within Cargo related to version detection. One big implication of this change is that stability attributes are no longer suitable for use outside of std (if such a feature is desirable it is intended to be solved by Cargo).

[alexcrichton]

cc @wycats, this is a culmination in what feature-related work we were discussing at the work week.

[steveklabnik]

rendered

[steveklabnik]

extra grave after the must

[andrew-d]

Just thinking out loud: it would be interesting if we could use a Cargo feature to enable some potentially-unstable APIs in a crate. For example, a Cargo feature "use_experimental" to conditionally-compile in use of some unstable rustc feature, but still allowing stable or beta users to use the crate without that feature. Not sure how practical that is, though.

[Valloric]

This whole "rustc automatically determines the compatible version" idea is wishful thinking. There will be tons of subtle (and often unexpected) language/compiler/library changes that affect backwards compatibility across versions. This happens with every compiler/language/framework and while unfortunate, any expectations to the contrary are unrealistic.

Assuming that rustc can then automatically determine that any non-trivial library built and tested with rustc 1.5 will work just fine with rustc 1.3 because it only declares it's using features available from 1.3 onwards is again, wishful thinking. Such a library would have to be tested by the authors before one could confidently say it works with 1.3.

Any attempts at heuristically figuring out library & compiler/language compatibility will produce more harm than good. Someone will end up using a library that hasn't been tested with an older compiler/language/std version and if they're extremely lucky, the library will fail in obvious ways. If they're unlucky, it will behave only slightly differently than the author of the library intended.

A much better, simpler and more robust approach is to just enforce that a minimum language version is specified for a library. The KISS rule is a good idea here. Any attempts by a software system at "lets figure out what he user actually meant" almost always end in tears. Examples:

• The JavaScript equality table.
• "Error-correcting" parsers for HTML4.
• Perl.

[nrc]

This all sounded good when we were discussing it in person, but now I see it written down, it does look rather complex.

[alexcrichton]

We do understand that this is not a rock-solid guarantee (what the minimum version of a crate is), but we also see the benefits of this detection to far outweigh the alternatives. One alternative (which it sounds like @Valloric is arguing for) is to simply do nothing, but this is strictly less useful than the system proposed here. If the compiler detects that 1.3 is required, then it must have at least 1.3, the failure mode is if a newer compiler is actually required. This is why a manual specification via Cargo.toml is allowed if necessary. We do not expect the majority of code to require a custom annotation in their Cargo.toml to request a particular nightly version.

Put another way, I personally doubt that adding this infrastructure will do more harm than good, and I also personally expect it to be quite a useful tool in upgrading rustc over time.

[Valloric]

I'm not suggesting to do nothing, I'm suggesting the simplest and sanest approach of just demanding that every library state what is the minimum version of rust it requires (I mentioned this in my previous post). Libraries already have to specify the version of a library dependency (AFAIK you can't say "I depend on libfoo" without specifying the version of libfoo) and for the same reasons they should also specify the version of the language they require.

Extend this idea of heuristics to library dependencies to see how absurd it is: libraries say they require libfoo and cargo waves a magic wand, applies some heuristics and pulls out a "compatible" version. It's obvious this won't work, and yet we want to do this for the language.

The language itself is a versioned dependency much like any other library. Why is it being treated differently?

[nrc]

"never" seems very strong here - we should be able to patch if there is a critical security issue, for example

[tshepang]

@nick29581 Rather the version be removed from crates.io than a different version be made to take the same name. In this case, the version with a critical security issue would be 1.0.1, and 1.0.0 would be unavailable so people don't use it.

[alexcrichton]

Yes this means that 1.0.0 will never change, not that we'll never release 1.0.1

[emk]

Thank you for looking into this!

As a library author, I'm going to miss #[unstable]. I was using this to indicate that certain APIs were available for testing purposes, but that I hadn't yet committed to full SemVer guarantees, and that I reserved the right to make changes before declaring something #[stable]. This was a nice escape hatch. Without it, I'm concerned that post-1.0 libraries will be a bit too much like "quick-setting cement"—the instant something appears in an API, I need to live with it for a very long time. It would be nice to say, "Hey, this is available, and if you use it, you'll get a warning that it might change."

In fact, #[unstable] was one of my favorite minor Rust features. :-) Would it be possible to have a compiler-and-std-specific version of #[unstable], and leave the existing attribute for use by libraries? If not, no worries.

[emk]

SemVer for Rust and std?

As the maintainer of heroku-buildpack-rust, I need to know exactly what compiler version should be used to build a git repository. Here are some use cases and a proposal that I posted in rust-lang/cargo#1044:

First, some use cases:

• ACME, Inc. has a large Rust application hosted on Heroku using a Rust buildpack. The application is a cash-cow: It earns a lot of money, and it has a lot of users, but it's only updated rarely. Six months after the last update, a critical bug is discovered. A two-line patch is produced, and it needs to be pushed to production.
• Rustalicious LLC is a boutique software consultancy specializing in Rust applications. They help maintain ACME's cash cow application. They're also working on Rust projects for 3 other clients, in varying stages of active development and maintenance.

In these examples, both ACME and Rustalicious need to control when they upgrade their Rust toolchain. ACME is trying to push an urgent fix to production, but they haven't touched the codebase in 4 Rust releases, and they don't have the time to upgrade to the lastest Rust compiler and verify that nothing is broken. They want to make a two-line fix using their old toolchain.

Rustalicious wants to use the latest and greatest Rust compiler for everything, because it's shiny. But they have multiple maintenance mode clients who depend on different versions of the Rust platform. So they potentially need to install multiple versions of Rust side-by-side, and manage them using a specialized tool.

Prior art: Bundler + buildpacks + RVM

In the Ruby world, it's possible to use Gemfile to specify the version of Ruby that should be used with an application:

`ruby '1.9.3'`

This information is read by Heroku's Ruby buildpack, and by various third-party tools like RVM that manage multiple, parallel Ruby implementations.

Proposed solution: Treat rustc and the standard libraries like any other versioned dependency

rustc and the standard library are an actual dependency for every Rust application. So it might be reasonable to specify their version numbers somewhere in Cargo.toml:

rust = "1.1.2"

This could go either in the package section, or possibily even in the dependencies section. It could potentially allow full version specifiers, and possibly even be recorded in Cargo.lock like any other dependency.

What would Cargo do with this information?

Two possibilities come to mind:

1. Verify that all packages can be build with the specified version of Rust, and if not, give an error message explaining the constraints. This would offer significant value to ordinary Cargo users, because they'd get comprehensible messages telling them when they needed to upgrade their compiler, instead of random error spew from one of the libraries they're using.
2. Store an exact version in Cargo.lock.

Third-party tools might also read this information to configure deployment servers (e.g., buildpacks), manage parallel Rust toolchains (e.g. RVM etc. for Rust), and decide which Rust version(s) should be used for continuous integration builds.

Alternatives

• Trust the compiler and standard libs to never have regressions.
• Store this information in a .rust_version file and coordinate between tool developers.

I'm not hugely attached to the details of this proposal, but I will need to address both the ACME and Rustalicious use cases in the future, with any luck. :-)

If it's useful, I'd be interested in contributing some code, but I'd need code review from the Cargo team.

I would love to be able apply SemVer rules to the "Rust platform" the same way I apply them to any other crate that I manage with Cargo.

As for the idea of automatically detecting the appropriate compiler version to use with a given source tree, my reactions are:

1. This has a huge number of moving pieces.
2. It doesn't satisfy any of the "ACME" or "Rustalicious" use cases described above. For example, if you've had an application running unchanged in production for 6 months (which is perfectly normal for some of my consulting clients!), and you need to make a 1-line bug fix, you really don't want your toolchain to play "guess a Rust version" for you. You want to make the 1-line change, recompile with whatever version of Rust you were using originally, and push to Heroku (or whatever).

To be fair, I can just modify heroku-buildpack-rust to use a .rust_version file, if my needs diverge too much from what everybody else needs.

[huonw]

Is this meant to be 1.0.0-beta?

[brson]

No. I intend beta and nightly artifacts to not include version numbers. It's mostly just so that old beta artifacts don't stick around forever - the beta artifacts are intended to be transient, unlike betas from a typical (non-channel-based) release process.

[huonw]

Oh, I see. ("1.0.0-nightly" is mentioned in the previous paragraph.)

[brson]

@huon I think maybe it will be better to have the version number in the beta artifacts. Makes it clearer what you are getting, but it does mean that 1.0.0-beta-*.tar.gz will get overwritten with updates during the release cycle. What do you prefer?

[nodakai]

Which "channel" will you recommend to a newcomer?

[tshepang]

@nodakai stable

[quantheory]

s/note/not/

[alexcrichton]

@nodakai yes, as @tshepang said we'd recommend the stable channel.

[emk]

@alexcrichton Thank you for your response! My apologies for being so stubborn about this issue, too.

This is actually one of the key points about #[unstable] which we don't think will work for library developers. As part of the standard distribution this is a hard error by default, and you have to get on nightly to make it only a warning.

I agree that a hard error #[unstable] makes no sense for library authors.

My proposal is to have two versions of #[unstable], with different names:

1. Rustc/std could use something like #[rustc_unstable] or #[std_unstable], with hard error semantics, as proposed here.
2. Third-party libraries could use something like #[unstable] which defaulted to a warning: "Hey, you know, this API may still change." I'd guess this would be a very short lint plugin, and I'd be happy to try to contribute it.

Here's a use-case for for something like #[unstable] in a library:

#[deriving(Show, Default)]
pub struct Hints<'a> {
    /// A value from an HTTP Content-Language header.  The value "fr,en"
    /// will bias the decoder towards French and English.
    pub content_language: Option<&'a str>,

    /// The top-level domain associated with this text.  The value "fr"
    /// will bias the decoder towards French.
    pub tld: Option<&'a str>,

    /// The original encoding of the text, before it was converted to
    /// UTF-8.  See `Encoding` for legal values.
    #[experimental]
    pub encoding: Option<Encoding>,

    /// An extra language hint.
    pub language: Option<Lang>
}

Here, I've finalized 3 out of 4 fields in a struct, but the encoding field is tricky. Right now, it's represented by a low-level Encoding enum taken directly from the C headers, but it should be a high-level type from rust-encoding. Unfortunately, there wasn't any way to depend on the rust-encoding library without dragging in some pretty huge data tables. So I marked this one field #[experimental] until we can sort this out. I don't want to block the release of an entire library over one minor design decision.

Basically, I feel that #[unstable] is far to useful a name to be reserved for something that's only used internally by std. I agree that the standard library needs a complicated and specialized mechanism for handling these issues. I just feel that library writers should have a simple and straightforward way to mark certain features as falling outside of the usual SemVer rules.

Needless to say, everything I argue here goes equally for #[deprecated]. Third-party libraries really do need some way to deprecate APIs, too.

Anyway, thank you for patiently listening to this long argument in favor of library-level #[unstable] and #[deprecated]. :-) I know it must get tedious to read lots of posts about, "Hey! Where did my feature go?" And I really appreciate the work the core team is putting in to make this release happen.

[aturon]

@emk

Thanks for your clear articulation of what you'd like to see as a library author. I definitely see where you're coming from -- I am very sympathetic to wanting stability attributes in the Cargo ecosystem.

It's a feature we definitely want to provide at some point. But we don't necessarily want to just keep the old stability system alongside the release channel system for Rust -- that would give two very different meanings to stability attributes. We also want to gain some experience using release channels for Rust itself before moving forward with something for the libraries.

In short, I think we're likely to provide these facilities for external libraries in some form in the future, but we want to take our time to get the design right. In the interim, once plugins are stabilized it should be possible to build something like our existing stability system externally.

I hope you can understand where we're coming from here -- as with so many feature requests, we'd really like to offer it, but we need time before we can commit to a design, and there is a lot of other stuff in flight right now.

[emk]

Thank you for your kind response, @aturon! I'll try to figure out the design disconnect one last time, offer to prepare a PR, and then drop this issue. :-)

But we don't necessarily want to just keep the old stability system alongside the release channel system for Rust -- that would give two very different meanings to stability attributes. We also want to gain some experience using release channels for Rust itself before moving forward with something for the libraries.

I think this is where the disconnect is happening. The std crate needs a pretty sophisticated design:

1. A fine-grained way to enable some experimental features without enabling all of them.
2. Release channels.
3. Some way to automatically promote crates from a nightly compiler to a stable compiler when features are accepted.

As a library author, I need:

1. Some way to say, "Hey, be careful with this, it's not stable yet."
2. Some way to say, "Hey, this is going away at the next x.0.0 release."

I don't need fine-grained control over multiple unstable features, or release channels, or anything else like that. A really simple system with nothing but #[unstable], #[deprecated] and a couple of lint warnings would cover at least 95% of the use cases for libraries in the crates ecosystem. At least in theory, this could be implemented today: It's distinct from what std is doing; it's ultra-simple to design; and it's probably not much code.

Now, given the realities of getting 1.0 out the door, I understand that it might be necessary to remove the simple versions of #[unstable] and #[deprecated], and not provide them to crate authors until a more complex replacement system can be designed and thoroughly tested in the real world. But I would personally prefer to keep a simple-but-stupid system for library authors that covers the 95% case, instead of waiting for a really mature design that handles all kinds of exotic (for library writers) use cases.

Anyway, I'd be happy to contribute code, if that's the primary thing needed. But if there's no consensus in favor of an ultra-simple #[unstable] for libraries, I can just resort to using notes in the documentation for the time being, and hack up a standalone lint if that ever becomes possible on the stable channel.

Anyway, thank you to everybody for listening to this plea. And I'll leave this alone from here on, unless somebody asks for a PR or something. :-)

[brson]

I've pushed a new revision that does a few minor things:

• It renames the attributes to 'staged_' to reserve the 'nice' names for future use, and explicitly mentions that the current names will have no semantic meaning (but will not be warned about). The ugly names are fine since they're just for in-tree use.
• It emphasizes that the automatic version detection scheme is fairly speculative.

[Valloric]

It emphasizes that the automatic version detection scheme is fairly speculative.

I appreciate the recent changes made here, but the proposed system still seems like lots of unnecessary complexity that will end up shooting users in the foot. I've asked this before without receiving an answer, so I must ask again: the language itself is a versioned dependency much like any other library, so why is it being treated differently? Why aren't we asking the user to always explicitly state which version of the language their library requires? There should be no "shoot yourself in the foot" option where cargo guesses for you. I'm fine with cargo verifying that the version you stated as a minimum actually works (if you say stable but are using feature directives, cargo should complain) and even cargo recommending a version based on its analysis (while stating that the recommendation is a best-effort guess and shouldn't be taken as gospel), but it really should always be leaving the final decision to you.

This is the simplest approach and the one that's least likely to lead to nasty surprises.

[wycats]

Why aren't we asking the user to always explicitly state which version of the language their library requires? There should be no "shoot yourself in the foot" option where cargo guesses for you. I'm fine with cargo verifying that the version you stated as a minimum actually works (if you say stable but are using feature directives, cargo should complain) and even cargo recommending a version based on its analysis (while stating that the recommendation is a best-effort guess and shouldn't be taken as gospel), but it really should always be leaving the final decision to you.

The approach I had in mind wasn't that Cargo says "this will definitely work on Rust 1.3", but rather "this will definitely not work before Rust 1.3". The idea is that if people leave out the version number, at least we can give a useful error message to people using the library on versions of Rust we know for sure won't work.

[Valloric]

@wycats Thank you for the explanation. That's a lot more reasonable.

The idea is that if people leave out the version number

What I'm saying is don't make that an option. You must state the language version number when you upload a library to crates.io. No confusion possible. If you leave out the version number, cargo complains saying you must, and tells you it has detected a certain minimum version based on what you used in the code. Say, it detects 1.2.1 as a min version. You on the other hand know that that version has a bug in a std library that affects your code, and you pick 1.2.2 as the min language version.

If you pick 1.2.0, cargo complains that the version you picked can't compile your code (based on its analysis; it should even tell you why).

[wycats]

What I'm saying is don't make that an option. You must state the language version number when you upload a library to crates.io. No confusion possible. If you leave out the version number, cargo complains saying you must, and tells you it has detected a certain minimum version based on what you used in the code. Say, it detects 1.2.1 as a min version. You on the other hand know that that version has a bug in std library that affects your code, and you pick 1.2.2 as the min language version.

We could do this, but I'm not sure how this would be an improvement. Most of the time, people would just blindly copy in what Cargo tells you to use, so why not just have us insert it automatically?

[Valloric]

We could do this, but I'm not sure how this would be an improvement. Most of the time, people would just blindly copy in what Cargo tells you to use, so why not just have us insert it automatically?

Because it forces you to think about it. By doing it automatically, you're teaching people to blindly trust the version cargo spits out. By not doing it automatically, you make it explicit that cargo is making an (educated) guess.

If you're aiming for your library to be compatible with 1.1, but cargo tells you it detects 1.2 as a minimum, you know you messed something up and you know it before publishing the library.

[reem]

I agree completely with @emk and would be very sad if the complicated and very niche needs of the standard library for versioning ruined the system for the rest of us. The std library should just use another set of attributes which could optionally be exposed to library authors and there should remain a simple set of attributes for setting stability of library APIs.

[aturon]

@reem To be clear, everybody wants the stability attribute system to be useable in the crates.io ecosystem. The problem is that the current system has an extremely coarse-grained "opt in" where you have to accept unstable APIs from any sources, rather than saying only that certain unstable APIs are OK with you. We'd like to change this before committing to the design.

I'm curious whether you agree that this is a problem with the design.

It may be that using a feature naming system in crates.io is the right way to go; or maybe an item-based opt-in. It's not clear yet. But we plan to look into this ASAP.

Also, we discussed the possibility of letting the attributes be used but not yet linting against them, which would at least retain a way to signal stability without committing to the blanket opt-in.

[alexcrichton]

This RFC has had some pushback against removing the stability attributes for this RFC, but I believe that @brson has addressed many of these concerns by renaming the attributes to their less appealing counterparts (prefixed with staged_). One of the primary reasons for removing them as-is for now is that the opt-in mechanism isn't something that we'd like to stabilize as part of the language today, and we certainly plan on pursuing other vectors for a stability-attribute-like system, perhaps through Cargo. The removal of stability attributes as-is is likely to be quite temporary!

Additionally, @Valloric has pointed out that the "Cargo version detection" mechanisms may be too ambitious, but with @wycats's clarification and @brson's update the wording around this has been toned down significantly. As @wycats points out, the precise implementation in Cargo can develop over time.

As a result, we've decided to merge this RFC as-is. Thanks again for the discussion everyone!

[alexcrichton]

Tracking issue: rust-lang/rust#20445

[l0kod]

Maybe out of scope but rust-lang/rust#18815 should be considered too.

[quantheory]

Though this was merged while I was on vacation, I spilled a lot of words before that and want to point out a couple of final things:

• The revisions of @brson and explanation from @wycats were very helpful; I am pretty much on board with the "detection" proposal now (especially since further developments are now up to Cargo).

• I'm somewhat reassured by @alexcrichton's explanation. I'm also happier that the more ergonomic/mneumonic "unstable"/"stable"/"deprecated" names have been freed up for either a future version or plugin use. But when we get closer to stabilizing some kind of interface for plugins, this needs to be revisited, I think.

  In particular, it would be very nice to be able to tweak some aspects of rustdoc styling/formatting with directives. (An obvious example is the current color-coding, where stable is green and deprecated is purple, but regular/italic text or crosshatching, for the visually impaired or colorblind or for B/W printing, would be neat too.)

[daira]

I like this a lot, I think I'll use it in my own language, Noether (not necessarily tied to the "train" release model).