Color API Thoughts

[cart]

As the color conversation is "hot" again and animation apis are starting to move forward, it makes sense to take a final pass over the Color api.

Please note that this initial message is meant to kick off a conversation, not dictate a final design.

What are the goals for Color as a type?

1. Provide stable/lossless read/writes for color definitions within a given space (reading and writing the same H value to an hsla-represented color shouldn't drift due to conversion)
2. Be a "user facing" color type that supports popular color representations (rgb (srgb), hsl (srgb), linear)
3. Ergonomic to use
4. Support color math operations, especially interpolation
5. Free of surprises

Ive shown that (1) is in fact a "problem" here by adding a palette adapter for sprite colors (with linear srgb as the "base type" and HSL as the type converted to/from each frame). However after a few frames of precision loss the color stabilizes on a final value. The precision loss is "small" (at least in this specific case) and in practice seems like it won't manifest visibly in a meaningful way. I'd want to prove that globally before assuming thats true though. What it does absolutely prevent is things like if color == Color::RED because setting a color to red would immediately "drift" to something "approximately full red". Clippy would yell at you for float comparison ... but in practice i think this type of comparison in this context is both fine and useful. Floats are stable for specific values ... it's operations that introduce problems.

Should we use an external library for colorspace conversion? If so which one?

I personally don't think we need one. The conversion algorithms are well documented / straightforward and none of the implementations I'm aware of meet all of the requirements above. The options I'm aware of fall into one or more of these categories:

1. Completely separate types for each representation and therefore are incompatible with the first requirement above (ex: palette)
2. Use a single untyped vec3/vec4 to store components, then store the type information separately (ex: kolor, colstodian)
3. Introduce significant type complexity (ex: palette's types are hyper generic ... which is cool and flexible but not something I want our users to think about)

I'm mostly against (2)-style apis for aesthetic reasons. For a "public" monolithic color api, it feels wrong (and confusing) to store the H component in HSL and the R component in RGB inside the same vec3.x field. Especially when enums can give complete type safety and clarity without munging things together. There are valid reasons to use an untyped vec (mathy reasons like "color space conversions can be represented as matrix transforms" and performance reasons like "we can more easily use simd ops"), but none of those are worth the clarity cost to me or feel particularly relevant to our public color api. With enum representations we can use Rust features like pattern matching (if let Color::rgb { r, .. } = some_color) and users looking at the type can immediately understand how it works in a clear, magic-free way.

Theres also the concern that we can't easily tweak types in upstream crates. Needing to bug maintainers to add support for traits like Reflect or Component isn't fun for anyone.

I'm very open to adding opt-in support for conversion to/from external library colors to our Color type, but I don't think we should be adopting one of their types as our user-facing Color type.

What should the developer UX the Color type look like?

• Define colors in your preferred format Color::rgb(1.0, 0.5, 0.2), Color::hsl(0.5, 0.8, 0.2), Color::rgb_linear(1.0, 1.0, 1.0). Internally these are represented as an enum (ex: Color::Rgb { r: 1.0, g: 0.5, b: 0.2 }).
• Use "nice" names where possible. rgb == srgb. Most users shouldn't need to know or care about the distinction. Experts that do care about the distinction can learn the defaults.
• When performing color math operations, define the target "output" representation. If the representation doesn't match the inputs, they are converted to that representation.
  • lerp(Color::Rgb(...), Color::Hsl(...), 0.5, ColorSpace::Rgb)
    • output is a Color::Rgb representation
  • add(Color::Rgb(...), Color::Hsl(...), ColorSpace::Rgb)
    • output is a Color::Rgb representationd
  • in practice these could be methods. Ex: some_color.add(Color::RED, ColorSpace::Rgb)
• (Not clear: discuss this): Choose sane defaults for ergonomic apis: some_rgb_color.lerp(Color::Hsl {...}, 0.5) would default to HSL (the "right hand side") because the color should end on the exact Hsl value when it hits 1.0. Whether position 0.0 in the interpolation should be RGB (left hand side) or HSL (right hand side) is an arbitrary choice we'd need to make. I'm personally biased toward setting it to RGB to maintain the "integrity" of the original position, but in practice I don't think this will matter much. The "flip" approach feels consistent with animating types like bools, which will need to "flip" their state at some point (ex: godot supports animating bools in their high level system, which sounds weird but is very useful in practice).
  • Adopting defaults makes animating color with "high level" bevy property animation straightforward.
  • This is more important (and intuitive) for lerping than for things like adding.

In practice, this is largely how things work already. The biggest change would be deprecating operations that do implicit color conversion (ex: multiplying and adding two Colors) and replacing them with more explicit apis.

Do we need an RFC for this?

I personally don't think so unless there is significant pushback here. I think we're very close to having our ideal api. But feel free to voice concerns.

[alice-i-cecile]

I'm on board with these.

Support color math operations, especially interpolation

Color interpolation is 100% the most important math operation.

Completely separate types for each representation and therefore are incompatible with the first requirement above (ex: palette)

I don't like this much.

Use a single untyped vec3/vec4 to store components, then store the type information separately (ex: kolor, colstodian)

I strongly dislike this.

Introduce significant type complexity (ex: palette's types are hyper generic ... which is cool and flexible but not something I want our users to think about)

This is unsurprisingly the sacrifice I'd personally be most willing to make, but I think it's the wrong choice.

Whether position 0.0 in the interpolation should be RGB (left hand side) or HSL (right hand side) is an arbitrary choice we'd need to make. I'm personally biased toward setting it to RGB to maintain the "integrity" of the original position, but in practice I don't think this will matter much.

Yeah, no strong feelings here. Either would be fine; the important thing is that the data representation is correct when at rest. I suspect that doing the conversion immediately will result in nicer properties, since presumably the target space is the one whose metric you care about.

I'm very open to adding opt-in support for conversion to/from external library colors to our Color type, but I don't think we should be adopting one of their types as our user-facing Color type.

Yes please. We should have optional features for conversions into all of the big color crates. I don't want end users to end up having to newtype their colors to get conversion.

[james7132]

Color interpolation is 100% the most important math operation.

By proxy, this extends to color blending as well for animation. If we go with a.interpolate(b, op_space) -> op_space, we either will need to design our animation blending to either:

• Support parameterizing blending on the op space (not optimal, as this requires plumbing a lot more through)
• Fixing a single colorspace for animation blending purposes (limits usability of the feature)
• Baking it into the type system via separate colorspace types. (a lot more workable, but dumps the colorspace choice into userspace)

[cart]

I'm curious what your thoughts are on my "(Not clear: discuss this)" bullet point, which outlines another option.

[james7132]

I personally think a single fixed default for color blending is probably sufficient, as most developers and artists are used to that kind of behavior in gradient editors, etc. Easiest would probably be SrgbaLinear, though that might be a bit surprising for developers typically used to animating color to operate in Srgba (mostly 2D devs). Forcing this choice on animators/game designers when animating seems counterproductive in general, though I can see many edge cases where it is desirable for either the programmer or artist to explicitly force a colorspace. Ideally this would be a decision made by the programmer or artists that is transparently presented in the editor of some kind.

[mockersf]

Do we want to keep an internal representation of the color close to what the gpu is expecting? If I set up a color as HSL, as the gpu is expecting a color in RGB the conversion will happen on every frame. If we keep the converted value inside the color type, and update it only when the color change in HSL, we can reduce the number of times we do the conversion.

[alice-i-cecile]

Right, we could have a private cached value for this conversion. I like it, but it's really something I'd want to see benchmarks for.

[superdump]

I’ve thought about doing this for matrix inverses too but don’t you need to query &mut Color then, which messes up change tracking?

[alice-i-cecile]

We can easily implement a change detection bypass; it was just waiting on consensus that it's a good idea.

[cart]

The biggest downsides I see are:

1. (If the gpu format is stored on the same component) No more matching directly on internal representations. They would need to be hidden behind accessors to ensure we keep everything in sync.
2. If sync-ed directly on mutation, a color that is modified many times per frame would be converted many times per frame. The current "only sync once at the end of each frame" sets an upper bound.
3. If it's a separate component, then it will be out of sync for some period of time. Additionally, color is usually a field on a bigger type, so each case would need its own synced version.
4. More memory usage.

Another solution to consider (that is admittedly more of a research area): I do want to explore only extracting changed components / resources and re-visiting entity lifecycles across apps. I suspect we could make this work nicely and win a bunch of perf back.

[superdump]

2. I was thinking it would be cleared on mutation and updated if necessary on access. I'm going to talk about view matrices - so if you mutate the view matrix, it would clear the inverse. And then when you try to access the inverse, if the inverse is there, you use it, if not, you update it and then use it. And I think this kind of caching of calculated values is a separate question to specifically caching GPU representations of things.

In response to your last paragraph: that sounds tasty.

[superdump]

The Discord discussion about operations on colours was color.op(other, operation_space) -> operation_space, if I understood correctly. Both color and other would be converted to the operation_space, and the operation would be applied in that space and the output produced. Your suggestion differs from this. Why?

[superdump]

And, given this, your lerp API behaviour suggestion where if the spaces of color and other differ, it would flip the space of the output from colour to other… that’s inconsistent API-wise and could very well introduce a discontinuity/error at the flip. Which seems to be something you want to avoid in the drift case.

[cart]

What I suggested seems identical to what we discussed on discord and what you just described:

(pasted in from the main post above)

And, given this, your lerp API behaviour suggestion where if the spaces of color and other differ, it would flip the space of the output from colour to other… that’s inconsistent API-wise and could very well introduce a discontinuity/error at the flip.

That suggestion is certainly my most controversial one, but I still stand by it. The discontinuity feels like a feature to me. Interpolating between Enum::A(A) and Enum::B(B) has no inherent meaning. From my perspective, position 0.0 should exactly equal Enum::A(A) and position 1.0 should exactly equal Enum::B(B). 0.0 means you haven't started moving "into" Enum::B yet.

If we define the interpolation to be "between A and B converted to some target C" then we are free of discontinuities. But that also throws away lossless representations of the start and end state. I personally think preserving user inputs losslessly is the better default behavior (even if it introduces a discontinuity). It also happens to be much easier to fit into a generic interpolation system because it doesn't require parameterizing the interpolation (with the target type). However supporting both approaches also seems reasonable to me.

Which seems to be something you want to avoid in the drift case.

Drift isn't a problem here because start and end values will both be stored in the interpolator (along with the duration and other metadata). The final value is just the "output".

[superdump]

What I suggested seems identical to what we discussed on discord and what you just described:

Yes, you're right. I was just thrown off by the lerp suggestion.

And, given this, your lerp API behaviour suggestion where if the spaces of color and other differ, it would flip the space of the output from colour to other… that’s inconsistent API-wise and could very well introduce a discontinuity/error at the flip.

That suggestion is certainly my most controversial one, but I still stand by it. The discontinuity feels like a feature to me. Interpolating between Enum::A(A) and Enum::B(B) has no inherent meaning. From my perspective, position 0.0 should exactly equal Enum::A(A) and position 1.0 should exactly equal Enum::B(B). 0.0 means you haven't started moving "into" Enum::B yet.

If we define the interpolation to be "between A and B converted to some target C" then we are free of discontinuities. But that also throws away lossless representations of the start and end state. I personally think preserving user inputs losslessly is the better default behavior (even if it introduces a discontinuity). It also happens to be much easier to fit into a generic interpolation system because it doesn't require parameterizing the interpolation (with the target type). However supporting both approaches also seems reasonable to me.

I see what you're saying. And I think that if the discontinuity creates a visible jump in the interpolated colour that was otherwise interpolating smoothly, it will be something that people will not want to use in those cases. See what I mean? There is also the problem of colour spaces, to use set theory terminology, have non-empty complements. That is, given colour space A and B, not all colours in A can be represented by B and not all colours in B can be represented by A. Presumably then the interpolation needs to happen in a fake (I'm thinking something like values outside the normal range, say less than 0.0 or greater than 1.0 or something) / actual space that is a superset of both A and B. Note that I don't really know what I'm talking about exactly with colour spaces. I still need to fill my head with that knowledge. :)

Which seems to be something you want to avoid in the drift case.

Drift isn't a problem here because start and end values will both be stored in the interpolator (along with the duration and other metadata). The final value is just the "output".

This was not a comment on lerp also being a problem for drifting, rather a comment on my perception that you expressed a desire for losslessness, which I interpret as a desire to avoid unexpected behaviours in the output values.

[james7132]

Could it make sense to have a enum AND colorspace types? For example:

pub enum Color {
  Srgba(Srgba),
  SrgbaLinear(SrgbaLinear),
  HSL(HSL),
  ...
}

pub struct Srgba(Vec4);

impl From<Srgba> for Color {
    ...
}

impl From<Color> to Srgba {
    ...
}

This adds notable complexity internally to the color API, but provides the same level of flexibility in userspace and stringent consistency where we don't need that flexibility (i.e. GPU submit, internal animation operations, etc). We can still have nice user-facing shorthands like Color::rgb(...), which will choose the right variant, but also ensure colorspace conversion boundaries enforced at the type system level.

[cart]

Yeah the biggest downside I can think of is a loss of ergonomic enum-matching:

// before
if let Color::Srgba { r, g, .. } = color {
}

// after
if let Color::Srgba(Srgba { r, g, .. }) = color {
}

You now need to double-define the type name. But I do also suspect that this type of pattern matching will be less common. Most people will probably just use color.r() and color.g() functions. But for cases that are trying to both check "internal color representation" and extract relevant values from it, I think losing ergonomic matching would be a usability hit.

[molikto]

I don't get why you want to have color.r(). color.as_srgb().r or color.as_lin_srgb().r makes it clear which color space the r is in

[cart]

I don't get why you want to have color.r(). color.as_srgb().r or color.as_lin_srgb().r makes it clear which color space the r is in

Thats kind of the point. "random 16 year old thats still learning how to program" shouldn't need to think about srgb vs lin_srgb, what the s means, or what linear vs non-linear means. I would assert that the vast majority of entry level gamedevs dont need to know the difference. I would also assert that a sizeable portion of "professional" gamedevs couldn't tell you the difference. In the context of user code, color.r() should mean "srgb" in 99.9% of cases. We should make that easy and unsurprising to use.

[viridia]

I started another ticket about color APIs: #10986 . Apologies for the duplication; this was a spin-off from the Color::lerp() discussion and I didn't realize that this ticket existed.

I've put together a small prototype library here: https://github.com/viridia/quill/blob/main/crates/bevy_color/src/lib.rs

Key aspects of this library:

• It lives in it's own crate, which I call bevy_color.
• Color spaces are distinct types, which currently support: Srgba, LinearRgba, Hsla, Oklaba.
• Conversions between color spaces are handled via From: let linear: LinearRgba = Srgba::WHITE.into().
• There is an enum, ColorRepresentation which holds all of the various color spaces, but it should be used sparingly if ever. It's generally better for color spaces to be known at compile-time.
• Other than that one enum, there is no limitation on end-users implementing additional color spaces without having to modify Bevy.
• For cases where the color space is known at compile time, automatic conversions can be handled via parameters of type impl Into::<LinearRgba>.
• Colors can be converted into CSS4 string formats, e.g. color(oklab 12% -0.5 .71 1.0).
• Lerping of colors is provided by the Mix trait.
• It has a modest set of unit tests.

I considered adding additional traits such as Darken, Lighten, Saturate, Desaturate and so on, but these are pretty app-specific (mainly used by editors), and unless you are super concerned about performance it's easy to convert to HSL and back.

For the rare case where you have the wrong encoding (your image asset reader gives you SRGB types, but the actual image data is linear) there's a way to transmute without converting: LinearRgb::from_components(srgb.to_components()).

Note that the JavaScript 3D framework, three.js, makes it the user's responsibility to convert colors into the correct color space before calling framework APIs. Unfortunately, their solution is not type-safe, making it somewhat of a footgun; with Rust we can do better by ensuring the user does the conversion to the correct type. Still, it provides support to the idea that the right long-term solution for users is not to hide color spaces from them, but to educate them in their proper usage. Otherwise we're just sweeping problems under the rug.

[viridia]

My prototype library can now be found here: https://github.com/viridia/bevy_color