10 Challenges for Bevy UI Frameworks

[TimJentzsch]

In the past few months, there has been a lot of ongoing development in the Bevy UI space.
Some examples of active projects:

• native bevy_ui
• Cart's Bevy UI evolution proposal
• kayak_ui
• bevy_dioxus
• bevy-lunex
• cuicui
• belly
• bevy_egui

These projects cover vastly different approaches on how to design and manage your UI. This makes it increasingly harder to maintain an overview of the project's capabilities and to compare them among each other.

To this end, I propose a set of 10 UI challenges, which we can try to implement in each framework. This should give us a rough overview of what's easy and what's hard to do in which plugin, as well as giving us roughly comparable examples of implementation code.

I tried to select the challenges with these criteria:

• Cover a broad range of UI features.
• Relatively broad application in games (i.e., each challenge should be relevant to multiple different games).
• Reasonable in scope, small enough to be practical as example code.

However, I must make the following disclaimers:

• There will be more UI features that are not covered by these challenges.
• Not all of these challenges are equally relevant to a broad set of games.
• The example screenshots/videos don't need to be replicated exactly to keep the scope manageable. Especially the 3D scenes can be simplified a lot.

If you are a developer of a Bevy UI framework, feel free to take on any of these challenges, implementing them as an example to your library. Then you can create a comment on this post, linking to the code and ideally with a screenshot/video showcasing how it looks like.
If you want you can also add an analysis of which points were easy and which ones hard to implement in your framework!

Challenges

1. Game Menu

Motivation

Menus are needed for almost every game.
Here we can showcase basic input functionality like buttons, sliders, and dropdowns.

Acceptance Criteria

• Main menu with sub menus for audio and graphics.
• Simple buttons for option selection.
• Slider for volume.
• Dropdown for graphics quality (low/medium/high).
• Navigation possible with mouse, keyboard and controller.
  • Mouse: Separate styles for hover and press.
  • Keyboard/Controller: Separate styles for currently focused element.

2. Inventory

Motivation

Inventories are also important in a broad range of game genres.
This can be used to test drag and drop, as well as a combination of image and text elements.
Furthermore, a tooltip implementation is tested.

Acceptance Criteria

• Fixed-size grid, some spaces with items and some empty.
• Each item slot has an image of the item and the item count overlayed on the image.
• Items can be moved with drag and drop.
  • Both image and item count move along with the cursor while dragging.
  • The image and item count are not visible in the original position while dragging.
  • You can leave the bounding box of the inventory while dragging.
• A tooltip with the item's name is shown when hovering over an item.

3. Health Bar

Motivation

Again, widely used feature in multiple genres.
Uses world-space UI and an information flow from the game to UI.

Acceptance Criteria

• Simple 3D Scene with a character (sphere).
  • Can be moved around with WASD/arrow keys.
• A health bar and character name is anchored to the character in world-space.
• The health starts at 10 and decreases by 1 every second. The health should be stored and managed in Bevy ECS.
• When reaching 0 HP, the character should be despawned together with UI.

4. Responsive Menu

Motivation

A combination of responsive design, for games which need to save multiple display sizes and formats. And nine-patch UI, which is a common styling solution in games.

Acceptance Criteria

• A simple game menu, with buttons that use a nine-patch system for design (i.e., composed of images for the corners and middle segments) and an image to the right of the buttons.
• For normal screen sizes, the menu is centered in the middle of the screen
• For 400px width and lower, the buttons fill the screen width and the image is above the buttons.

5. Character Editor

Motivation

This example showcases how a 3D scene can be integrated in the UI as well as having an information flow from UI to 3D scene.
Additionally, we feature a simple text input and a scroll box.

Acceptance Criteria

• A UI on the right with a 3D scene of the character on the left.
  • The character can be simple 3D shapes.
• The UI is composed of multiple buttons to select options.
  • The selected option is highlighted.
  • There are too many buttons to fit in the box, so the box can be scrolled vertically. You can duplicate buttons or choose a small box size to simulate this.
• Changing the selection in the UI changes the 3D shapes in the 3D scene.
• On the top of the UI is a text field for the character name.

6. HUD

Motivation

Needed in many games.
Showcases how UI can be overlayed on top of 3D scenes and how it can be aligned in the different corners.

Acceptance Criteria

• Top left: Image (minimap).
• Bottom left: HP counter and HP bar.
• Bottom center: Game time and scores of both teams.
• Bottom right: Ammo counter and indicator.

7. Bug Report Form

Motivation

This example might not be as widely applicable, but showcases complex text input.

Acceptance Criteria

• A dropdown for the type of bug (UI/cosmetics/gameplay).
• A one-line text input for the bug title.
• A multi-line text input for the bug description.
• The text editing should have the following features:
  • Cursor, which can be moved with arrow keys and mouse click.
  • Text selection.
  • Copy/paste/cut with the usual shortcuts.

8. Scoreboard

Motivation

Another commonly used UI element, which features a grid layout and a mix of images and text.

Acceptance Criteria

• Grid with players as rows and avatar, name, kills (K), deaths (D), assists (A) as headings.
• Avatars should be images.
• Grid should be sized dynamically (i.e. to fit the names).
• Grid content (e.g. player names, K/D/A) should reside in the Bevy ECS.

9. Dark/Light Theme

Motivation

Having theming options makes it easier to adjust the design of the UI at later stages of the development, avoiding hard-coding the style properties.

Acceptance Criteria

• A simple UI layout (e.g. main menu).
• A button to toggle between light and dark theme.
• The theme should change all colors in the UI.

10. Design Specification

Motivation

Most of the other example feature on widgets and features, but of course styling is a major part of UI.
In this challenge, a design specification should be followed as closely as possible.

Acceptance Criteria

I didn't find a good open source design specification yet, please let me know if you find one!

[nicopap]

I really like the idea of a set of "game UI" examples to benchmark UI solutions. It's much better than 7GUIs as it's specialized for desktop apps, or even just the bevypunk example we've been going with for a while… Thank you for taking the time to write this down. I'll definitively try to implement them in cuicui.

Can we add an additional criteria on (2)? I would like to see tooltip on hover. It seems like a reasonable thing, and it's how it works in Minecraft.

[TimJentzsch]

Thanks for your feedback!
A tooltip is definitely a good idea, I forgot about that.
I added it to the inventory example.

Based on the feedback I received on Discord, I also added a scrollbar requirement for the character editor.

[databasedav]

here's my implementation for challenge 1 using haalka: https://github.com/databasedav/haalka/blob/main/examples/challenge01/src/main.rs

video

run the challenge by cloning the repo and doing

cargo run -p challenge01
# or using just
just example challenge01

what follows is some assorted information/thoughts about haalka, bevy ui, reactivity, etc.

haalka

• port of signals-powered rust web frontend frameworks https://github.com/Pauan/rust-dominator and https://github.com/MoonZoon/MoonZoon (which adds higher level abstractions/ergonomics on top of Dominator); the ported layers are quite thin and enabled me to rig bevy ui nodes with essentially the same signal semantics as MoonZoon, which I have quite a bit of playtime in, allowing me to bootstrap the framework and this solution very quickly
• lightweight, simply a thin reactivity layer on top on raw bevy ui nodes; additionally, because haalka's core abstraction RawHaalkaEl is generic over anything that impl Bundle, the API allows one to create arbitrary hierarchical reactivity graphs for any entities, not just bevy ui nodes
• minimal macros (one must use map_ref! if they need to react to signals of different types, but note this is an easy, high level macro), no DSL, no VDOM, fine-grained O(1) reactivity, and novel signal semantics that are foreign at first but extremely expressive and powerful once gotten used to (o wow kinda like rust)

bevy ui + reactivity

• there's nothing wrong with bevy ui, it's simply bevy's DOM layer, what's missing is ergonomic reactivity semantics on top of the DOM
• imo, operationally, rigging up reactive ui layouts starts and ends at ergonomics; one wants to draw reactive nodes onto the screen and they need a painless way to express the reactivity in code; signal semantics offer downright enjoyable ergonomics entirely in idiomatic rust
• regarding ecs/ui state sharing, there's three classes of state in this ecs/ui world:
  1. only used by ecs: reactivity is still relevant here because in the editor we want all components to be interactive; we can support this in haalka with a derive macro that recursively creates reactive analogs for all components and gets updated through the ui with .component_signal
  2. only used by ui: just use reactive primitives wherever needed
  3. used by both: if one needs to update the ecs world reactively from the ui world, just use the async_world() (provided by https://github.com/dlom/bevy-async-ecs, basically an async command queue) that haalka uses internally for all reactivity; if they want to update the ui world from the ecs world, either wrap the component's data with a reactive primitive e.g.Mutable or update one indirectly with a system

challenge 1

• i'm pretty happy about the solution, some holes were identified but i now think one should be able to express any layout they want, including things that are difficult/require hacks on the web like global z index management
• the easy (note that i'm not filtering out my bias here, i have significant playtime with these semantics so this difficulty evaluation is more to say that "these patterns are easy", not that a beginner could reliably execute them):
  • reusable, composable (headless?) widgets: almost every widget in the menu is just modifying or composing the Button widget
  • reactivity: signals make reactivity easy, and fun too
  • layout/alignment: this is where (my port of) MoonZoon's higher level element wrappers El, Column, Row, and Stack (and soon Grid) come in super handy, siblings and content can always get to where one needs them to be; want the x button to layer on top of the menu and be in the top right corner? simply declare it
• the hard:
  • focus management: i had to manage focus manually to handle keyboard/controller inputs; it ends up being pretty clean but not so friendly to consume, but i do think i should be able to emulate HTML tabindex (which i didn't even realize existed until today, thanks @viridia) to provide a much better high level api
  • hover event bubbling: i had to manually order some pointer event systems in order to reliably get the hover bubbling behavior i wanted
• the holes:
  • sometimes the background/border color of buttons desync: e.g. a button will stay in a pressed/hovered state even though it no longer is, but only for exactly one of background and border color, i've never seen both, but maybe that's just because the stars haven't aligned; i have yet to debug this but i'm guessing it's future/task related
  • collections of varied widget types are unwieldy: Element's are not type-erased (they carry their Bundle around) so one needs to wrap them in something common and coordinate common trait consumption without dyn; compare this to the buttery ergonomics of when the collection of widgets is of a single type

please let me know what you think, i'm happy to answer any questions :)

[databasedav]

here's my implementation for challenge 2 using haalka https://github.com/databasedav/haalka/blob/main/examples/challenge02/src/main.rs

video

haalka_challenge02.mp4

run the challenge by cloning the repo and doing

cargo run -p challenge02
# or using just
just example challenge02

misc notes

• i'm very happy with this solution, the widgets required are far less diverse than that of challenge 1 so i was able to do everything in around 500 lines (excluding boilerplate); some cells have some extra behavior added on top like the crafting input/output, but otherwise it's just a layout of the same cell widget
• i ported MoonZoon's Grid abstraction to help with the layout here and it worked out well, although for some reason bevy ui needs a Display::Grid parent node for the Style.grid_template_columns to apply which i noticed is not required for HTML DOM nodes; it's not a massive deal, but it does restrict the trait bounds for Grid a bit (its Bundle requires impl Default), so i'm curious why this is required and if it could be removed
• signals made expressing the drag and drop behavior very "easy" e.g. here's how the crafting output reacts to the inputs

  inputs.signal_vec_cloned()
  .map_signal(|input|
      input.signal_cloned()
      // this says "retrigger" the outputter every time any of the input's
      // texture atlas index or count changes
      .map_some(|cell_data| map_ref! {
          let _ = cell_data.index.signal_ref(|_|()),
          let _ = cell_data.count.signal_ref(|_|()) => ()
      })
      .switch(signal::option)
  )
  .to_signal_map(|filleds| filleds.iter().map(Option::is_some).all(identity))
  .for_each_sync(move |all_filled| {
      output.set(all_filled.then(|| random_cell_data(&mut rand::thread_rng())));
  })

  i was able to figure it out very quickly thanks to my playtime with signals, but there's a lot going on here and there is definitely a learning curve before one can reliably write signals like this; that being said, i'm super interested in how one would accomplish this ergonomically using other frameworks, as this looks and feels extremely clean in my (very biased) eyes
• the hardest part of this one was managing the tooltip/drag and drop position transforms

all thoughts and criticism are welcome and appreciated :)

[databasedav]

@UkoeHB skill issue

[UkoeHB]

@UkoeHB skill issue

I want to read your comments. This is unprofessional and uncalled-for.

[databasedav]

@UkoeHB likewise

[colepoirier]

I really like how the implementation went. I recall you mentioned that there were a few difficulties around interaction and things getting stuck in states for challenge 1, did you experience any of that for this challenge as well?

[databasedav]

@colepoirier thanks for the continued support Cole 🙏 good question, i didn't see any of that in this example (and i can confirm that the issue is still happening in challenge 1), which leads me to believe it has something to do with the "hovered" and "pressed" event handlers not executing in the correct order consistently, since in this challenge, there are no background/border effects on "pressed", only "hovered"

thanks for reminding me about this, i'm pretty sure a .configure_sets(PreUpdate, (pressed_system, hovered_system).chain()) should fix it 🙏

[wilk10]

Thanks for starting this, it seems like a great idea and I look forward to more submissions.

I am looking at this from a user perspective, not a Bevy dev. There has been only 2 submissions so far from the same dev, but it would be very beneficial for me as a user to have a clear separation between what is the code that the users will need to write vs what is the code provided by the framework. I know that for the evaluation from a Bevy dev perspective both are very important, but a clear demarcation between the two would help looking at the challenges from a Bevy user point of view too (eg: as I user I just want to know how to place and style a Button like I want, not how the Button works, as long as it works as expected).

Is this something that could be added to the general evaluation rules?

[TimJentzsch]

I envisioned these challenges to be implemented in the examples folder of the library.
In that case, the code would use the library exactly the same way as a normal user (i.e. app developer) would.

Does that work for you? I can explicitly add it to the post if you want :)

[wilk10]

Sure, sounds good. Thanks!

[databasedav]

here's my implementation for challenge 4 using haalka https://github.com/databasedav/haalka/blob/main/examples/challenge04/src/main.rs

video

haalka_challenge04.mp4

run the challenge by cloning the repo and doing

cargo run -p challenge04
# or using just
just example challenge04

misc notes

• haalka is a reactivity layer on top of Bevy ui, it doesn't fill in any features missing from Bevy's DOM, it simply provides semantics for coordinating reactive updates for entities/components; as such, implementing a nine-patch design system is outside haalka's scope (and lies at a lower level closer to the DOM), so i simply used https://github.com/Lommix/bevy_nine_slice_ui to fulfill that part of the acceptance criteria

• this challenge gave me the opportunity to provide a very concise demonstration of how haalka's reactivity can be added to external bundles, so i'll briefly highlight some of the code

  struct NineSliceEl<Bundle = NineSliceUiMaterialBundle>(El<Bundle>);
  
  impl_haalka_methods! {
      NineSliceEl => {
          NineSliceUiMaterialBundle => [
              style: Style,
              nine_slice_texture: NineSliceUiTexture,
          ],
      },
  }
  
  impl NineSliceEl {
      pub fn new(frame_signal: impl Signal<Item = usize> + Send + 'static) -> Self {
          Self(
              El::from(NineSliceUiMaterialBundle {
                  nine_slice_texture: NineSliceUiTexture::from_slice(
                      nine_slice_texture_atlas().clone(),
                      Rect::new(0., 0., 32., 32.),
                  ),
                  ..default()
              })
          )
          .on_signal_with_nine_slice_texture(frame_signal, |nine_slice, frame| {
              if let Some(bounds) = &mut nine_slice.bounds {
                  bounds.min.x = frame as f32 * 32.;
                  bounds.max.x = 32. + frame as f32 * 32.;
              }
          })
      }
  }

  the focal point is the .on_signal_with_nine_slice_texture function

  • syntactic sugar for .on_signal_with_component::<SignalType, NineSliceUiTexture>, conviently derived by the impl_haalka_methods! macro, where NineSliceUiTexture is a component of the NineSliceUiMaterialBundle provided by https://github.com/Lommix/bevy_nine_slice_ui
  • it means: "on every frame signal update, give me mutable access to the NineSliceUiTexture component and run this function on it"
  • the function simply adjusts the viewport into the nine slice texture atlas and is ported from the ui example of the nine slice library; the example drives the frame with a simple looping system, while this challenge allows the frame to be controlled by any usize signal, e.g. in some cases, the frame can be static

    NineSliceEl::new(always(3))

    or it can react to hovered/pressed state

    fn nine_slice_button() -> impl Element {
        let hovered = Mutable::new(false);
        let pressed = Mutable::new(false);
        NineSliceEl::new(map_ref! {
            let hovered = hovered.signal(),
            let pressed = pressed.signal() => {
                if *pressed {
                    2
                } else if *hovered {
                    1
                } else {
                    0
                }
            }
        })
        .with_style(|style| {
            style.width = Val::Px(100.);
            style.height = Val::Px(50.);
        })
        .hovered_sync(hovered)
        .pressed_sync(pressed)
    }

  one can use this pattern to add haalka-esque reactivity to any impl Bundle, not just Bevy ui nodes; just imagine the ergonomic benefits of reactively changing components or the entity hierarchy in cases where traditional synchronous ecs systems are unwieldy

• because child/sibling alignment/layout managment is encoded into the high level element structs that haalka provides (El, Row (x-stack), Column (y-stack), Stack (z-stack), and Grid), there is no even-higher level API for switching between them (this is also true for MoonZoon btw), e.g.

  enum Layout {
      Horizontal,
      Vertical
  }
  
  El::new()
  .layout_signal(
      .map(|layout| {
          match layout {
              Layout::Horizontal => Row,
              Layout::Vertical => Column,
          }
      })
  )
  .children(...)

  instead, one must rebuild the tree with the high level structs appropriate for the given layout e.g.

  El::new()
  .child_signal(
      layout.signal()
      .map(|layout| {
          match layout {
              Layout::Horizontal => Row::new().items(...),
              Layout::Vertical => Column::new().items(...),
          }
      })
  )

  note that if one does not need haalka's high level alignment management with Align, they can always just manage Style.flex_direction (and all other styling) directly e.g.

  RawHaalkaEl::new()
  .on_signal_with_component::<Layout, Style>(layout.signal(), |style, layout| {
      style.flex_direction = match layout {
          Layout::Horizontal => FlexDirection::Row,
          Layout::Vertical => FlexDirection::Column,
      }
  })
  .children(...)

  there's also nothing stopping one from editing the styling built-in to the high level structs (plz don't) e.g.

  Row::new()
  .with_style(|style| {
      // style.flex_direction = FlexDirection::Row;  // built-in !
      style.flex_direction = FlexDirection::Column;  // don't care lol
  })

  although this would have weird effects on child/sibling alignment/layout

  in practice, this isn't a huge deal because layout changes are infrequent, but special considerations must be made in more advanced use cases, e.g. what if the tree is rebuilt due to a layout change while a dynamic async task (e.g. clicking a button started an http request) attached to some entity is running? then one must decouple the lifetime of the async task and the entity (this one is easy, just use Task.detach())

  for reference, in this challenge, the "one must rebuild the tree" part from above looks like this

  raw_el.child_signal(
      width
          .signal()
          .map(|width| width > 400.)
          .dedupe()
          .map_bool(horizontal, vertical),
  )

  where horizontal and vertical are fn() -> RawHaalkaEl<NodeBundle>

• updating the width shown/used in the ui from the ecs is easy: just store a reactive f32 in a resource

  #[derive(Resource)]
  struct Width(Mutable<f32>);

  and update it with a system

  fn on_resize(width: Res<Width>, mut resize_events: EventReader<WindowResized>) {
      for event in resize_events.read() {
          width.0.set(event.width)
      }
  }

[mahulst]

Nice work, again. In the on_resize method you are using a Resinstead of ResMut, while still mutating the value in the resource. Can this cause issues in combination with the bevy scheduler?
So, are the Row and Column just Nodes with the flex properties set? Or is there something more going on?

[databasedav]

@mahulst

In the on_resize method you are using a Res instead of ResMut, while still mutating the value in the resource. Can this cause issues in combination with the bevy scheduler?

no and yes 😅

the reactive primitives of futures-signals use interior mutability to (among other things) allow mutating the value without &mut, and is internally just an FRP signal abstraction on top of a synchronous RwLock

no: if one has multiple systems with a Res mutable, even if they are run in parallel across threads, when acquiring the write lock via the call to .set, subsequent lock acquiring systems will simply block their respective threads until the write lock can be acquired

yes: but this doesn't mean deadlocking isn't possible, consider the following

#[derive(Resource)]
struct Mutable1(Mutable<usize>);

#[derive(Resource)]
struct Mutable2(Mutable<usize>);

fn s1(mutable1: Res<Mutable1>, mutable2: Res<Mutable2>) {
    mutable1.0.set(1);  // mutable 1 lock dropped inline
    mutable2.0.set(2);
}

fn s2(mutable1: Res<Mutable1>, mutable2: Res<Mutable2>) {
    mutable2.0.set(2);  // mutable 2 lock dropped inline
    mutable1.0.set(1);
}

this wouldn't deadlock because the write locks acquired via .set are dropped within the .set function, but what happens when one acquires the lock and doesn't drop it? e.g.

fn s3(mutable1: Res<Mutable1>, mutable2: Res<Mutable2>) {
    let lock = mutable1.0.lock_mut();
    mutable2.0.set(2);
    // mutable 1 lock dropped here
}

fn s4(mutable1: Res<Mutable1>, mutable2: Res<Mutable2>) {
    let lock = mutable2.0.lock_mut();
    mutable1.0.set(1);
    // mutable 2 lock dropped here
}

if these systems ran in parallel across threads, s3 and s4 would acquire write locks (.lock_mut) to Mutable1 and Mutable2, respectively, and then deadlock because the following calls to .set would attempt to acquire a lock to the mutable currently locked by the other system

so it's up to the user to ensure that such deadlocks don't occur, as it's a higher level logical bug; this sorta thing makes me realize that the "fearless concurrency" that rust offers does not mean that writing concurrent code is easy, but that developers can shift their focus from ridiculous memory and lifetime issues to logically preventing deadlocks in massively parallel code (which is also basically what Bevy does?)

So, are the Row and Column just Nodes with the flex properties set? Or is there something more going on?

yes, flex properties are set and there's a bit more going on: the alignment of the node and its children are managed differently depending on which high level struct is used; check out the Alignable and ChildAlignable traits to see how

the meat of ChildAlignable is basically providing a reactive emulation of the CSS Child Combinator (what a name lol) so that the alignment of the child depends on the type of the parent; and one can use this same pattern to define their own high level elements that have particular effects on their children, e.g. a "tree" element where each "branch" child reacts to the health of the tree (or something)

[ycysdf]

My Challenge 1 Implementation for using Rxy UI

https://github.com/ycysdf/rxy_ui/blob/main/examples/game_ui_challenges/game_menu.rs