XR support with OpenXR backend

[zarik5]

This is an in-progress implementation for XR integration into bevy. It is incomplete because it does not address the interoperability with bevy_render and bevy_wgpu, which is postponed after the rendering rewrite.

Related RFC.

[bjorn3]

How does this relate to #2166?

[zarik5]

@bjorn3 As I wrote in the RFC, @blaind's work is not easy to integrate into bevy. It touches too many crates and is is not very organized IMHO. @blaind is aware of this: they started working on my same direction by working on exposing raw gfx-hal objects in wgpu. While their PoC focuses on OpenXR, my work considers OpenXR only as a backend and exposes a XR backend-agnostic API.

[NathanSWard]

I just left some first pass comments for stuff that immediately stood out to me.
If you'd like me to do a more in depth review just lmk (or if you want me to wait for the PR to be in a more stable state that's cool too!)

[NathanSWard]

When the new labels get updated, we need to remember to tag this with the XR label

[zarik5]

@NathanSWard Thanks for the feedback! For now I would prefer some feedback on the general shape of the API. I tried to make a compromise between ergonomics and abstraction layer "thickness", but there are always opportunities for improvement.

[zarik5]

I have a few questions about my proposed API, mainly about conventions.

1. Is it ok to panic if XrConfig is missing? Or should a default be used instead?
2. Is Res<XrState> a good pattern? This is a ways to have a direct access to OpenXR, but it sort of goes around the ECS.
3. is it ok to group controller button events into a single struct GenericControllerPairButtons? I did this to avoid proliferation of structs, but this lead to structs like BinaryInput where it must store both the state and if the event occurred for the specific button.
4. Is it ok to use events for input? OpenXR input system is poll-based and right now inputs are polled every update loop regardless if they are used or not.

[NathanSWard]

4. Is it ok to use events for input? OpenXR input system is poll-based and right now inputs are polled every update loop regardless if they are used or not.

I would say yes. As the way we currently handle input events, we push them through an Events.
However, I would also add that we should add an Input<XrInput> or something of the sort, since that is usually much nicer to use than directly using EventReader EventWriter

[NathanSWard]

1. Is it ok to panic if XrConfig is missing? Or should a default be used instead?

Based on the behavior of other config like structs, we probably want to use a default if none is specified.
For example, I would look at the WindowDescriptor lifecycle for an example.

[zarik5]

@NathanSWard The has been a discussion in #xr on Discord, to change the current input system to match more the OpenXR API, in particular to bring remappable actions. This would probably lead to switch from a event based to poll based input system.

[NathanSWard]

The has been a discussion in #xr on Discord, to change the current input system to match more the OpenXR API, in particular to bring remappable actions. This would probably lead to switch from a event based to poll based input system.

Well a poll based input system is sort of the purpose of ecs systems (as they get polled every tick).
It think there would have to be a strong argument for having a new input API for only xr when bevy uses the event/Input<T> API everywhere else.

[NathanSWard]

Is Res a good pattern? This is a ways to have a direct access to OpenXR, but it sort of goes around the ECS

Well it depends on what XrState is in charge of doing. We definitely have examples of globals like this (e.g. AssetServer TaskPool etc...).
However if XrState is in charge of everything, then that's more of an anti pattern.
We should try to use already existing bevy utilities or break XrState into smaller parts (think AssetServer, Assets<T>, AssetEvent<T> for example)

[zarik5]

Well a poll based input system is sort of the purpose of ecs systems (as they get polled every tick).
It think there would have to be a strong argument for having a new input API for only xr when bevy uses the event/Input API everywhere else.

I agree, there is the opportunity to make it more integrated.

Well it depends on what XrState is in charge of doing. We definitely have examples of globals like this (e.g. AssetServer TaskPool etc...).
However if XrState is in charge of everything, then that's more of an anti pattern.
We should try to use already existing bevy utilities or break XrState into smaller parts (think AssetServer, Assets, AssetEvent for example)

There is the opportunity of splitting XrState but the problem was actually giving a name to the different objects. I'll give it a try anyway.

[zarik5]

@NathanSWard Is it ok for the user to manually add a bevy plugin? For what I want to do, a simple resource like XrConfig cannot be used. Basically I want to do something like this:

App::build()
    .add_plugin(XrActionsPlugin::new()
        .register_binary_input(MyButton, "my_button")
        .register_vec2_input(MyJoystick, "my_joystick")
    )

which registers a XrBinaryInput<MyButton> and XrVec2Input<MyJoystick> (which should behave similarly to bevy_input::Input<T>). XrActionsPlugin needs to be a plugin because it needs access to AppBuilder to register resources and systems. MyButton and MyJoystick cannot be added in a struct because I want to add multiple different user types at once; register_binary_input and register_vec2_input would use a type argument to construct the type-aware systems that update the inputs. XrActionsPlugin would also need to register a resource containing a registry of every action (just the string, not the user type) to be used for constructing and binding the backend actions.

[NathanSWard]

@NathanSWard Is it ok for the user to manually add a bevy plugin? For what I want to do, a simple resource like XrConfig cannot be used. Basically I want to do something like this:

I personally haven't seen uses of a builder API for a Plugin, but I like it a lot!
It should probably be named like:

App::build()
    .add_plugin(XrActionsPluginBuilder::default()
        .register_binary_input(MyButton, "my_button")
        .register_vec2_input(MyJoystick, "my_joystick")
        .build()
    )

However, yes, I do like this kind of configuration.

Take a look at derive_builder crate. This could be helpful for what you're doing :)

[zarik5]

I completely rewrote the interaction system. Now the API more closely matches WebXR instead of OpenXR. Input can now be read with XrButtons and XrAxes which work similarly to bevy_input::Input. XrButtons has 3 states (Default, Touched or Pressed) + a float value for each button. XrButtons and XrAxes replace the virtual controller; custom layouts are not supported anymore, but XrButtons and XrAxes provide a "union" between all vendor controllers inputs. OpenXR backend can't provide custom actions anymore, a default mapping for buttons and axes is provided but the user can override it. The API for polling poses for arbitrary times has been moved to bevy_openxr.

I added a bevy_webxr crate but I don't plan to fill-in the functionality yet. There are two problems: web_sys is a bit too low level and there should be a wrapper crate for XR Device API; web_sys make extensive use of Promises (including for the XR Device API) and mapping this to bevy seems impossible without blocking the main thread. Bevy should provide parallel async APIs, at least for plugins creation.

[zarik5]

This is a summary of the API so far. It is missing everything related to rendering (but most of the rendering stuff should be handled by the engine). I will copy this to the RFC once things settle.

This should be the lifecycle when the lifecycle API is implemented (requires #2432):

bevy_xr

API inspired by WebXR Device API and WebXR Augmented Reality Module.

Resource availability:

	Startup/WaitingForDevice/Exit	SessionCreated/Idle/SessionEnd	Resume/Running/Pause
XrSystem	✔️	✔️	✔️
XrInteractionMode		✔️	✔️
XrEnvironmentBlendMode		✔️	✔️
XrVisibilityState			✔️
XrProfiles			✔️
XrButtons			✔️
XrAxes			✔️
XrTrackingSource			✔️

• XrSystem: Used to enumerate and select the session mode.
• XrInteractionMode: ScreenSpace | WorldSpace, preferred mode for drawing HUDs (depends on if the XR device is head-mounted or handheld).
• XrEnvironmentBlendMode: Opaque (VR) | Additive (AR) | AlphaBlend (MR)
• XrVisibilityState: Hidden | VisibleUnfocused | VisibleFocused, makes the game aware of the interaction state.
• XrProfiles: Backend-specific strings used to select the controllers 3D model.
• XrButtonState: Default | Touched | Pressed
• XrButtons: Used to read state and events for controller buttons.
• XrAxes: Used to read the state of controller axes.
• XrTrackingSource: Used to get/set the reference space type and poll controllers/target rays/hand skeleton poses for the next vsync.

Other objects:

• XrSessionMode: ImmersiveVR | ImmersiveAR | InlineVR | InlineAR
• XrRigidTransform: position + orientation
• XrPose: rigid trasform + linear/angular velocity
• XrJointPose: pose + joint radius
• XrReferenceSpaceType: Viewer (head-locked) | Local (origin at the starting head location) | Stage (origin at the floor level)
• XrButtonType: Menu | Trigger | Squeeze | Touchpad | Thumbstick | FaceButton1 | Facebutton2 | Thumbrest, (WebXR ordering)
• XrAxisType: TouchpadX | TouchpadY | ThumbstickX | ThumbstickY, (WebXR ordering)
• VibrationEventType: Apply{...} | Stop
• VibrationEvent: vibration event type + hand type

bevy_openxr

The bevy_openxr crate exposes backend-specific resources. It is all optional.

Resource availability:

	Startup/WaitingForDevice/Exit	SessionCreated/Idle/SessionEnd	Resume/Running/Pause
openxr::Instance	✔️	✔️	✔️
OpenXrSession		✔️	✔️
Arc<OpenXrTrackingContext>			✔️

• openxr::Instance: Useful when working with the session.
• OpenXrSession: Wrapper of openxr::Session but made drop-safe (using a wgpu::Device handle). It is clonable and can be used concurrently.
• Arc<OpenXrTrackingContext>: Contains resources used for tracking. It can be used for polling poses at arbitrary times.

Other objects:

• OpenXrFormFactor: HeadMountedDisplay | Handheld, used to select the form factor when creating the plugin.
• OpenXrContext: Used to create the plugin. Creates everything needed to initialize graphics.
• OpenXrError: Loader(...) | InstanceCreation(...) | UnsupportedFormFactor | UnavailableFormFactor | GraphicsCreation(...)
• ButtonPaths, AxesBindings, VibrationBindings, PosesBindings, OpenXrProfileBindings, OpenXrBindings: Used to define the controller mappings.
• OpenXrTrackingReference: Contained in OpenXrTrackingContext, editable behind a RwLock.

Utility functions:

• openxr_pose_to_rigid_transform(pose) -> XrRigidTransform
• openxr_pose_to_corrected_rigid_transform(pose, reference, prediction_time) -> XrRigidTransform: used to account for pending recenterings.
• predict_pose(space, reference, prediction_time) -> Option<XrPose>: poll pose for arbitrary times. If the time is rejected by the runtime, returns None.
• predict_skeleton_pose(hand_tracker, reference, prediction_time) -> Option<Vec<XrJointPose>>: similar to predict_pose but for skeletal hand tracking.

[lizelive]

would this spec be compatable with webxr wrt hand tracking? esp XRHandJoint required layout. Also is their plan to support webxr?

[zarik5]

@lizelive The joint layout of this API corresponds to WebXR. The only difference with OpenXR is that OpenXR has a palm joint (which I remove). Also yes, the plan is to support WebXR. I would like if someone could write a nice WebXR wrapper on top of web-sys, since it is not very ergonomic working with all JsValues and dynamic casting.

[zarik5]

There have been some more discussions on the bevy Discord server about the interaction API. I want to settle this once and for all so I'll summarize again the options.

The objective is to choose a common interaction API to abstract both OpenXR and WebXR backends. The problem is that the two APIs are very different and conflicting with each other. One of the two APIs should be chosen and the backend corresponding to the other API will require some amount of glue code. This PR currently choses WebXR as the common API for interaction.
Below I use 🕸 for arguments in favor of WebXR API, 🟣 for arguments in favor of OpenXR API.

Problem

XR game controllers are overall similar to each other in function but the button layout/number and type of buttons differs considerably from vendor to vendor. Without the help of a higher level API, the user would need to limit the game for support to a specific type of controller or they need a lot of boilerplate to program compatibility with multiple controllers.

OpenXR 🟣 solution

OpenXR interaction API is based on actions and bindings. Actions is an abstract concept where the user gives a meaning to them in the context of the game. Buttons of different types of controllers can be bound to the same action; the user has to provide these bindings. The state of the actions can then be polled without worrying about the underlying source of the event.

WebXR 🕸 solution

WebXR reuses the Gamepad API. The Gamepad API maps different types of controllers to the same button layout, creating a sort of virtual controller. In the context of WebXR, the buttons of each type of controllers are mapped to each other deterministically, so it is possible to assign an index and name to each virtual controller button and expect it to be mapped to real controller buttons which have similar purpose/location. The Gamepad API bindings are already provided by controller vendors and are immutable.

Arguments

🟣 Actions are more expressive and higher level
🟣 In case bindings are not speficied for some type of controller, the XR runtime should remap the actions to the closest available controller.
🟣 On some PCVR runtimes, actions can be rebinded from the runtime dashboard.
🟣 Multiple buttons, even on the same controller, can be mapped to the same action. The same button can be bound to multiple actions, that can be used in different contexts (for example using the thumbstick to walk or drive, depending on the player state in the game).
🕸 The virtual controller provides a sensible default, 🟣 but it is a leaky abstraction, and doesn't consider future controllers that could have a very different button layout. One problem that is already present now is that the Gamepad API mappings cannot distinguish between closed hand and force-squeeze of the Valve Index controllers. 🕸 Currently, to add support for new types of buttons, WebXR just appends more to the list.
🕸 If using the WebXR API, on the OpenXR backend the virtual controller bindings can be customized at startup; 🟣 if using OpenXR as the surface level API, controller buttons could be rebinded at any moment.
🟣 If using the WebXR API, for the OpenXR backend, the default virtual controller bindings needs to be updated periodically to add new controllers that enters the market.
🟣 If using the WebXR API, all buttons of the virtual controller will be defined but inevitably some will not receive events because absent on some types of controller. Instead with OpenXR actions are bundled into action sets and the user can select to enable only one action set at a time.
🕸 For each button, while OpenXR needs the user to deal with touches and presses with separate actions, WebXR combines them into three states: Default, Touched, Pressed, which removes the not-touched + pressed state which is invalid.
🕸 Actions might be too high level and "overkill" most of the time.
🕸 A gamepad-like API is more intuitive to use if coming from a game-dev background using gamepads or mouse+keyboard. Instead with OpenXR, the user needs to learn about Actions and action sets.
🕸 WebXR is seemingly the easiest common abstraction to achieve between the two APIs. On the OpenXR backend, bevy can just reuse the same bindings profiles of the official WebXR input profiles repository
🕸 A OpenXR-like API implementation has been tried, using different approach, but no solution has been found that is both ergonomic and performant, even just to wrap the OpenXR backend itself.
🟣🕸 Both APIs provide a way of knowing which specific controller type is being used, so the user can use the correct 3D model.
🟣🕸 Both APIs can provide access to the other underlying API, but with secondary support/lesser ergonomics.
🟣 WebXR backend could be mapped to a OpenXR-like API by considering the Gamepad API as a single controller that can be mapped to the actions, 🕸 but to preserve identifiability of controller profiles for selecting the 3D model, the bindings will have to be duplicated for each concrete controller.

As a last note, the best API is also a matter of preference. It is important to keep in mind that the API needs to be ergonomic and should fit nicely with the rest of bevy engine.

One last option is no convergence at all of the two APIs, the user will need to write two separate bevy apps to support both OpenXR and WebXR.

I want to solve the dispute with a poll. If you think we should use a WebXR-like API, please react with 🚀. If you think we should use a OpenXR-like API, please react with 🎉. If you have other ideas, want to add arguments in favor of either APIs or want to make clarifications, please comment below.

EDIT: added argument from @Ralith

[Ralith]

[OpenXR actions] require the user to specify bindings for each controller, so the game is not automatically future-proofed for new types of controllers (even with the same button layout)

As I previously explained in discord, this is false. The OpenXR runtime is explicitly responsible for adapting whatever bindings an application suggests to the available hardware. For example, see the SteamVR 1.19.3 changelog, which introduces support for automatically deriving G2 and Cosmos controller bindings from application-specified Oculus Touch bindings. This provides the best of all worlds, as application developers can specify precise bindings for devices of particular interest, the runtime can make approximate mappings for other devices, and the end user can customize to taste.

Further, the WebXR API cannot be implemented on top of the OpenXR input API without flooding user-facing binding UIs provided by OpenXR runtimes with the fake hardcoded actions used to emulate the WebXR controller. This degrades the user experience, and will make Bevy VR apps feel like second-class citizens compared to native VR apps.

to preserve identifiability of controller profiles for selecting the 3D model, the bindings will have to be duplicated for each concrete controller.

This is also false. Querying the current interaction profile is not the correct way to determine the physical controller in use. As explained in the spec:

The runtime may return interaction profiles that do not represent physically present hardware, for example if the runtime is using a known interaction profile to bind to hardware that the application is not aware of.

The correct way to determine an appropriate controller model, if needed, is the XR_MSFT_controller_model extension, or its future successor.

[bjorn3]

Why are you forcing a specific version? This will cause an opaque unresolvable dependency resolution error if any dependency depends on a different patch version.

[zarik5]

web-sys = "=0.3.51" is forced by wgpu. This bug is caused by resolver = 2 which is needed by wgpu. Maybe this will be resolved after switching to the 2021 edition.

[dannymcgee]

Hey, sorry for jumping in here a bit late!

A OpenXR-like API implementation has been tried, using different approach, but no solution has been found that is both ergonomic and performant, even just to wrap the OpenXR backend itself.

@zarik5 Would you be willing to offer a little more context about the problematic ergonomics? My feeling is that I would much rather deal with compromised developer ergonomics if the alternative is to be limited in the type of user experience I'm able to provide. But I'm not 100% sure if I'm reading all of this right, so feel free to correct me if my assumptions are incorrect.

Unreal's OpenXR plugin (which was contributed by Valve) provides first-class support for Steam's input system, which allows users to customize action mappings via the SteamVR overlay and enables developers to provide default mappings for various input devices. It's an excellent user experience because it offers flexibility and familiarity, but most importantly because it doesn't depend on the developer to bake in support for a particular input device. It's been a real headache trying to play a lot of content that isn't using a proper OpenXR integration, because the generic controller mappings rarely translate well across devices. Sometimes this just means controls are clunkier than they ought to be, but often it's outright game breaking. One glaring example is that Fallout 4 VR was virtually unplayable on an Oculus Rift at launch, due to the joysticks haphazardly emulating Vive touchpads in a way that made the map impossible to use.

At the end of the day, I'm not going to cry about it if Bevy doesn't provide my ideal version of OpenXR support out of the box, because the plugin system is versatile enough that I can always work around it if need be, but I just wanted to throw that out there for your consideration. And either way I really appreciate all the work you've invested in this!

[zarik5]

In my latest commit I changed the interaction API to OpenXR-style action-oriented. I kept also the old Button API, that gets decomposed into "{button action}_touch", "{button action}_click", "{button action}_value".

[Dessix]

It looks like the Rendering Rewrite was released with Bevy 0.6; We're 278 commits behind- what all is needed to catch this feature back up to upsteam Bevy, and get this back on the move?

If we can gather up a list of rough tasks and refine them, it'll help less-acquainted contributors like myself to jump in and help.

[kcking]

In my spare time I've been trying to continue zarik5's work on oculus quest 2 at https://github.com/kcking/bevy/tree/xr

I migrated zarik5's fork to wgpu 0.12. I now run into

thread '<unnamed>' panicked at 'Cannot get the native window, it's null and will always be null before Event::Resumed and after Event::Suspended

which I think can be resolved by incorporating blaind's android lifecycle work.

[Dessix]

Your branch compiles once a fixed version of openxrs is specified; I went with ca06a64557abc94559bdb30ceafb51b031e5ac9a and updated the velocity accessors to match the new Option-based API.

Sadly, it seems the current iteration crashes on startup if XR isn't available, instead of offering a way to selectively activate it.

Additionally, it looks like it's still based on Bevy 0.5 instead of 0.6, which might take some work to port across?

[kcking]

Additionally, it looks like it's still based on Bevy 0.5 instead of 0.6, which might take some work to port across?

Ah I did this just before the actual version bump to 0.6; I just (force) pushed a rebase of the official bevy 0.6 tag into the branch.

Sadly, it seems the current iteration crashes on startup if XR isn't available, instead of offering a way to selectively activate it.

I think this is just because the bevy_openxr feature is enabled by default: https://github.com/kcking/bevy/blob/xr/Cargo.toml#L29

[Dessix]

It's still probably worth finding a way to make the feature have an opt-in capacity beyond "this executable will crash if your VR runtime isn't active right now", but if we make it an opt-in plugin, it at least won't break the pancake cargo-examples.

The change to tracking.rs was simply updating lines 48-53 to:

    let linear_velocity = velocity
        .linear_velocity
        .map(|x| to_vec3(x));
    let angular_velocity = velocity
        .angular_velocity
        .map(|x| to_vec3(x));

[TheButlah]

From @kcking:

ah looks like we may be able to incorporate #3412, but pass the swapchain images we get from openxr

Just reposting from discord so that the #3412 is linked to this issue. Also there is a lot of discussion going on right now to get this working in @kcking's fork. https://discord.com/channels/691052431525675048/931772673195905124

[alice-i-cecile]

@zarik5 have you commented in #2373? It would be nice to be able to build off this worry-free.

[zarik5]

@alice-i-cecile Done.

[thedocruby]

Whats the status on this PR? Would love to see it merged soon, would like to know how complete the API is and what still needs to be done before merge.

[alice-i-cecile]

@thedocruby this needs to be revived and championed, likely by a different author or coordinated subteam. Adding the Abandoned label to reflect that.

[Type1J]

XR in Bevy needs to happen for Bevy to be selected for an upcomming project of mine. Will this happen anytime soon?

[TheRedXD]

XR support would be extremely useful for us! We've been looking into bevy for quite a while now, and would like to see this actually implemented, we're working on a game that's supposed to have XR support eventually.

[Type1J]

Is XR in Bevy going to happen?

[mockersf]

it's happening at https://github.com/awtterpip/bevy_openxr at the moment

[Type1J]

@TheButlah Why the thumbs down? Does that mean that you don't like the question, or does it mean that you don't want XR in Bevy?

[TheButlah]

Because its just unnecessary noise on the PR - obviously XR is being worked on, there is no need to ask if its going to happen. The answer is yes - eventually.

See awtterpip's PR above, and you can also take a look at https://github.com/NexusSocial/skilltree/ for some concrete examples.

Support is very nascent right now, but there are several people actively working on this, and there is a lot of opportunity to flesh out the third party crates also. Feel free to mess around with them and see if there is something that you'd like to contribute to

[alice-i-cecile]

Ongoing development work is occurring in https://github.com/awtterpip/bevy_openxr. Eventually, I'd like to push to get that upstreamed. For now though, I'm going to close out this PR.