Pre RFC: dataflow driven scheduling API

[dburrows0]

Hi all,

Recently, I've been collecting ideas and notes on aspects of the Bevy ECS API related to scheduling and ordering of systems. I'd like to convert these into a series of RFCs, but first, I created a proof of concept implementation, in order to convince myself that the whole set of ideas together was cohesive and implementable. I hope it will also be interesting to others.

You can find the code here. Docs can be built with rustdoc or found in the repository.

The most important principles behind the design are:

1. Action never happens at a distance and there are no global variables. Within the API, every Resource and Component has a single owner that takes responsibility for managing it. Other code must be passed a handle to the data.
2. Things happen in the order that they are described -- as far as the caller can tell, anyway. If I initialize a resource, run a system to prepare it, then run a system that needs the prepared value, the second system never runs before the first one or before initialization. Same thing if I run a system that spawns entities, then another system that collects them into a reverse index.

An important point about the implementation is that it's a layer on top of bevy_ecs. I think this is a solid technical direction, because it allows some decoupling between the two crates: bevy_ecs can focus on organizing data for efficiency and high performance, while bevy_flow provides a nice high-level API. Once scheduling is complete, there should be virtually no overhead from bevy_flow.

As a bonus, bevy_flow also contains proof of concept implementations of entity tables and derived data (aka indexing), built on the foundations above (@alice-i-cecile, I know you are interested in derived data).

I'd be interested to hear what others think about this direction, and particularly interested to hear feedback if anyone experiments with the API. I'm especially interested in the overall structure and organization -- I expect that names and function signatures will change in the RFC process.

Thanks!

[mtsr]

Might be interesting to look at this issue and particularly this specific comment by AlphaModder (don't forget to unfold the folded code at the bottom of the comment) for how Commands might fit into the scheduling part of this. Edit: It seems you're already doing something like it.

One comment I have on the code example is that the ergonomics as written are not good. The extra type parameters such as AllTables alone look intimidating to new users.

[dburrows0]

The AllTables issue you point out is interesting. I agree that it's somewhat unfortunate for it to show up in types. On the other hand, it's hard for it to do its job if it doesn't, at least the part of its job that involves conveying type information from Query to things like EntityDespawner, to ensure that entities being despawned are in the expected table scope.

But, here's a crazy idea: it occurs to me that with some trickery, it might be possible to allow system declarations like:

fn my_system(impl Query<Item = (&C1, &C2)>) {}

and hide both the table type and the filter type, which the system doesn't really care about. bevy_flow could supply a thin "trampoline" system which is a generic function instantiated with the right types (determined by the parameters to Schedule::run) for bevy_ecs to invoke, which then calls out to the user's system. You only need to declare the table type if you need it (e.g, to pass into a matching Despawner), which means that when it shows up, it actually matters.

This looks a lot more ergonomic, but I'm a little worried that it might produce incredibly obtuse compiler errors. I think that hard to understand compiler errors are a much bigger problem than extra required type parameters or extra syntax, and I'm already worried that the complexity of the API is making it hard for Rust to report meaningful information when things go wrong.

Another possible issue is that the type to use is not very discoverable (can also be an issue with the current setup, but at least you get an error message telling you the exact type that was expected).

But it might be worth an experiment!

[TheRawMeatball]

Hmm, this looks very promising but right now, the boilerplate looks just too much :(
A cheap win could perhaps be mitigated by moving types like AllTables to the end and using type defaults, but even then this drastically increases the cost of adding a component, a resource or a system.

The added control of systems being piped in handles is also a double-edged sword. On one hand, it helps in managing side effects and understandably, but on the other hand I feel like it could quickly lead to plumbing mess, as when a system decides it needs a component / resource and suddenly it needs to be piped through 3 modules to get it there.

[dburrows0]

Thanks for looking over my work!

A cheap win could perhaps be mitigated by moving types like AllTables to the end and using type defaults, but even then this drastically increases the cost of adding a component, a resource or a system.

One thing I'll note is that AllTables and friends are somewhat separable from the rest of the API. The "table subsystem" adds some structure to entity access and (if plumbed into bevy_ecs) would allow more code to run in parallel otherwise, but it's not really needed for the system to work. I included it because I wanted to understand what additional complications it would add.

If there was consensus that everything else here was fine, but that the table part of the API was too much, I would be happy to write designs and code for the rest of bevy_flow, and have a separate conversation about Tables later on. I was already planning to stage it towards the end of the project anyway.

The added control of systems being piped in handles is also a double-edged sword. On one hand, it helps in managing side effects and understandably, but on the other hand I feel like it could quickly lead to plumbing mess, as when a system decides it needs a component / resource and suddenly it needs to be piped through 3 modules to get it there.

It's true that having all the data be globally accessible makes it easier to use it from new places. In fact, some types should be global (most of the resources and components exported by Bevy are examples of this).

My proposal includes an Ids type associated with the root API object (the terribly named Builder), which is meant to hold the ought-to-be-global stuff, along with some derive macros so you don't have to write boilerplate to set this up. The nested bevy_flow_ids crate declares the handles for all the stuff I could find in Bevy. This could be abused to make more stuff global than really should be, but that's up to the developer.

Another possibility I considered was adding a "global" registration mode. So when you declare the resource or what have you, you tag it as "this is supposed to be global". From that point, either it's really easy to get a handle, or users can just inject it without ceremony (I prefer the first, though, because otherwise the API becomes inconsistent in weird ways).

I personally like the Ids solution because (1) it provides structure to the sorta-global stuff, including automatic support (using Rust visibility and methods) for things like read-only access to data, and (2) it doubles as an easily discoverable reference for all the stuff that the user can inject into their Systems. But I think I could be OK with either of these approaches, as long as a non-global handle can't be converted to a global one (except by the owner). It would even be possible to implement both, but I don't like that as much because it seems hard to pick which one to use.

[cart]

First: this is very cool!

Buuut this appears to be a pretty major digression from the bevy app model, in that it aims to undo "inversion of control" from the public api while still using it in the internal schedule execution. There are definitely benefits, especially for single-plugin use cases, but in a "modular" context I think this starts to create problems. When a system needs an AssetServer reference, instead of just saying "give me the AssetServer" using fn system(server: Res<AssetServer>), now the AssetPlugin needs to somehow hold on to an "asset server resource handle", which then needs to be manually passed in to every plugin that consumes assets. Imagine how nasty this would get when a plugin needs AssetPlugin::AssetServer, SpritePlugin::{Sprite, Assets<Sprite>, AssetEvent<Sprite>}, TransformPlugin::{Transform, GlobalTransform}, RenderPlugin::{Camera, OrthographicCamera, Assets<Shader>,

[dburrows0]

First, thanks for taking a look!

Yes, this is definitely a departure from the existing Bevy model -- that's one reason that I'm engaging this way instead of just sending an out of the blue RFC for it (the other is that I, uh, haven't actually built any code on top of it yet, and I want to do that before I send a formal proposal saying we should do this). I am pretty sure that I would like to use it, and I bet others would too. I wouldn't propose replacing the existing ECS with it, for one thing because there are clearly users who are happier with it.

Since bevy_flow is an external layer, both it and bevy_ecs can co-exist, but I'm hoping that I can persuade you that bevy_flow belongs in Bevy. I think it'll be a better experience for at least some users, and having it built in rather than as an external dependency would be a really nifty feature for them.

That said, I'll answer your specific comment here:

When a system needs an AssetServer reference, instead of just saying "give me the AssetServer" using fn system(server: Res), now the AssetPlugin needs to somehow hold on to an "asset server resource handle", which then needs to be manually passed in to every plugin that consumes assets. Imagine how nasty this would get when a plugin needs AssetPlugin::AssetServer, SpritePlugin::{Sprite, Assets, AssetEvent}, TransformPlugin::{Transform, GlobalTransform}, RenderPlugin::{Camera, OrthographicCamera, Assets,

Yes, this is definitely a potential issue!

I spent some time thinking about this, and came to the following conclusion: it's OK for the types of Bevy to be global. There's no possible harm from everyone who wants access to Assets getting it, if all they need it for is to create a Handle or resolve one that they already have. (I'm not sure about some of the other functionality in Assets, but I think those two cases are like 99% of the use scenarios) Any code that wants to should be able to read from the keyboard. The use of Transform is a bit iffier in my opinion, but it's also needed pretty broadly (my general plan was to use Tables to constrain this, so anyone could get Transform but only some systems could use it in a particular Table; however, this ran into some complexities and I'm not sure it's feasible yet).

The pattern I implemented in bevy_flow to address this is to add a bundle of handles to the root Builder type. This type has to be passed to everything anyway, since it is the interface for calling back to register systems, similar to app::AppBuilder or to Golang's Context. So I added a type parameter Ids which is meant for carrying around pseudo-global data. The "standard" implementation provides a collection of handles for all the standard Bevy API types. By doing it this way, we get a couple benefits:

1. It's extensible: the user can build their own Ids type to add stuff that should be global in their app. I included a procmacro to simplify this. The Ids are just normal Rust structs, so they can use visibility and methods to control access in a fine grained way.
2. It's flexible: if some subsystem should have more -- or fewer! -- handles available, it's possible to replace the Ids field and hand that subsystem something more appropriate. (it's not currently very ergonomic, but that's easy to fix)
3. It's discoverable: by reading the definition of the Ids type, the user can find all the stuff they can inject into their system. Currently, it's not clear from just reading the Bevy Rustdoc which parts of the library are part of the "system API".

This addressed most of my concerns with the API on paper. It is a bit abusable, but I would feel better about giving people a well lit path to build structure, than about giving them an API that doesn't provide any way to build a structured system around it.

There is an alternative, which you can see in my reply to @TheRawMeatball above, but I wanted to put in another word for my favored approach!

[dburrows0]

It's been bothering me since yesterday that I cryptically wrote:

The use of Transform is a bit iffier in my opinion

I was trying to avoid getting into a discussion on Tables, but everyone who replied to my OP noticed them, so I think I should explain myself. Here goes.

First, I will say: there's absolutely no reason from an architectural point that Transform shouldn't be global. Nothing is gained by restricting which code can access it. But that still doesn't leave me in what I would consider a good situation, which is where my "iffier" comment came from.

As a user of Bevy who wants to manage my app's architecture, the problem is that the "write to Transform" capability is all-or-nothing. What I really want to do is hand over a capability like "write to the Transform of the Player entity" to the module that manages Player related things.

The "tables" feature in bevy_flow is about breaking up the monolithic write (and read) capabilities of these widely shared components so that subsets of them can be addressed, with a promise that code "owns" the subset that it has access to. There are some other nice things you get (like the ability to run systems in parallel), but the architectural division is the core motivation for me.

The part of it that doesn't work quite as I'd like is that I wasn't able to convince myself that I could get rid of the "write to all tables" capability (AllTables). It's needed for two reasons: (1) support for entities that aren't in Tables, such as interoperation with non-bevy_flow code; (2) support for systems that operate across all tables, especially ones that use the "derived data" feature to automatically provide columns. For (2), I considered saying that those systems have to be explicitly registered on a per table basis, but I wasn't able to convince myself that it was worth giving up the nicety of one-time encapsulated registration.

The problem with "write to all tables" hanging around is that it's basically a second mutable pointer to table data. This isn't a Rust-safety problem, since bevy_ecs manages pointers, but it does mean that you lose the promise that no one is fiddling with your data "behind your back". I have a few ideas for addressing this, but haven't thought them through yet.

There are also some "ergonomics" issues introduced by the extra complexity of tables.

Because of these complications, I think it's good to sever the "tables" feature from the rest of bevy_flow and file a separate RFC to tackle it and explore the possible solutions to these issues.

[cart]

it's OK for the types of Bevy to be global.

In my opinion, there should be no major distinction between "bevy types" and "user types". Internal Bevy Plugins should use the same conventions / patterns as user Plugins. So if we don't use that as our criteria, its seems like "usability in a wide context" is the criteria we would use.

But I have a couple of problems with that:

1. we no longer have "one way" to consume types. When a plugin exports a component, you no longer know if you can just "grab it globally" or if you need to fish around the plugin to find where it exports the handle.
2. "usability in a wide context" is hard to define / arbitrary.

Imo that is a significant reduction in UX (both for plugin developers who now need to make another "design decision" for every type they define, and plugin consumers, who now need to find which consumption pattern is required for each type). I don't think enabling a "traditional control flow" pattern outweighs that cost. From my perspective, Bevy should either consistently be an "inversion of control" framework, or it should consistently be a "traditional control flow" framework. And you will have a hard time convincing me to remove inversion of control entirely, due to usability concerns around common types.

[concave-sphere]

In my opinion, there should be no major distinction between "bevy types" and "user types".

I agree that there isn't a sharp line. What I mean is that the types that show up in the Bevy API are meant to be basically global everywhere, and I think that makes up most of the components and resources that Bevy defines (apologies if this is mistaken). In user code, you're going to see the reverse: most types have a limited scope, and only a few types are meant to be widely shared.

Like you said, both modes need to be supported. But in general, my expectation is that the number of "private" types will tend to be much larger over time than the "public" types, particularly in code written by "end users".

(this is not really connected to my work at this point and is just a philosophical comment on use cases)

[Ratysz]

I've spent some time poking around and trying to figure this out, but I still can't definitively answer this question: what problem does this solve?

In the "Why is this useful?" section of the docs:

A system should only be able to act on data if it was given access to that data in a normal Rust way. [...]

What does this mean, exactly? What is different in bevy_flow from what we have now? Currently, systems aren't given access to anything they didn't ask for in their signature, so it works with normal encapsulation as expected: if a system can't see a type (it's not pub for it), it can't have it in the signature, and thus cannot access it.

It’s much easier to reason about the order in which things happen if it’s consistent with the order in which they are described in the code.

This seems to be contrasted with current "order is not defined unless you explicitly define it" approach, and advocating a regression to the previous overly-restrictive and rigid model that implicitly preserved the order of insertion of systems in their order of execution. In fact, the whole endeavor feels like a way to build that again on top of the new model.

[Ratysz]

[...] sorry if I'm out of line with my ideas.

You have nothing to apologize for. I've been told before that at times I come off as overly critical, and, evidently, I still haven't learned from that. I do always intend for my critique to be constructive - I'm not trying to shoot you down, I'm inviting you to defend your ideas.

I can see there's some history here that you feel strongly about. If you don't mind, I'd be interested to read the user scenarios that convinced you of this. Do you have a link to them? Thanks!

Well, #1144 is a good start. I would also recommend this thing I wrote, as a primer; the vocabulary has evolved since then but it should be easy enough to draw parallels to where things are now.

There have been numerous discussions prior to that PR's inception, during its implementation, and in its aftermath - these are hard to track down because they mostly happened on Discord, a good chunk of a year ago.

Old model implicitly preserved system insertion order when executing them, the only systems that were parallelized are disjoint neighbors. This ends up inferring a lot of execution order dependencies, some of which are spurious, i.e. the operations performed by systems in question are commutative and the user doesn't actually care which order they are performed in. In practice (so far, from user reports) it turned out to be quite a large portion of them. The rules of inferrence weren't trivial to reason about, and there was no way to break the inferred dependencies. It was impossible to allow systems of one plugin to run in parallel with systems of another plugin if they touched same data somewhere.

New model is dead-simple: "there is no order unless you explicitly define order". No implicit dependencies, no inferred order. System insertion order doesn't put any constraints on execution order, so refactoring plugin initialization no longer alters the execution graph.

I should add one more thing: there are still missing bits of ergonomics. For example, I wanted to add a shorthand to define sequences of systems without having to write them out as a string of dependencies, but we had to cut off the 0.5 release somewhere (I took an extended break since the eve of its release and nobody else seems to have picked up the torch). It's not hard to implement, but it is hard to design.

[Ratysz]

Another point I forgot, concerning this from a different reply:

Since bevy_flow is an external layer, both it and bevy_ecs can co-exist, but I'm hoping that I can persuade you that bevy_flow belongs in Bevy.

bevy_flow can be built on top of what we have now, but the reverse doesn't seem to be possible. An engine's core should be as flexible as possible (read: feasible), precisely to allow end users to build whatever they want.

[concave-sphere]

So, I think I should say that I'm not surprised if there are scenarios my code doesn't handle. Generally I approach a problem like this in stages:

(1) Figure out a model of the space that seems to line up with the most important needs (whatever those might be)
(2) Design/implement something based on that model. This is probably a system that's restricted/limited to just the model. The goal at this point is to make sure the system works "on its own terms" for the scenarios it's actually supposed to work for.
(3) Find important use cases not covered by (1) and (2), and find ways to cover them. This can come in the form of expanding the model, building "escape hatches" where users can weaken the model in well defined ways, or creating ways to interoperate with code outside the model.

I can expand more on why I made the technical decisions I did, and I probably should when I have more time, but in this post, what I want to say is that when I wrote the OP, I was thinking that I was more or less done with (2) and needed to start on (3). In other words, I know this system is incomplete and not flexible, and the reason I was putting it out there was to find out what additional scenarios I need to cover. The reason I do things in this order is that it's hard to predict which edge cases are actually important, and it can really clutter up the API if you try to cover every imaginable case from the start.

[I actually don't think I'm quite done with (2) -- over the weekend, I implemented a trivial little Tetris on top of it, and the stuff that I want to be easy wasn't, so I may spend a bit more time on that before moving on]

I can see there's some history here that you feel strongly about. If you don't mind, I'd be interested to read the user scenarios that convinced you of this. Do you have a link to them? Thanks!

Well, #1144 is a good start. I would also recommend this thing I wrote, as a primer; the vocabulary has evolved since then but it should be easy enough to draw parallels to where things are now.

There have been numerous discussions prior to that PR's inception, during its implementation, and in its aftermath - these are hard to track down because they mostly happened on Discord, a good chunk of a year ago.

Old model implicitly preserved system insertion order when executing them, the only systems that were parallelized are disjoint neighbors. This ends up inferring a lot of execution order dependencies, some of which are spurious, i.e. the operations performed by systems in question are commutative and the user doesn't actually care which order they are performed in. In practice (so far, from user reports) it turned out to be quite a large portion of them. The rules of inferrence weren't trivial to reason about, and there was no way to break the inferred dependencies. It was impossible to allow systems of one plugin to run in parallel with systems of another plugin if they touched same data somewhere.

So, did you find that commutative systems were pretty common, and that users cared about parallelising them? That's pretty interesting! Right now, command-issuing systems in bevy_flow are "commutative" if you ask to defer their work, but there isn't a way for user types to plug into this mechanism, and I was actually unsure if the feature was worth the complication it introduced. Generalized commutativity is one of the scenarios I had identified as a potential issue, and I have some notes on how to handle it, but I wasn't sure it would really matter. Do you have any links to concrete examples of this that I could read up on?

My idea here was to either build a targeted feature for commutative types, or a generalized "escape hatch" for disabling order dependencies on some identified set of values. My general preference would be to start with the escape hatch, which will cover a large swath of scenarios and is probably needed anyway, and only build something targeted if it really seems to provide a lot of value.

New model is dead-simple: "there is no order unless you explicitly define order". No implicit dependencies, no inferred order. System insertion order doesn't put any constraints on execution order, so refactoring plugin initialization no longer alters the execution graph.

This is the main pain point I have with Bevy, and from reading issues and other sources, I don't think I'm alone. Just the other day, I saw someone explaining on Discord that they were going to compute values using "eventual consistency" to avoid the pain of having to calculate the right order across stages. It's a clever workaround, but it seems like a clear sign that something is missing.

[concave-sphere]

Another point I forgot, concerning this from a different reply:

Since bevy_flow is an external layer, both it and bevy_ecs can co-exist, but I'm hoping that I can persuade you that bevy_flow belongs in Bevy.

bevy_flow can be built on top of what we have now, but the reverse doesn't seem to be possible. An engine's core should be as flexible as possible (read: feasible), precisely to allow end users to build whatever they want.

I thumbs-uped this because I 100% agree! If / when I start sending RFCs (and I need to get to stage 3 in my rough plan above first), I would either have to (A) have a sufficient number of "escape hatches" to allow bevy_flow to be used in a general way, or (B) have good interoperability with bevy_ecs and ask to incorporate bevy_flow as a layer on top of it. I don't know at this point which way it will go, but a lot of execution/scheduling APIs I've used in the past had this kind of layering -- a simple, flexible, and general core execution engine, and an upper level API layer that provides a more abstracted model. It works well, as long as the two APIs cooperate and mesh well -- ensuring good cooperation is one of the reasons I wanted to work with you on this.

[Ratysz]

I can expand more on why I made the technical decisions I did [...]

That might be nice, but I don't think it will answer my initial questions.

Do you have any links to concrete examples of this that I could read up on?

Not on hand. If it's not somewhere around #1144 then it's not on Github, and I'm not going to go dredging for months-old convos on Discord. There was feedback from at least two very vocal opponents of the new model: partway through migrating their code from old model to the new they both realized that they don't have to worry about the precise execution order of a good amount of their systems, so the amount of dependencies they specified could be decreased greatly.

An example I remember is mutating the transform of something: if you have multiple systems nudging objects in different directions, you only care about where the objects end up by the end of tick, and the order of the nudges doesn't matter. A practical example of that is naive UI positioning.

This is the main pain point I have with Bevy, and from reading issues and other sources, I don't think I'm alone. Just the other day, I saw someone explaining on Discord that they were going to compute values using "eventual consistency" to avoid the pain of having to calculate the right order across stages. It's a clever workaround, but it seems like a clear sign that something is missing.

Bear in mind that positive feedback doesn't often make it to Github. From the issues I've seen, the problems mostly stem from misunderstanding of how commands and/or states work and how they interoperate with stages; the rest are about things we have plans for and just haven't gotten around to yet. #1375 should greatly simplify at least the states; it will make commands easier to reason about as well, but I think their main issue is in their obligate omnipotence rather than the way they're scheduled.

Bottom line is, if I understand correctly what you're trying to solve, we know how to fix or at least alleviate all of those problems, and the solution is not a return to implicit ordering - because that would just bring back the problems we solved by ditching it.

[alice-i-cecile]

So, I finally have time to give this a proper look. I share the concerns about weak / unclear motivation and boilerplate, but I'm open to being convinced :)

It's much easier to reason about the order in which things happen if it's consistent with the order in which they are described in the code.

My intuition is that this is true for very small programs, and actively false for large ones. Unfortunately, it's impossible to achieve perfect modularity. As a result, your high-level units of code (plugins, in Bevy's case) end up having subtle cross-dependencies. By encoding these dependencies into the order in which these high-level dependencies are composed you a) reduce flexibility to solve issues (what if some of A needs to run before competitors in B, but the opposite is also true) and b) increase implicit mental overhead (oh wait this only works because x <: X and y :< Y and X > Y) and c) increase fragility (good luck refactoring safely when there are huge implicit dependencies cutting across your code base).

For quite a few use cases (e.g. scientific simulation, rollback networking, ironing out tricky bugs), I am convinced that full schedule determinism is a critical goal. However, the three obvious ways to do this are all woefully inadequate at scale. The first, to implicitly resolve ambiguities based on system ordering, has been tried with poor success, as outlined above.

The second is to just run everything in sequence. This again struggles due to overspecification issues, creating boilerplate and causing fragility, as it doesn't map cleanly onto the reasons why you need that particular order. Of course, this also comes with a huge performance hit.

The final obvious option is to just run the ambiguity checker and arbitrarily insert dependencies between pairs of systems (and explicit ambiguities for where you know it "should be impossible for them to conflict"). The problems with this approach though are:

1. Lots of boilerplate.
2. Huge number of equivalent representations. Some of these are more constraining than they need to be.
3. Lots of noise: many of these decisions are just made arbitrarily.
4. Fragility to refactoring. As you modify the code, the explicit ambiguities you've added silently break, the reasons why X must run before Y goes stale, the fact that you've constrained A > B > C arbitrarily causes issues and so on.
5. Hard to reason about: you have a hugely complex web, constructed in a piecemeal fashion.

Pulling this together, I think scheduler determinism is a worthy goal (for some users), but the core challenge is ensuring that your model that describes the data flow dependency graph corresponds tightly with why it must be that way. Mess that up, and your code base becomes opaque, implicit and fragile.

[concave-sphere]

So, I finally have time to give this a proper look. I share the concerns about weak / unclear motivation and boilerplate, but I'm open to being convinced :)

It's much easier to reason about the order in which things happen if it's consistent with the order in which they are described in the code.

My intuition is that this is true for very small programs, and actively false for large ones. Unfortunately, it's impossible to achieve perfect modularity. As a result, your high-level units of code (plugins, in Bevy's case) end up having subtle cross-dependencies. By encoding these dependencies into the order in which these high-level dependencies are composed you a) reduce flexibility to solve issues (what if some of A needs to run before competitors in B, but the opposite is also true) and b) increase implicit mental overhead (oh wait this only works because x <: X and y :< Y and X > Y) and c) increase fragility (good luck refactoring safely when there are huge implicit dependencies cutting across your code base).

Interesting. My intuition is that for small programs, I can reason about all the interactions between all the parts. For large ones, there are O(N**2) possible interactions, and any change is likely to break something if all interactions are possible. This is especially bad if you have a large number of coders working on the system who might be making different sets of assumptions.

The key to making large scale refactorings possible is that the dependencies between different parts of the program need to be explicitly declared. Having implicit transfer of data between different systems via the ECS back channel is a huge obstacle, whether that manifests as "I can't reorganize my setup" or "I can't reorganize my order dependencies".

The systems I've worked with that handle this the best do it by implicitly encoding state into the objects you manipulate. For instance, by coupling the computation of a value into the process of making it nameable. This is where a lot of the token passing in bevy_flow is coming from.

(snip)

Pulling this together, I think scheduler determinism is a worthy goal (for some users), but the core challenge is ensuring that your model that describes the data flow dependency graph corresponds tightly with why it must be that way. Mess that up, and your code base becomes opaque, implicit and fragile.

Ah, yeah, I think we agree on this point. This is exactly what I'm aiming for, although I might not have achieved it yet.

Although the level of boilerplate is higher than I want, I was actually pretty optimistic after hooking up a mini-Tetris last weekend. The dependencies between different parts of the app that I deliberately put in showed up in the top level "routing logic", which is exactly what I was aiming for.

[concave-sphere]

Pulling this together, I think scheduler determinism is a worthy goal (for some users), but the core challenge is ensuring that your model that describes the data flow dependency graph corresponds tightly with why it must be that way. Mess that up, and your code base becomes opaque, implicit and fragile.

Ah, yeah, I think we agree on this point. This is exactly what I'm aiming for, although I might not have achieved it yet.

Although the level of boilerplate is higher than I want, I was actually pretty optimistic after hooking up a mini-Tetris last weekend. The dependencies between different parts of the app that I deliberately put in showed up in the top level "routing logic", which is exactly what I was aiming for.

I wanted to clarify this a bit more. Basically, what I have right now is a "box of parts" that you can use to build an API that's safe in this way, by doing something like this pseudocode:

fn new_sim_component(builder: Builder) {
  let sim_component = builder.data.component.register_component::<MySimComponent>();
  builder.schedule.run(setup_sim_component.system(), (sim_component_inputs, sim_component));
  return sim_component;
}

What I haven't got is APIs and/or design patterns that do this for you.

I think that for simple cases like the above, the ideal APIs are pretty obvious -- and it might be that that's all I need. I've used other APIs where the "simple easy" mode worked for 80-90% of what you typically needed, and that turned out to be fine. You could break into the more general fragment of the API for the couple of places you needed it. It did mean having to deal with a lot of APIs to cover all the different easy cases, but if each API is simple and well named, that can be handled with documentation (I think Rust's Iterator is an example of this).

I'll need to experiment a bit to see if this hypothesis is correct, though.

[Ratysz]

It feels like there's a fundamental misunredstanding of how ECS wants you to think about things: behavior is supposed to be decoupled from the data, while what you're suggesting seems like it will implicitly derive behavior from data. Order of systems shouldn't be dictated by properties of the data the systems operate on.

[alice-i-cecile]

Commands

I agree, the current "commands can do anything!" model poses serious issues for both ergonomics (due to the necessarily delayed effects) and code readability (well this system takes in Commands so who knows what it might do...). I think this is important to improve.

Derived data

This is a promising angle: I'd very much like this for a number of features (namely indexing). I worry about the scheduler costs in terms of overhead and complexity, but I think this could be killer.

Tables

I am very nervous that this isn't a great model for many games, and will result in a lot of gnashing of the teeth well into the development cycle. I can certainly see the value (see #1481 after all!), but I'm not convinced by the model.

Each Entity belongs to at most one Table.

To me, this is throwing away one of the most powerful advantages of the ECS: the ability to dynamically compose behavior across wildly disparate types of game objects in a trait-like way. Either you make this extremely broad (in which case it has very minimal safety benefits) or narrow enough to enforce invariants appropriately, at the risk of code duplication.

I worry that this ends up in the same place OOP game object models do: with a handful of unopinionated Mega Classes, into which all behavior is shoehorned. Like with the scheduling API, this feels like a leaky abstraction: appealingly simple, and then endlessly frustrating when you get into the messy details because it doesn't track the reasons why things must be that way as tightly as it should.

[concave-sphere]

To me, this is throwing away one of the most powerful advantages of the ECS: the ability to dynamically compose behavior across wildly disparate types of game objects in a trait-like way. Either you make this extremely broad (in which case it has very minimal safety benefits) or narrow enough to enforce invariants appropriately, at the risk of code duplication.

I'm not sure I agree, but I'm open to being convinced. I certainly agree that this is one of the weakest parts of the design at the moment, but maybe for different reasons than you're pointing out. I actually suggested above that I might drop it, but having thought about this some more (the questions you've been asking have been very helpful in clarifying my thinking), I think that there is a need in the design that is filled by this feature, and I'd have to find a replacement.

I should explain what tables are doing in bevy_flow. I didn't put them in "just because" -- I put in the stupidest form of tables I could because they solve a very specific problem: they allow type-safe partitioning of a Component. Without this, it becomes really problematic, for example, to ever have two systems that generate a Transform. You end up with potential interactions between any pair of these systems, and there's no way for them to credibly promise not to interfere with each other (which is critical to avoid the O(N**2) growth of complexity as you add systems).

What I didn't do is add a top down schema to declare tables. With bevy_flow, you can still put any component in any table; it's only when you spawn an Entity that you're forced to choose a table. I feel like this is a reasonable compromise -- it's hard for me to imagine a situation where you'd spawn an Entity without knowing the table that it belongs to.

I could have called them something else (like "domain" or "zone") -- theoretically, I think you could set up tables at the scope of an encapsulation boundary, and still get most of the benefits of not having other folks messing around in your data. There isn't really a requirement for a Table to be homogeneous: currently, what it really represents is "a set of entities whose Queries will conflict with each other".

But honestly, as someone who has done a lot of work with databases, I see a lot of value in having a clear schema, and I'll probably use them as tables in my own code. I view them as a formalization of the practice of sticking a tag onto every Entity to mark its "primary" type, which I already do. I don't see this especially as an "OOP" thing -- e.g, Rust has traits, but every value still has a primary concrete type. The traits (extra tags) are a property of the primary type, not a replacement for it.

Maybe there's another way to do this? The main alternative I can imagine off the top of my head would be to allow different components to be partitioned along different lines from each other. That might be possible, but I'm having trouble imagining a compelling use case, and it would definitely introduce complexities into the API (e.g, how do I declare the scope in which a Query operates?).

My issues with tables

For me, the problem that I have with tables, in their current incarnation, is that (1) they add a bunch of syntactic overhead everywhere, and (2) systems that run on all tables have repeatedly created opportunities to "smuggle" data along paths that shouldn't exist.

(2) is especially concerning to me. An example of this is buried in the module docs: if you have a TableReadOnly, this doesn't actually prevent a system that uses AllTables from mutating your table. IMO this breaks the invariant I was trying to set up with TableReadOnly -- and it's an important one because moving things to "read only mode" is how you "finalize" a value or set of values, and finalizing the values is one way to cut off large swathes of unwanted potential interactions.

One option would be to drop the whole concept of setting tables to "read only", which has the benefit of simplicity, but the drawback that you can no longer express the concept that "I am no longer mutating the Transform component of my Players table". Another option would be to detect this particular error at run time (while registering systems) rather than in the type system. Another option is to come up with some kind of convoluted scheme to tag components as "usable with AllTables", but I haven't come up with a scheme that I actually liked.

I've even gone so far as to consider ditching the ability to run on AllTables, and saying that you have to register every system for every table it processes (this would also help with (1), by letting me drop the Table<Foo> wrapper), but that feels kind of horrifying to me and I don't want to do it, for exactly the reasons you call out -- it penalizes systems that are using the ECS the way it's "supposed" to be used. Imagine having to register that every object should be signed up to use the Transform -> GlobalTransform system! Yuck!

[concave-sphere]

Another option would be to detect [mutable use of an immutable Table] at run time (while registering systems) rather than in the type system.

Actually, this would be terrible. Taken literally, if we tried to replicate the Bevy Transform pattern, it would prevent GlobalTransform from being written by the cross cutting system. So, strike that notion.

[TheRawMeatball]

Hmm, while I see your point, I don't think tables are the way to do it. Partitioning components in a type-safe manner is something we already do using marker components, and is a pattern we hope to enhance using archetype invariants (#1481). I'd highly recommend reading it if you haven't, and highlighting any shortcomings you think tables would fix.

[concave-sphere]

@TheRawMeatball @alice-i-cecile

I think it's quite possible archetype invariants would also work. It looks to me like they support a superset of what tables support. Tables are basically equivalent to the "disjoint" constraint.

My main goal (which I can actually articulate now that I've thought through some of your questions) is to constrain dependencies between code modules and make the dependencies that exist explicit, especially by supporting a style of programming where modules don't export a capability to read data until it's ready for write, and where data is "mostly write-once"[0] (I think there are a few other related patterns, but this is one of the big ones).

I think archetype invariants, once available, should work for this.

I thought through my worries above, and although it introduces some "ceremony", I think maybe I should be looking to Rust for a solution here. Splitting a component by table, or partitioning invariant, is pretty reminiscent of splitting a container. The problem of AllTables access conflicting with individual table access is similar to the problem of access to the un-split container conflicting with access to the split container. If the smart folks who built Rust couldn't find a better solution, I doubt I can.

To make this work with invariants, I think there would need to be some sort of call to explicitly "split" a ComponentMut given a bundle of tag components or what have you (I'm not sure there's a point in splitting a ComponentReadOnly). The API could either return a corresponding bundle of split components (I doubt this is even implementable, and it seems like a lot of overhead), or it could return a token for the split component, from which the splits can be retrieved with runtime borrow checking. Either way, it would hold a mutable borrow on the parent value until it was dropped, preventing the problematic double-use.

At least for the code I work on, unsplit components are only really needed for cross cutting functionality. So the recommended pattern would be something like this at a high level:

fn configure_workflow() {
  let ids = setup_ids(); // Takes ownership of all the components that we need
  { // New scope to constrain use of `split_ids` -- in reality it's probably another function
    let split_ids = ids.split(); // Split whatever needs to be split (mechanics TBD).
    configure_main_logic(split_ids); // Register the logic that's per split of the IDs
  }

  run_crosscutting_logic(ids);
}

There are some questions this raises for me (like, what if we need to split IDs multiple ways at once), but it doesn't seem like an obviously unreasonable direction. I don't know how generalizable it is. Maybe other people want to use splits differently from me.

I'll keep tables in my prototype until something better is available, though. The mechanics are a bit different to archetypes, but the high level issues they raise and address are similar, so I think they're a good stand-in for the stage of development I'm at.

[0] to forestall, or at least anticipate, questions: it's not that multiple writes aren't supported, just that once I have the API cleaned up a bit, it should work particularly well for this pattern.