Saved-object references used to model a "composition" relationship

[kobelb]

Saved-object references are currently being used to represent a "composition" relationship, where end-users don't think of the two saved-objects as being separate entities but a single composite entity.

An example of this is the Security Solution's cases. A case itself is modeled using a cases saved-object which has references to the cases-comments, cases-configure, and cases-user-actions saved-objects. Cases can't currently be exported using saved-object management, but if we made them importable/exportable we'd have a number of issues.

When the end-user exports the case saved-object, the related comments, configuration, and user-actions will not be automatically included. Additionally, we don't want the comments, configuration, and user-actions to be listed on the saved-object management screens as they're conceptually just a component of the case. Saved-object references will also have repercussions on the behavior of authorization once sharing saved-objects in multiple spaces is implemented, which I don't think we want for cases since end-users won't think of cases and their referenced saved-objects as distinct entities.

Potential Solutions

Use a single saved-object

When confronted with a similar problem when discussing how to reduce the time to visualize, we decided that the Dashboard saved-object itself should embed the Visualizations which are only used in the context of the specific Dashboard. This way, when the end-user exports the Dashboard they automatically get the Dashboard and all of the embedded Visualizations.

If we decide that this is the recommended approach, we'll need to provide developers a method of migrating from multiple saved-objects into a single saved-object. This solution will likely require rather significant changes in situations where multiple saved-objects are currently being used.

Use different saved-objects, support the "composition" relationship

This would allow developers to continue to use multiple saved-objects to model a single "entity". This would at a minimum require changes to export to automatically include the related saved-objects.

My greatest fear with this option is how it'll work with future features like Object-level security. With OLS, each saved-object will have an ACL that specifies which users have access to the saved-object. This will potentially complicate the logic when authorizing the end-user to perform an action with one of the "composed" saved-object, as we should be using the ACL that is specified at the root entity.

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[rudolf]

Using the DDD terminology of "entity" vs "value object" might be useful here. A value object has no identity (id field) so it's usually embedded in an entity (in a relational db it's inlined instead of being in it's own table). In your example, Dashboards are always entities, whereas visualizations can be an entity if it's shared with other dashboards or a value object if used only in a single dashboard.

I think ES has powerful features to make embedding value objects in entities easy. This also solves many other problems like cleaning up all child objects when the top-level parent is deleted (like deleting all case-comments when a case is deleted)

There might be more reasons, but I think at least one of the reasons we often model value objects as entities in Kibana is because of the limited querying/filtering ability of saved objects for nested documents #81009 (comment).

[kobelb]

@rudolf, I don't think it has any practical impact on the matter, and I think we're on the same page about the benefits/drawbacks here. Also, it's been years since I read the DDD book, so I'm a bit out of practice thinking in these terms. Even if we determined that we're only dealing with entities and there are no value objects, doesn't the concept of aggregates still apply here? For example, if we determine that cases and cases-comments are entities because both can have their attributes changed and conceptually have the same identity, we could still model them as an "aggregate" with the case being the root entity, and the cases-comments being a "non-root" entity? If that's the case, we would still want the case and case-comments to be persisted as a document per aggregate.

[kobelb]

/cc @elastic/siem

[pgayvallet]

When the end-user exports the case saved-object, the related comments, configuration, and user-actions will not be automatically included. Additionally, we don't want the comments, configuration, and user-actions to be listed on the saved-object management screens as they're conceptually just a component of the case

This part of the issue should be handled by the resolution of #82027, right?

[kobelb]

This part of the issue should be handled by the resolution of #82027, right?

If we have a solution in mind that addresses both problems, that's great! I was initially thinking of them as requiring separate solutions, but we can couple them together if you think it makes it easier to solve.

[pgayvallet]

I maybe have misunderstood the problem then

When the end-user exports the case saved-object, the related comments, configuration, and user-actions will not be automatically included.

Will not or should not? If it's the first, and if cases cannot currently be exported 'just' because they are a hidden SO type, wouldn't #82027 directly address this?

[kobelb]

Lemme try again :) When the end-user exports the case saved-object, the related comments, configuration, and user-actions will not be automatically included and they should be automatically included.

Addressing #82027 potentially solves the import/export problem. However, we'll continue to have an issue when it comes to sharing saved-objects in multiple spaces and in the future for OLS.

[mshustov]

When the end-user exports the case saved-object, the related comments, configuration, and user-actions will not be automatically included and they should be automatically included.

It seems we need to start with defining several basic recommendations (and probably document them):
In what cases should we store data in SO as a separate entity?
If comments, configuration, and user-actions cannot exist outside of the case context, they shouldn't be stored as a separate SO object, maybe? Instead, the case entity controls how SO are created, copied & destroyed.
Are there any valid use-cases when we need to store them separately?
Are there any technical limitations to store them separately? Search by comment text, maybe?

[kobelb]

@spong @XavierM when you all modeled Cases and Timelines as separate saved-object types, was this because of technical complexities that prevented you from just using a singular saved-object?

[XavierM]

@spong @XavierM when you all modeled Cases and Timelines as separate saved-object types, was this because of technical complexities that prevented you from just using a singular saved-object?

It was to avoid conflict with the saved object version since we can have multiple users adding a comment on the same case/timeline (maybe in concurrence).

[joshdover]

It was to avoid conflict with the saved object version since we can have multiple users adding a comment on the same case/timeline (maybe in concurrence).

This makes sense from a simplicity standpoint, but I don't know if it's a great reason to keep this architecture. This should be solveable with a compare-and-set + retry strategy instead, without degrading the user experience.

Using a single document for an entity makes many other operations much simpler (as noted above), though it does have flexibility limitations. I can contrive examples, but I'm not sure any of them are valid use cases we'd actually want to build. If we can't identify valid usecases for the "composition" model, then I lean heavily towards keeping this simple and using a single document.

[kobelb]

I agree that cases could most likely be modeled using a single document; however, there are some situations where this approach doesn't work.

If we were to try and model the Security Solution's exception lists using saved-objects, we likely wouldn't be able to use a single document to manage it. From what I understand, end-users generally think of the exception list as the entity, and each item in the exception list only belongs to that exception list. However, there are so many items in the list, that loading them all into memory at the same time causes potential issues. This is further exacerbated by the workflows around exception lists, as they wish to allow end-users to add/remove/edit items from the list.

[pgayvallet]

@kobelb retrospectively, I assume we're fine closing this as won't fix?