Extensible event types & fallback in Matrix

[benparsons]

Documentation: https://docs.google.com/document/d/1FUVFzTOF4a6XBKVTk55bVRIk4N9u5uZkHS4Rjr_SGxo/edit#heading=h.m568t57r6od9
Author: @ara4n
Date: 18/02/2018

[ara4n]

Earlier generations:

• https://docs.google.com/document/d/1m4VTRqclB3JEMZBjbr4t5cvIMQUNSGxy6rYN4YtopIk/edit
• https://docs.google.com/document/d/1cLSlMZ0tulRJM4ki-ImQ_z6DSySQgTRafO2m13RtgZo/edit#
• https://docs.google.com/document/d/1l08DL_F_CHo1pIORXzcqWLlZQpztogMFUTYmxd4Gr5s/edit
• Straw-man HTML messages proposal #92 and Attaching data to messages #93 sort-of

[alphapapa]

This is an interesting proposal, but please don't over-complicate the protocol. The more complicated it gets, the fewer clients will be available, and those that do will have more bugs and incompatibilities. It seems to me that it's already the case that the vast majority of non-technical users only use Riot, and the more complicated the protocol gets, the more likely that is to remain the case. (Not that there's anything wrong with Riot, but I'd certainly prefer a "thriving ecosystem" of clients rather than one fully-implemented client and a dozen half-baked ones that can't be fully recommended.)

[msrd0]

I disagree with @alphapapa. With events not being extensible, this means that if I wanted to have an html message with a custom field, I'd need to create my own event type which no clients other than my own one is going to support. However, if I was able to extend the org.matrix.custom.html message type, different clients supporting that could display my html message and I could still add a custom field to it.

[alphapapa]

@msrd0 I don't understand what you disagree with me about. I said, "Please don't overcomplicate the protocol." You're saying you do want the protocol overcomplicated? :)

If I wanted to have an html message with a custom field...

Can you give me an example of what you mean?

However, if I was able to extend the org.matrix.custom.html message type, different clients supporting that could display my html message and I could still add a custom field to it.

I think that's a fine idea, but I'm not sure that "mixins" are the solution. Clients can just ignore keys they don't recognize, just like they ignore HTTP headers they don't recognize.

[msrd0]

@alphapapa Yes, you are right, this proposal will make the protocol more complicated.

What I disagree with is that this proposal will mean less clients will be usable. Because if we can extend events that would mean that we can build events that are completely supported by one client, but another client might only know the event that it extended, and so can still display something.

I know as of right now the spec says that events that have a body can be displayed as text (at least in riot) but there is currently no way to e.g. add html to a custom event.

Clients can just ignore keys they don't recognize, just like they ignore HTTP headers they don't recognize.

That basically means we can get rid of the event type completely, because clients will filter what they know and discard everything else? Well, there must have been a reason to invent event types

[alphapapa]

What I disagree with is that this proposal will mean less clients will be usable. Because if we can extend events that would mean that we can build events that are completely supported by one client, but another client might only know the event that it extended, and so can still display something.

My point is that, in general, the more complicated a protocol is, the fewer clients will be implemented, and of those that are, fewer will support the protocol well.

I know as of right now the spec says that events that have a body can be displayed as text (at least in riot) but there is currently no way to e.g. add html to a custom event.

I'm far from an expert on the protocol, so maybe I'm just ignorant here, but: if it's a custom event, why do you need any special support for anything? It's a custom event. If you want HTML in it, put HTML in it, and the clients who support that event will know what to do with it, because they are made to support it.

That basically means we can get rid of the event type completely, because clients will filter what they know and discard everything else? Well, there must have been a reason to invent event types

ISTM that event types are primarily a convention. e.g. HTTP has standard headers: Accept-Encoding, Range, etc. But any server or client can also send other headers (which are generally supposed to be prefixed with X-, if they aren't official). In the end, they're just text keys and text values. Isn't the same the case for Matrix? Define some official event types so clients can have a baseline of interoperability (e.g. m.room.message, not one client having message and another having room_message).

Other than that, you either allow custom event types (keys) or you don't. And if you do, then why complicate that with rules for how to mix them together, multiple inheritance, etc? That sounds like a nightmare for client developers to implement.

I feel like my own client's handling of event types in the /sync handler is complicated enough already, when they're all listed in the API docs with specific keys. If I had to support types and subtypes and types that inherit from each other and mix together... I'd either support just the most plain, basic types and leave the rest out (which means another client that doesn't support the full protocol and looks bad in comparison to Riot), or I'd give up writing a client altogether, because no one's paying me to do that, and it reaches a point where it's not fun anymore.

I recently read an article by one of the creators of the OAuth protocols, about how OAuth2 was, in his opinion, a dismal, design-by-committee-for-the-enterprise failure, and how he recommends the older, OAuth1 protocol, for general use. I would hate to see that kind of outcome for Matrix. I'd hate to see people just keep using IRC forever, and Matrix fade away into obscurity (of course, many would say it already is). I'd hate to see it become so much work to implement that no one bothers unless they are paid to.

And one could even say that that is already the case, because it doesn't seem very practical for me to implement encryption support for my client (lack of a straightforward way for me to use libolm, but that's not necessarily Matrix's fault).

My 2 cents as an amateur Matrix client developer.

[msrd0]

@alphapapa There are a lot of things in your comment that I agree with, especially the libolm part.

However, those things aren't the scope of this issue. What's part of this issue is event extension.

I'm far from an expert on the protocol, so maybe I'm just ignorant here, but: if it's a custom event, why do you need any special support for anything? It's a custom event. If you want HTML in it, put HTML in it, and the clients who support that event will know what to do with it, because they are made to support it.

Currently (ok, I didn't read the spec yesterday, so point me out if anything changed in the meantime), the spec recommends always displaying the body as plain text if the client doesn't recognize the event type.

Of course I can put HTML into a custom event, but there is currently no standardized way to do so. I could add my own field and call it whatever I want, but no client, not even Riot, would display that HTML, eventhough capable of displaying HTML in general.

With this proposal, if I understand things correctly as I'm not an expert myself, I can extend an event with HTML content supported by all major clients, and add my own fields with my own event type. This way, clients, like your client, my client, Riot, or any other client, without knowing the event type itself, can still display the HTML as if it was an event they'd recognize.

[alphapapa]

Of course I can put HTML into a custom event, but there is currently no standardized way to do so. I could add my own field and call it whatever I want, but no client, not even Riot, would display that HTML, eventhough capable of displaying HTML in general.

That seems to me like a reason for fields to have optional MIME-types associated with them. I guess that would add complexity as well, but it would seem to me to make sense. It seems like it should be similar to MIME in that each message would potentially have alternative formats, and the client could choose which to render. Rather than have body and org.matrix.custom.html, there should be a body with text/plain and text/html formats available for it, something like:

{
    "body": {
        "text/plain": "*Hello, world!*",
        "text/html": "<b>Hello, world!</b>"
    },
    "image": {
        "url": {
            "image/jpeg": "http://example.com/image.jpg",
            "image/webp": "http://example.com/image.webp"
        },
        "caption": {
            "text/plain": "An example image",
            "text/html": "<i>An example image</i>"
        }
    }
}

[ara4n]

This has now been replaced by #1767, which:

• converts it from googledoc to MD
• simplifies it by removing displayhints (for now)
• removes all references to mixins, as the term was scaring people and making it feel far too type-theoretic
• replaces the clunky m.text.1 idea with lists for types which support fallbacks
• removes the concept of optional compact form for m.text by instead having m.text always in compact form
• hopefully ends up with a result similar enough to @alphapapa’s proposal that it doesn’t feel overcomplex.