Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

rejected event used as an auth event #9595

Open
richvdh opened this issue Mar 11, 2021 · 11 comments
Open

rejected event used as an auth event #9595

richvdh opened this issue Mar 11, 2021 · 11 comments
Labels
S-Major Major functionality / product severely impaired, no satisfactory workaround. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues.

Comments

@richvdh
Copy link
Member

richvdh commented Mar 11, 2021

in my database, $h+BUjV0LuuSRmFh5ZFeSVZkP+oo6v7bhyJ+1wYjpLb4 (a regular membership event) uses $NncVOvrPzkKl0u1Q8FIJ7Pz1kZH1iF2lwWj6pZ6+Kkg (a membership event which was rejected due to missing auth events) as an auth event. This seems wrong: $h+BUjV0LuuSRmFh5ZFeSVZkP+oo6v7bhyJ+1wYjpLb4 should have been rejected too.

Logs from the arrival of that event:

2021-03-08 16:00:19,930 - synapse.handlers.federation - 189 - INFO - PUT-2238636-$h+BUjV0LuuSRmFh5ZFeSVZkP+oo6v7bhyJ+1wYjpLb4 - handling received PDU: <FrozenEventV2 event_id='$h+BUjV0LuuSRmFh5ZFeSVZkP+oo6v7bhyJ+1wYjpLb4', type='m.room.member', state_key='@yan:yetanothernerd.xyz'>
2021-03-08 16:00:20,090 - synapse.handlers.federation - 2406 - INFO - PUT-2238636-$h+BUjV0LuuSRmFh5ZFeSVZkP+oo6v7bhyJ+1wYjpLb4 - auth_events refers to events which are not in our calculated auth chain: {'$NncVOvrPzkKl0u1Q8FIJ7Pz1kZH1iF2lwWj6pZ6+Kkg'}
2021-03-08 16:00:20,090 - synapse.state - 452 - INFO - PUT-2238636-$h+BUjV0LuuSRmFh5ZFeSVZkP+oo6v7bhyJ+1wYjpLb4 - Resolving state for !BAXLHOFjvDKUeLafmO:matrix.org with 2 groups
2021-03-08 16:00:20,091 - synapse.handlers.federation - 2446 - INFO - PUT-2238636-$h+BUjV0LuuSRmFh5ZFeSVZkP+oo6v7bhyJ+1wYjpLb4 - After state res: updating auth_events with new state {}
2021-03-08 16:00:20,501 - synapse.state - 573 - INFO - PUT-2238636-$h+BUjV0LuuSRmFh5ZFeSVZkP+oo6v7bhyJ+1wYjpLb4 - Resolving state for !BAXLHOFjvDKUeLafmO:matrix.org with groups [2956930, 2831560, 2969615, 2956433, 2893843]
2021-03-08 16:00:21,115 - synapse.storage.databases.main.event_federation - 230 - INFO - PUT-2238636-$h+BUjV0LuuSRmFh5ZFeSVZkP+oo6v7bhyJ+1wYjpLb4 - Unexpectedly found that events don't have chain IDs in room !BAXLHOFjvDKUeLafmO:matrix.org: {'$NncVOvrPzkKl0u1Q8FIJ7Pz1kZH1iF2lwWj6pZ6+Kkg'}
2021-03-08 16:01:19,925 - synapse.http.site - 219 - INFO - PUT-2238636 - Connection from client lost before response was sent
2021-03-08 16:03:22,528 - synapse.state.v2 - 531 - WARNING - PUT-2238636-$h+BUjV0LuuSRmFh5ZFeSVZkP+oo6v7bhyJ+1wYjpLb4 - auth_event id $S38_aRG4Qz8eIdrTyzJuQdH3Ieus1Y1dhczoFKnOB7M for event $NncVOvrPzkKl0u1Q8FIJ7Pz1kZH1iF2lwWj6pZ6+Kkg is missing
2021-03-08 16:03:30,846 - synapse.http.server - 636 - WARNING - PUT-2238636 - Not sending response to request <XForwardedForRequest at 0x7fd1acb68860 method='PUT' uri='/_matrix/federation/v1/send/1614863178168' clientproto='HTTP/1.0' site='8008'>, already disconnected.
2021-03-08 16:03:30,846 - synapse.access.http.8008 - 316 - INFO - PUT-2238636 - 2a00:1098:84:1c8::157 - 8008 - {matrix.org} Processed request: 190.923sec/-130.921sec (0.982sec, 0.012sec) (0.080sec/187.732sec/31) 0B 200! "PUT /_matrix/federation/v1/send/1614863178168 HTTP/1.0" "Synapse/1.29.0rc1 (b=matrix-org-hotfixes,61a970e25)" [3 dbevts]
@richvdh
Copy link
Member Author

richvdh commented Mar 26, 2021

this room is continuing to cause severe problems on my server. I'm going to leave it to try to make federation work again.

@ShadowJonathan
Copy link
Contributor

(Doing /join on that room, it looks to be the "IRC Matrix Bridges" room, for anyone's reference, #irc:matrix.org)

@richvdh
Copy link
Member Author

richvdh commented Aug 24, 2021

The main problem here is that _check_event_auth, and its helper _update_auth_events_and_context_for_auth, make incorrect assumptions about whether the auth events being passed in in claimed_auth_event_map have themselves been authed. Unfortunately it's not quite as simple as just checking that they have been authed, because the whole reason we have claimed_auth_event_map (rather than just pulling the events out of the db) is that they haven't been persisted, either.

Really, we need to get rid of the whole premise of claimed_auth_event_map - we should not be persisting events before their auth_events are persisted. Unfortunately, if we just remove it, auth_events does something completely different wherein it tries to work out what the auth events should be based on the current state of the room.

The two methods are also very confused about whether the auth events they are working with are those according to the auth_events of the event being persisted, or what we think the auth events should be given the state of the room at that point (which obviously only makes sense for non-outliers, not that that stops us trying to do it anyway).

@reivilibre reivilibre added T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. S-Major Major functionality / product severely impaired, no satisfactory workaround. labels Aug 31, 2021
richvdh added a commit that referenced this issue Sep 30, 2021
When we consider whether to accept events, we should not accept those which
depend on rejected events for their auth events.

This (together with earlier changes such as
#10771 and
#10896) forms a partial fix to
#9595. There still remain code
paths where we do not check the `auth_events` at all.
richvdh added a commit to matrix-org/complement that referenced this issue Oct 1, 2021
We should make sure that events that refer to rejected events in their
auth_events are themselves rejected.

See matrix-org/synapse#9595.
richvdh added a commit that referenced this issue Oct 5, 2021
When we consider whether to accept events, we should not accept those which
depend on rejected events for their auth events.

This (together with earlier changes such as
#10771 and
#10896) forms a partial fix to
#9595. There still remain code
paths where we do not check the `auth_events` at all.
richvdh added a commit to matrix-org/complement that referenced this issue Oct 6, 2021
We should make sure that events that refer to rejected events in their
auth_events are themselves rejected.

See matrix-org/synapse#9595.
@richvdh
Copy link
Member Author

richvdh commented Oct 7, 2021

So I think we now have PRs that should stop this happening again in the future. The next question is whether we can do anything about existing brokenness in peoples' databases.

@ShadowJonathan
Copy link
Contributor

The next question is whether we can do anything about existing brokenness in peoples' databases.

I actually had a heated discussion about this in #matrix-spec yesterday, the consensus (and logical decision) is to - when re-validating - throw away the entire history of a room from that point on.

Here is a link to the discussion, but the talk about this requirement and me wrestling with it and trying to find an alternative is a bit further up and down from that point.

Thank you for clarifying that the issue has been fixed, though.

richvdh added a commit that referenced this issue Oct 18, 2021
Currently, when we receive an event whose auth_events differ from those we expect, we state-resolve between the two state sets, and check that the event passes auth based on the resolved state.

This means that it's possible for us to accept events which don't pass auth at their declared auth_events (or where the auth events themselves were rejected), leading to problems down the line like #10083.

This change means we will:

 * ignore any events where we cannot find the auth events
 * reject any events whose auth events were rejected
 * reject any events which do not pass auth at their declared auth_events.

Together with a whole raft of previous work, this is a partial fix to #9595.

Fixes #6643.

Based on #11009.
richvdh added a commit that referenced this issue Oct 25, 2021
This is the final piece of the jigsaw for #9595. As with other changes before this one (eg #10771), we need to make sure that we auth the auth events in the right order, and actually check that their predecessors haven't been rejected.

To do this I've reused the existing code we use when persisting outliers elsewhere.

I've removed the code for attempting to fetch missing auth_events - the events should have been present in the send_join response, so the likely reason they are missing is that we couldn't verify them, so requesting them again is unlikely to help. Instead, we simply drop any state which relies on those auth events, as we do at a backwards-extremity. See also matrix-org/complement#216 for a test for this.
@callahad
Copy link
Contributor

callahad commented Nov 4, 2021

@richvdh #11012 described itself as "the final piece of the jigsaw for #9595"

With that merged, can we can close this issue?

@richvdh
Copy link
Member Author

richvdh commented Nov 4, 2021

I guess so. The problem is that we're still going to see this on an ongoing basis until we clear out existing problematic data.

@worldofgeese
Copy link

Is this related to a batch of rooms I can't join that I saved the join links to that when pressed all give "Auth events could not be found"?

Can someone help me clear the database entries in my postgres db so I can join these rooms?

I can join other rooms but once I've "poisoned" a room with a join link then I can't enter that room even if I manually find it through the room browser.

Here are the join links (beware they will probably break the rooms for you too)

https://matrix.to/#/!EoRhMvNpnWxCMTMPeP:libera.chat?via=geese.party&via=libera.chat&via=matrix.org

https://matrix.to/#/!YLTeaulxSDauOOxBoR:matrix.org?via=geese.party&via=gitter.im&via=matrix.org

https://matrix.to/#/!hokCjFXtQcxTAIXSdZ:matrix.org?via=geese.party&via=matrix.org&via=privacytools.io

https://matrix.to/#/!GryYovOTNVgikENmcX:libera.chat?via=geese.party&via=libera.chat&via=matrix.org

https://matrix.to/#/!NicAJNwJawmHrEhqZs:matrix.org?via=geese.party&via=matrix.org&via=nordgedanken.dev

@parisni
Copy link

parisni commented Jan 10, 2022

I guess so. The problem is that we're still going to see this on an ongoing basis until we clear out existing problematic data.

Do we have a way to fix the database ? I have several channel with this issue. @richvdh

UPDATE: I did upgrade the room. People can join the new room. However the previous room still is not accesible from Click here to show older message

@pidongqianqian
Copy link

I guess so. The problem is that we're still going to see this on an ongoing basis until we clear out existing problematic data.

Do we have a way to fix the database ? I have several channel with this issue. @richvdh

UPDATE: I did upgrade the room. People can join the new room. However the previous room still is not accesible from Click here to show older message

@parisni What do you mean “upgrade room”? could you tell how to do that? i have same issue and i just want can join new room.

@parisni
Copy link

parisni commented Feb 17, 2023

Upgrade a room

This will make the current room read-only, and create a new room see
rfc
.

Simply type this as a message (you will be asked to invite every participant) :

/upgraderoom 7

Run an api request :

curl -H 'Authorization: Bearer <token-access>' -H "Content-Type: application/json"  -X POST https://matrix.interhop.org/_matrix/client/r0/rooms/<room-id-url-encoded>/upgrade -d '{"new_version": "6"}'
  • token-access: In element > All settings > Help about > Access Token (bottom page)
  • room-id: In element > Room Info

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
S-Major Major functionality / product severely impaired, no satisfactory workaround. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues.
Projects
None yet
Development

No branches or pull requests

7 participants