can't federate with matrix.org #17678
Comments
|
There's no chance that it's this right ? |
We tried the steps mentioned in the ticket but it didn't help thus it seems we face a different issue. |
|
@erikjohnston Any idea? |
|
You'll need to figure out what exactly is being GET that is too large. There must be more data in the logs. |
|
I created a non-encrypted room to isolate the issue, and there I see that messages from matrix.org -> nitro.chat get delivered immediately, but messages from nitro.chat -> matrix.org are delayed by about 1 day. Could be a separate issue, though. |
|
Some observations: All failed connections to This is the last example: |
|
I found a workaround: if I divide all the PDU/EDU limits in [ ... 340 transactions with 40 EDUs, all successful, matrix.org replied with 200 OK ... ] Then this one is failing again So it looks like we try to deliver pending EDUs, and at some point After another restart, it again shortly went up to 40 EDUs, but then recovered: Since then no incident anymore (~18 hours). Also note that all transactions with 40 EDUs are done by federation_transaction_transmission_loop-7 and federation_transaction_transmission_loop-4, probably loops who deliver pending EDUs, and after that, every transaction is done by different loops. Here's the patch I applied, that seems to mitigate the issue: --- /opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/federation/sender/per_destination_queue.py 2024-09-02 18:16:07.000000000 +0200
+++ per_destination_queue.py 2024-09-13 19:36:30.462914769 +0200
@@ -50,7 +50,7 @@
import synapse.server
# This is defined in the Matrix spec and enforced by the receiver.
-MAX_EDUS_PER_TRANSACTION = 100
+MAX_EDUS_PER_TRANSACTION = 50
logger = logging.getLogger(__name__)
@@ -71,7 +71,7 @@
# Limit how many presence states we add to each presence EDU, to ensure that
# they are bounded in size.
-MAX_PRESENCE_STATES_PER_EDU = 50
+MAX_PRESENCE_STATES_PER_EDU = 25
class PerDestinationQueue:
@@ -790,7 +790,7 @@
# Now we look for any PDUs to send, by getting up to 50 PDUs from the
# queue
- self._pdus = self.queue._pending_pdus[:50]
+ self._pdus = self.queue._pending_pdus[:25]
if not self._pdus and not pending_edus:
return [], [] |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
We reported the federation issue #17371 which got fixed. Now we face a similar issue. Users at matrix.org can't decrypt messages from users at nitro.chat. It seems that other home servers are not affected. We think that the cause is at matrix.org as last time.
Steps to reproduce
Users at matrix.org can't decrypt messages from users at nitro.chat
Homeserver
nitro.chat
Synapse Version
1.114.0
Installation Method
Debian packages from packages.matrix.org
Database
PostgreSQL
Workers
I don't know
Platform
Ubuntu
Configuration
No response
Relevant log output
In the homeserver.log are some errors like: 2024-08-20 18:54:08,085 - synapse.http.matrixfederationclient - 1658 - WARNING - GET-2206194- {GET-O-110526} [matrix.org] Requested file is too large > 52428800 bytesErrors from the federation_sender.log are:
The text was updated successfully, but these errors were encountered: