Proposal: Scaling the relay

[thegostep]

Relay scaling proposal (v3)

This proposal aims to discuss various approaches to improving scaling of the relay during periods of heavy load.

context

The constant growth in usage of the flashbots relay has resulted in intermittent service degradation during periods of heavy load. We've observed an increasing rate of DOS attempts aiming to take down the relay during time periods with significant MEV opportunities. As such, it is time to introduce options to improve relay reliability and scaling.

design goals

1. ddos mitigation
   It's ok if less valuable bundles (measured by bundle score) get dropped during high load as long as the system still functions for some good users.
2. costs
   Hosting costs should scale (at most) linearly with real demand - don't want to allow costs to scale infinitely for prolonged periods of time. If we align fees such that the bundles pay/burn as much as our costs, then that's probably ok as well.
3. fairness
   Avoid barrier to entry for new searchers as much as possible / avoid sustained advantage to a subset of searchers.

proposals

Searcher Reputation

Users continue to use signing address, and we come up with some threshold for moving good users into a high priority queue. This queue operates in parallel to the current simulation/miner relay. We just need to make sure that the initial step which sorts users into high/low priority queues never goes down, which should be very doable. The relay has access to historical bundle data, inclusion rate, and miner payment which can be used to identify good users.

pros/cons

• unclear what the threshold for moving searchers to high prio should be
  • if it's too easy to be in high prio, then attacking becomes easy
  • if it's too hard, there comes a question of fairness
• transparency becomes a big problem. We need to be very clear when/why users move into/out of high prio
• low prio queue can still get overwhelmed and go down
• somewhat centralized. if we opened up the full bundle data, we could maybe make this less centralized
• no changes required on the searcher side

concrete example

• we determine a threshold for a "good" signing address. Let's say for instance, an address that has landed a highly profitable bundle in more than 3 blocks in the last 24 hours and whose rate of requests is below 2 per second
• the bundles from these users will be filtered out and sent to a independent pipeline for simulation/relaying to miners
• other users bundles will go to a similar, but completely independent pipeline
• the idea is that attackers won't end up in the "good" queue, and even if they do, we can add rules to move them out
• the threshold for "good" users might end up being dynamic based on load, e.g. we would tighten the threshold to 10 bundles landed in the last 24 hours, or bundle rates below 1 per second

Account Quotas

Users sign up for an account. They will either need to pay for the account, provide proof of work, stake in a contract, and/or wait some amount of time. This can be done implicitly in the relay without an explicit sign up step, but we need to associate some cost with the account creation for this to work.

pros/cons

• pay to play model increases barriers to entry / reduces fairness
• hard to figure out the exact amount users should pay/stake upfront to align with ongoing costs
  • amount too high may cut off large searcher population / non-mev use cases as willingness to pay likely follows power law distribution
  • amount too low may not stop motivated actors from performing dos
• still need a free tier for entry level users
• very centralized / permissioned
• rate limiting is super clear and easy to enforce with accounts

Micro-Fee

Each bundle includes a separate fee transaction that we either burn or send along to the miner. Could be either l2/state channel, we just need to make sure whatever we use is protected against double spend/replay.

pros/cons

• requires searchers to pay
• might be very complicated and too big a hurdle for searchers to use
• still weak to ddos if fee is too low
  • fee might need to be dynamic
    • oh god, we've re-created the txpool
• big questions to answer on how the payment actually works
• easy to decentralize relay
• aligns incentives nicely

concrete example

• for simplicity sake, lets assume we use a state channel that we own/operate. This is pretty much equivalent and interchangeable with having a staking contract on L1 ethereum and having time locks/rollups, if that option is better
• users open a state channel with us. This is owned/operated by us
• each bundle has a transaction fee that the user chooses, which will be burned (i.e. provably destroyed on L1 chain) at some point, although it might not be immediate
• we will set the minimum allowed fee dynamically, matching our costs and the opportunity cost of ddos'ing the block for everyone else. This can surge very high during periods of high load, since our geth cluster can take time to catch up to the load

[pyggie]

From my perspective as a searcher, the ideal outcome would be some kind of guaranteed, reserved capacity. I'd be happy with a system where I pay $X/day to be guaranteed the ability to submit K bundles/block even at the most congested moment. You could use that $X to provision enough capacity to always handle my requests.

From the same perspective, I hate the reputation idea because it adds yet another dimension of risk, uncertainty, and opacity. You'll have a never-ending cat-and-mouse game developing countermeasures to manipulation attempts, and these countermeasures will invariably snare honest, hard-working searchers. You will be tempted to withhold explanations of why a searcher has a low reputation out of fear that transparency will help the manipulators. Just the example given, where a "good" reputation requires landing bundles within the last 24 hours, eliminates an entire class of very-high-value searchers (liquidators) who may see weeks go by without an opportunity. Please observe Amazon and eBay's endless labors over the past 20 years and consider whether that is how you want to spend your time.

[Code0x2]

I like the idea of paying for the guaranteed ability, but that really depends on what % of the time the relay is congested, like if its congested 75% of the time and mostly paying bots are getting their tx's in then that creates a entry gap for casual searchers who find one or two opps in the blue, or those who are just trying it out for the sake of it. Wouldn't be too bad if congestion time was only around 5-15% tho.

[thegostep]

The biggest issue the relay is facing right now are intermittent dos attempts which take up all the relay capacity - regular usage has not been an issue. This means paying searchers won't crowd out non-paying users.

I think the solution will end up being to provide multiple paths to inclusion for searchers. Just like a good mmorpg, players may want to xp farm their way to high reputation, or they may want to take the easy path and pay to win ;)

Here is what that could look like:

• users default to being in the low reputation queue and risk getting crowded out
• low reputation searcher can play the reputation game and eventually move to high reputation queue if they have good performance over time
• low reputation searcher can do a donation or burn to boost their reputation, but risk dropping back down to low reputation if don't perform well
• any searcher can pay for guaranteed capacity in the high reputation queue

[0xprincess]

I have some ideas about the scaling, some of them I discussed in discord, but probably it's better to discuss it here.
The problem with current proposal (high- and low- priority queues), that I see, is the incentive for searchers attached to high-priority queue, to spam the low-priority queue with spam bundles from other accounts. It will make the execution of legit bundles of below-average reputation searchers (including new searchers with zero reputation) by the relay much harder, because most of the executed bundles inside the time limit will be spam bundles.
Imagine the worst-case scenario: the low-priority queue is filled with 10000 bundles, and 100 of them is actually legit bundles. And the relay can execute only 100 to fit in the time limit. There's a great chance that none of the executed bundles will be legit.
So this spam attack greatly reduces competition for high-priority queue searchers.

Here's the steps that, in my opinion, can prevent such attack:

1. Introduce one queue for bundles, sorted by searcher's reputation. So the spam accounts will quickly get zero repuation and will be on the bottom of a queue.
2. Give some base reputation (more than zero) to new searchers, that can be naturally increased or decreased later as a result of their actions. This will place the new searchers above the known spammers with zero reputation.
3. Use some approach to prevent spawning new id's in large amounts. The potential solution to this problem, as I mentioned in discord, could be issuing NFT's to searcher accounts. Minting of such identity token will cost some gas and eth, so it's economically and practically impossible for spammers to acquire a lot of new id's to spam the queue, but it will not be a problem for the legit searchers.

[0xprincess]

And the other approach that came to my mind recently, that will probably help to scale the relay, is to oblige searchers to include the stateDiff of their bundle, simulated on their side. If will help relay to sort bundles by profit, filter duplicates and intersecting bundles, etc., without an actual execution of a bundle on the relay.
This is not a complete solution, since it's necessary to use some additional method or checking strategy, to make sure that this stateDiff is correct.
Here's some simple one as an example: Skip bundle execution for searchers, which repuation is above the threshold. This is a good solution if we assume that for searcher, the best strategy is to always follow the rules, to be effective.

The best solution probably is to generate some zk proof for the execution result, but I don't really know if it's possible in such case.

[sambacha]

the lack of any sort of flashbots provided data metrics makes the task of improving the reputation system difficult. e.g. without knowing uptime information its hard to gauge the amount of DDoS attacks and therefore how much weight can be given to issues such as the one @SebyakinAndrei has described