Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add dos and monitoring docs #160
Add dos and monitoring docs #160
Changes from 10 commits
626b5ff
f475a8c
da36b11
104dbbc
6752ff4
e684b07
300fcf3
7b6c544
a6a83a5
dc691dd
c192206
File filter
Filter by extension
Conversations
Jump to
There are no files selected for viewing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few nitpics here:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In go-libp2p-core they're called
ConnManager
andResourceManager
. UsingConn Manager
feels weird and so doesResourceManager
although happy to make one change or the other if you feel strongly.I think our template should support the fixed width + hyperlink. I'll see if I can fix it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed this by adding a
text-decoration: underline
css property.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a rust-libp2p link.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we give pointers on how to do this?
For go-libp2p this means using resource manager right?
For rust, this isn't at the connection level, but somewhere I think we should be linking to https://docs.rs/libp2p/latest/libp2p/swarm/struct.SwarmBuilder.html#method.max_negotiating_inbound_streams . Maybe we say, "rust-libp2p relies on each protocol to limit the number of streams per connection in XXX. A global upperbound on negotiating/transient inbound streams can be set using https://docs.rs/libp2p/latest/libp2p/swarm/struct.SwarmBuilder.html#method.max_negotiating_inbound_streams."
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I should be more clear.
Here I'm talking about limiting the number of concurrent streams you need by design of the protocol, as opposed to using an existing protocol and trying to limit the streams at the end. For example imagine a RPC style protocol whose procedures are async and often take a long time to return (say > 1min). Here are two ways you could implement it:
Assume you make a lot of concurrent calls, method 1 would result in a large number of concurrent and mostly inactive streams. Method 2 would result in a fewer number of concurrent streams, and thus lower memory footprint.
If you add a limit here of say 10 streams, then method 1 will mean you can only have 10 concurrent RPC calls, while method 2 would let you have a much larger number of concurrent RPC calls.
Does that make sense? I should rephrase this to focus on the fact this is about protocol design (the inception stage of a p2p application) not about the deployed stage.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got it - makes sense.
Side: lets find/create a place to point to https://docs.rs/libp2p/latest/libp2p/swarm/struct.SwarmBuilder.html#method.max_negotiating_inbound_streams . Maybe there's a section about transient/negotiating connections and resources and that those should guarded against too. Go and Rust both have some protections here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will this link survive after the soon-coming repo consolidation?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have a comment to update this, but worst case will force a user to click through another link.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we give guidance on when to use this mechanism vs. the resource manager? I see this is a good hook for custom logic, but it seems like what Prysm is doing could be covered by go-libp2p resource manager right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, but I could add something here.
Not really. If you're trying to avoid an adversary that can connect to you and give you a ton of work to do all at once the rcmgr doesn't protect at all. This attack can easily be mitigated by rate limiting though.
Not all applications will want this rate limiting, or they may want to rate limit certain things (e.g. something in the protocol rather than in the connections). For example, if I'm Google I wouldn't want to rate limit any new connection to me. I would rather rate limit work per connection.
Should the rcmgr do this? I don't think so. It's not directly related to limiting the resources being used, and if it can be handled by a smaller component that already exists the better.
I hope that makes sense, but happy to expand more as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixing the rendering issue here: https://bafybeid4zqncc4v5epc4urfvl5ajgnmqeksksk4xrgabdwaxxswpsigh6y.on.fleek.co/reference/dos-mitigation/#leverage-the-resource-manager-to-limit-resource-usage-go-libp2p-only
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's a misuse of
(foo)[bar]
vs[foo](bar)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had a question about this in https://www.notion.so/pl-strflt/Guide-for-how-to-respond-to-resource-exhaustion-attacks-b10f55cc9a3d4917ae80c9b914e05e8c.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Answered. Copied here:
Exactly. fail2ban expands this to a regex that captures the host
(?:::f{4,6}:)?(?P<host>\S+)
. See https://www.fail2ban.org/wiki/index.php/MANUAL_0_8#FiltersThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had some comments about this section and format/layout changes in I had a question about this in https://www.notion.so/pl-strflt/Guide-for-how-to-respond-to-resource-exhaustion-attacks-b10f55cc9a3d4917ae80c9b914e05e8c.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Answered
By the filename. The filter is in a file go-libp2p-peer-status.conf and this rule references filter=go-libp2p-peer-status.
I'll leave a comment on the relevant line as well