Backpressure between components

[mxinden]

What is backpressure

A slow consumer should slow down (i.e. backpressure) a fast producer.

Why do we need backpressure

• Prevent unbounded growth of buffers.
• Potential DOS defense.
• Potential lower latencies.
• Potential better resource allocation.

See also coding guidelines - Bound everything.

Where do we enforce backpressure

• User -> Swarm

  • No backpressure
  • Useful when dialing a lot of peers via Swarm::dial.
  • ConnectionLimit enforces boundedness at least in the Pool.
  • Might not be worth fixing, i.e. bursts might be fine.

• User -> NetworkBehaviour

  • No backpressure
  • User can access NetworkBehaviour via Swarm::behaviour_mut
  • Useful e.g. when doing large amounts of DHT lookups via libp2p-kad, many libp2p-request-response requests, ...
  • Potential solution
    • In the case of libp2p-kad have a Kademlia::poll_find_closest_ready that needs to be polled before Kademlia::find_closest similar to Sink::poll_ready.

• Swarm -> NetworkBehaviour

  • No backpressure
  • Only system notification events (e.g. connection established), thus not important.
  • Inbound connection
    • Can only be droppped
    • We have static limits via Swarm
    • In the future, with generic connection management, this can be dynamic, e.g. based on your system memory swarm/: Support generic connection management #2824

• NetworkBehaviour -> ConnectionHandler

  • No backpressure
  • Swarm does not poll NetworkBehaviour in case it could not deliver previously returned event from the NetworkBehaviour to the destined ConnectionHandler.
    

    rust-libp2p/swarm/src/lib.rs

    Lines 1032 to 1034 in f9b4af3

    	match this.pending_event.take() {
    	// Try to deliver the pending event emitted by the [`NetworkBehaviour`] in the previous
    	// iteration to the connection handler(s).

  • NetworkBehaviour is blocked on single slow ConnectionHandler
  • Potential solutions
    • Drop the event
      • NetworkBehaviour already needs to handle the case where the connection closes and thus the event is never delivered to the ConnectionHandler.
      • When sending two events, first might be dropped (ConnectionHandler busy) while the second might be delivered. Not intuitive.
    • Return event back to the NetworkBehaviour
    • Add NetworkBehaviour::poll_connection_handler_event, providing a list of ConnectionIds with ConnectionHandlers that are ready to receive another event.

• NetworkBehaviour -> Swarm

  • NetworkBehaviourAction::Dial

• ConnectionHandler -> NetworkBehaviour

  • Backpressure
  • ConnectionHandler is blocked on slow NetworkBehaviour
  • Expected behaviour

• Connection -> ConnectionHandler

  • ConnectionHandler::inject_fully_negotiated_inbound
    • Inbound can only drop
    • how about poll_inbound_ready
    • Note that multistream-select needs to run first, otherwise we can't map to a specific ConnectionHandler
  • ConnectionHandler::inject_fully_negotiated_outbound
    • how about poll_outbound_substream
    • Or the ConnectionHandler needs to internally enforce a limit of pending outbound connections. (Lots of duplication given the amount of ConnectionHandlers)
  • ConnectionHandler::inject_event
    • No backpressure
    • Possible solution
      • Add ConnectionHandler::poll_inject_event_ready

• ConnectionHandler -> Connection

  • ConnectionHandlerEvent::OutboundSubstreamRequest
    • Can only be dropped

• Stream

  • Creation of outbound streams
    • Backpressure is only enforced by QUIC
    • Potentially one day by Yamux (Track the accept backlog and add an upper limit rust-yamux#150)
    • Potentially one day with qmux - a new stream muxer based on QUIC specs#394
    • For now, we need to drop streams
  • Bytes on stream
    • Fine with Yamux, QUIC and potentially one day qmux
    • No backpressure on streams with mplex connection/README: Recommend yamux over mplex specs#402

Related resources

• [libp2p-swarm] Opt-in connection-oriented backpressure for NetworkBehaviours. #1602
• Swarm does not honour max_negotiating_inbound_streams setting #3041
• https://github.com/libp2p/rust-libp2p/blob/master/docs/coding-guidelines.md

[thomaseizinger]

Relevant discussion happening here: #3411.

[mxinden]

Cross-referencing backpressure tracking issue for Kademlia here #3710.

[thomaseizinger]

Cross-referencing a discussion around backpressure: #4585.