Skip to content

Latest commit

 

History

History
102 lines (90 loc) · 4.19 KB

tip-550.md

File metadata and controls

102 lines (90 loc) · 4.19 KB
Raw
Tip: 550
Title: P2P message snappy compression
Author: allen.cheng@tron.network
Status: Final
Type: Standards Track
Category: Networking
Date: 2023-05-22

Simple Summary

This TIP proposes a small extension to the TRON protocol to enable Snappy compression on all message payloads after the initial handshake.

Abstract

The base networking protocol used by TRON currently does not employ any form of compression. This results in a massive amount of bandwidth wasted in the entire network, making both initial sync and regular operation slower and laggier. This situation can be improved by compressing network messages. After extensive benchmarks, the results show that data traffic is decreased by 40-60% for block synchronization.

Motivation

Currently, the block size of the TRON ​​network is around 80k bytes. And as the TRON ​​network continues to grow, the block size may continue to increase, the network load will also increase, and the bandwidth requirements of nodes will also increase. However, most data (blocks, transactions) are highly compressible. By enabling compression at the message payload level, we can reduce the amount of data transmitted by the message, reducing bandwidth usage and data synchronization latency.

Rationale

Enable Snappy compression on all TCP channel message payloads after the initial handshake. Compress the message before sending it through the channel. When receiving a compressed message, decompress it first and then continue the subsequent business processing. The handshake message is never compressed, since it is needed to negotiate the common version.

Specification

Pack the compressed message into CompressMessage for network transmission. The structure of CompressMessage is as follows:

message CompressMessage {
  enum CompressType {
    uncompress = 0;
    snappy = 1;
  }
  CompressType type = 1;
  bytes data = 2;
}

Implementation

Snappy dependency

implementation group: 'org.xerial.snappy', name: 'snappy-java', version: '1.1.8.4'

Compression implementation when sending messages:

public void send(Message message) {
  if (finishHandshake) {
    data = ProtoUtil.compressMessage(message.getSendData()).toByteArray();
  }
  ...
}

public static Connect.CompressMessage compressMessage(byte[] data) throws IOException {
    Connect.CompressMessage.CompressType type = Connect.CompressMessage.CompressType.uncompress;
    byte[] bytes = data;

    byte[] compressData = Snappy.compress(data);
    if (compressData.length < bytes.length) {
      type = Connect.CompressMessage.CompressType.snappy;
      bytes = compressData;
    }

    return Connect.CompressMessage.newBuilder()
            .setData(ByteString.copyFrom(bytes))
            .setType(type).build();
  }

Decompression implementation when receiving a compressed message:

@Override
protected void decode(ChannelHandlerContext ctx, ByteBuf buffer, List<Object> out) {
    byte[] data = new byte[buffer.readableBytes()];
    buffer.readBytes(data);
    if (channel.isFinishHandshake()) {
      data = ProtoUtil.uncompressMessage(Connect.CompressMessage.parseFrom(data));
    }
    ...
}

public static byte[] uncompressMessage(Connect.CompressMessage message) throws IOException {
    if (message.getType().equals(Connect.CompressMessage.CompressType.uncompress)) {
      return message.getData().toByteArray();
    } else {
      return Snappy.uncompress(message.getData().toByteArray());
    }
}

Backward Compatibility

This proposal is fully backward compatible. Clients upgrading to the proposed new version protocol should still support skipping the compression step for older version protocol connections.

Test

Download blocks with block heights from 46314350 to 46314360, and perform a compression and decompression test. The compression rate is about 51%. The specific test results are as follows:

Block size (byte) Compressed size (byte) Compression rate Compression/decompression time（ms）
89803 44361 0.50 <1
104504 50926 0.51 <1
70970 35247 0.50 <1
96544 46905 0.51 <1
82820 41005 0.50 1
83987 41878 0.50 <1
82296 40566 0.51 <1
77820 39108 0.50 <1
73715 36027 0.51 <1
90632 43918 0.52 <1