Idempotent Batch Publishing

[jaley]

Currently we do not attach the message.id field to outgoing Ably messages. This is because the Ably Realtime service does not support idempotent publishing in batches. Once this feature has been implemented on the realtime side, we can simply start populating this field again for exactly-once delivery guarantees. This is particularly useful as duplication of messages is currently very likely when the connector is restarted or autoscaled, etc.

The message.id needs to be set to a value that is deterministically derived from the source SinkRecord, otherwise delivery will not be idempotent. The (topic, partition, offset) triple should uniquely identify the SinkRecord and can simply be formatted into a string identifier for use as message.id.

[sync-by-unito]

➤ Automation for Jira commented:

The link to the corresponding Jira issue is https://ably.atlassian.net/browse/SDK-3708