Documentation and Config updates #142

jaley · 2023-06-30T11:12:44Z

This PR finished up the V3 release by adding docs for new functionality, removing irrelevant and superseded docs and config and fixing the local docker build.

@paddybyers and @tomczoink -- I added you because the README changes give you a decent summary of what's new and thought it might be good for you to review that.

…w that we support DLQs properly

AndyTWF

From a high level this looks good - will wait on a review from someone more familiar with the changes before I merge it :)

tomczoink · 2023-07-06T11:31:41Z

README.md

-| client.loglevel                   | Sets the verbosity of logging.                                                                                                                                                                                                  | *Integer* | 0       |
+| Property                          | Description                                                                                                                                            | Type      | Default |
+|-----------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|---------|
+| message.name                      | Ably message name to publish. Can be a pattern, as per `channel` above.                                                                                | *String*  |         |


I believe the context for patterns is in Dynamic Channel Configuration, with channel just directing to that section, so we could just direct to that section instead.

Done in aba1da6

tomczoink · 2023-07-06T11:32:35Z

README.md

+| Property                          | Description                                                                                                                                            | Type      | Default |
+|-----------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|---------|
+| message.name                      | Ably message name to publish. Can be a pattern, as per `channel` above.                                                                                | *String*  |         |
+| batchExecutionThreadPoolSize      | The maximum number of parallel outgoing REST API requests to publish content to Ably                                                                   | *Integer* | 10      |


For all of these limits are there any additional maximums and minimums to what these values can be?

For batch publishing, you'd be limited by the maximum payload size limits here:

https://ably.com/docs/api/rest-api#batch-publish

So it would be a case-by-case basis

Added a line about the limits: 162248a

README.md

tomczoink · 2023-07-06T11:49:22Z

src/main/java/com/ably/kafka/connect/config/ChannelSinkConnectorConfig.java

@@ -154,12 +136,12 @@ public class ChannelSinkConnectorConfig extends AbstractConfig {
    public static final String BATCH_EXECUTION_THREAD_POOL_SIZE_DEFAULT = "10";

    public static final String BATCH_EXECUTION_MAX_BUFFER_SIZE = "batchExecutionMaxBufferSize";
-    public static final String BATCH_EXECUTION_MAX_BUFFER_SIZE_DEFAULT = "1000";
+    public static final String BATCH_EXECUTION_MAX_BUFFER_SIZE_DEFAULT = "100";


This is 100, but in https://github.com/ably/kafka-connect-ably/pull/142/files#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R223 (which I believe is what this is the default value for) it's still 1000.

Done in 83672a0

tomczoink · 2023-07-06T11:50:16Z

src/main/java/com/ably/kafka/connect/config/ChannelSinkConnectorConfig.java

    private static final String BATCH_EXECUTION_MAX_BUFFER_SIZE_DOC = "Size of the buffer, records " +
        "are buffered or chunked before calling the Ably Batch REST API";

    public static final String BATCH_EXECUTION_MAX_BUFFER_DELAY_MS = "batchExecutionMaxBufferSizeMs";
-    public static final String BATCH_EXECUTION_MAX_BUFFER_DELAY_MS_DEFAULT = "5000";
+    public static final String BATCH_EXECUTION_MAX_BUFFER_DELAY_MS_DEFAULT = "100";


This is 100, but in https://github.com/ably/kafka-connect-ably/pull/142/files#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R224 (which I believe this is the default value for) it's still 5000.

Done in 83672a0

Co-authored-by: Tom Camp <tc13270@my.bristol.ac.uk>

tomczoink

Looks good, thanks!

jaley added 4 commits June 30, 2023 09:53

Remove irrelevant config and docs

b4084dd

Remove the KeyCheckSMT and related docs, this will cause confusion no…

61a1e08

…w that we support DLQs properly

Update documentation for 3.0 release

d452864

Update target paths for local docker builds

d329a1a

jaley requested review from lmars, paddybyers and tomczoink June 30, 2023 11:12

jaley added this to the V3 - Ably Batch Publishing milestone Jun 30, 2023

github-actions bot temporarily deployed to staging/pull/142/kafka-connect-ably June 30, 2023 11:20 Inactive

github-actions bot temporarily deployed to staging/pull/142/kafka-connect-ably-msk-plugin June 30, 2023 11:20 Inactive

AndyTWF approved these changes Jul 4, 2023

View reviewed changes

lmars approved these changes Jul 5, 2023

View reviewed changes

tomczoink suggested changes Jul 6, 2023

View reviewed changes

AndyTWF and others added 2 commits July 6, 2023 13:11

Fix typo

6255e52

Co-authored-by: Tom Camp <tc13270@my.bristol.ac.uk>

Fix typo

3aeab4a

Co-authored-by: Tom Camp <tc13270@my.bristol.ac.uk>

github-actions bot temporarily deployed to staging/pull/142/kafka-connect-ably July 6, 2023 12:16 Inactive

github-actions bot temporarily deployed to staging/pull/142/kafka-connect-ably-msk-plugin July 6, 2023 12:16 Inactive

AndyTWF added 2 commits July 6, 2023 13:19

Update reference to dynamic channel configuration

aba1da6

Update default values in README

83672a0

github-actions bot temporarily deployed to staging/pull/142/kafka-connect-ably July 6, 2023 12:26 Inactive

github-actions bot temporarily deployed to staging/pull/142/kafka-connect-ably-msk-plugin July 6, 2023 12:26 Inactive

Add maximum batch size reference

162248a

tomczoink approved these changes Jul 6, 2023

View reviewed changes

github-actions bot deployed to staging/pull/142/kafka-connect-ably July 6, 2023 12:56 View deployment

github-actions bot deployed to staging/pull/142/kafka-connect-ably-msk-plugin July 6, 2023 12:56 View deployment

AndyTWF merged commit 8bcbf80 into integ_v3 Jul 6, 2023
1 check passed

AndyTWF deleted the v3_cleanup branch July 6, 2023 12:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Documentation and Config updates #142

Documentation and Config updates #142

jaley commented Jun 30, 2023

AndyTWF left a comment

tomczoink Jul 6, 2023

AndyTWF Jul 6, 2023

tomczoink Jul 6, 2023

AndyTWF Jul 6, 2023

AndyTWF Jul 6, 2023

tomczoink Jul 6, 2023

AndyTWF Jul 6, 2023

tomczoink Jul 6, 2023

AndyTWF Jul 6, 2023

tomczoink left a comment

Documentation and Config updates #142

Documentation and Config updates #142

Conversation

jaley commented Jun 30, 2023

AndyTWF left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tomczoink left a comment

Choose a reason for hiding this comment