More detailed partition reconfiguration tracking #10201

mmaslankaprv · 2023-04-19T14:57:56Z

Enriched /reconfiguration API with more information allowing users to track the progress of partition reconciliation. Now the API returns a complete set of information related with partition reconfiguration that is taking place.

The API will now return the following JSON:

{
    "ns": "kafka",
    "topic": "topic-khbikkrzeo",
    "partition": 9,
    "previous_replicas": [
        {
            "node_id": 2,
            "core": 0
        },
        {
            "node_id": 3,
            "core": 0
        },
        {
            "node_id": 1,
            "core": 0
        }
    ],
    "current_replicas": [
        {
            "node_id": 4,
            "core": 0
        },
        {
            "node_id": 3,
            "core": 0
        },
        {
            "node_id": 1,
            "core": 0
        }
    ],
    "bytes_left_to_move": 190,
    "bytes_moved": 0,
    "partition_size": 190,
    "reconciliation_statuses": [
        {
            "node_id": 2,
            "operations": [
                {
                    "type": "update",
                    "core": 0,
                    "retry_number": 7,
                    "revision": 89,
                    "status": "Generic failure occurred during partition operation execution (cluster::errc:52)"
                }
            ]
        },
        {
            "node_id": 1,
            "operations": [
                {
                    "type": "update",
                    "core": 0,
                    "retry_number": 3,
                    "revision": 89,
                    "status": "Current node is not a leader for partition (cluster::errc:17)"
                }
            ]
        },
        {
            "node_id": 4,
            "operations": [
                {
                    "type": "update",
                    "core": 0,
                    "retry_number": 5,
                    "revision": 89,
                    "status": "Current node is not a leader for partition (cluster::errc:17)"
                }
            ]
        },
        {
            "node_id": 3,
            "operations": [
                {
                    "type": "update",
                    "core": 0,
                    "retry_number": 5,
                    "revision": 89,
                    "status": "Current node is not a leader for partition (cluster::errc:17)"
                }
            ]
        }
    ]
}

FIxes: https://github.com/redpanda-data/core-internal/issues/444

Backports Required

Release Notes

Improvements

more observability in partition reconfigurations

emaxerrno · 2023-04-19T16:01:57Z

@mmaslankaprv - what's the difference between "shard" and "core" can you use consistent naming.

emaxerrno · 2023-04-19T16:03:10Z

@mmaslankaprv - this needs some rpk progress bar of sorts. so you can watch it like

watch rpk repartition-progress or smth command line for it.

mmaslankaprv · 2023-04-19T17:42:05Z

@mmaslankaprv - what's the difference between "shard" and "core" can you use consistent naming.

you are right, i changed it to be consistent.

dotnwat · 2023-04-21T05:19:39Z

/ci-repeat 1

dotnwat

does this need a dedicated ducktape test?

dotnwat · 2023-04-22T00:29:46Z

src/v/redpanda/admin_server.cc

@@ -2605,6 +2605,32 @@ admin_server::mark_transaction_expired_handler(
      });
 }

+ss::future<ss::json::json_return_type>


doesn't look like this needs to be a coroutine

i use future<> here as in next commit i actually leverage this function being a coroutine.

dotnwat · 2023-04-22T00:33:51Z

src/v/cluster/controller_api.cc

+        co_await ss::maybe_yield();
+    }
+
+    using ret_t = result<std::vector<ntp_reconciliation_state>>;
+
+    auto node_results = co_await ssx::parallel_transform(


was the maybe_yield used because the set could be large? if that's true, then should concurrently be limited for the parallel transform?

the set of ntps can be indeed large, here we are limited to the number of nodes as the ntps are grouped, i think there is no need to limit concurrency here.

Signed-off-by: Michal Maslanka <michal@redpanda.com>

In order to provide a generic error code to express errors originating from outside of the cluster module (errors with different category) or an exceptions occurred in `controller_backend` we introduce a separate error code. Signed-off-by: Michal Maslanka <michal@redpanda.com>

src/v/cluster/controller_backend.cc

ztlpn · 2023-04-24T18:04:11Z

src/v/cluster/controller_backend.cc

+                if (ec.category() == error_category()) {
+                    it->last_error = static_cast<errc>(ec.value());
+                } else {
+                    it->last_error = errc::partition_operation_failed;


why can't we just save the error code as is?

we need it send it over the RPC we do not have way to serialize error category

src/v/cluster/controller_api.cc

src/v/redpanda/admin/api-doc/partition.json

src/v/redpanda/admin_server.cc

ztlpn · 2023-04-24T19:34:27Z

src/v/redpanda/admin_server.cc

+        size_t left_to_move = 0;
+        size_t already_moved = 0;


not blocking this pr, but it would be really great to have these available as metrics and drilled down per node to be able to see a dynamic.

good idea, we will do it as a follow up

bharathv

this is pretty cool.. I only have a minor comment.

src/v/cluster/controller_api.cc

Signed-off-by: Michal Maslanka <michal@redpanda.com>

Added revision, last error and retry count to backend operation. The information will be used to track partition reconfiguration progress. Signed-off-by: Michal Maslanka <michal@redpanda.com>

Added `controller_api` that allows caller to request partition reconciliation state from all the replicas where partition is currently hosted. The API returns a data structure containing operations that are executed by `controller_backend` on all of the replicas. Signed-off-by: Michal Maslanka <michal@redpanda.com>

mmaslankaprv · 2023-04-25T11:52:51Z

ci failure: #10163

The `/reconfiguartions` endpoint didn't provide an insight into the progress of partition reconfigurations. Added information that will allow user to check the operation progress and additionally check status of reconciliation on all replicas. Signed-off-by: Michal Maslanka <michal@redpanda.com>

Signed-off-by: Michal Maslanka <michal@redpanda.com>

vshtokman · 2023-04-28T13:40:59Z

/backport v23.1.x

vbotbuildovich · 2023-04-28T13:41:55Z

Failed to run cherry-pick command. I executed the below command:

git cherry-pick -x cd455277bf16c14d2f56cd1efec5b68d37b0f90a 711949d999520d41f2e45f5d1912c45b255e7f1b 8f83991236f209513a8e1e292d7bc0bc9037d9ca 8e01dd02678263b6fc4e46b4619af88292fe477d 179969311f7f249e90722f0c26c9651bc3a556d1 b2467b331364f5f6da21b049cf1155b246c49f8d d83a975e5ec44294c78451e4a3c88ef43a45c59c

Workflow run logs.

vshtokman · 2023-04-28T13:43:20Z

/backport v23.1.x

vbotbuildovich · 2023-04-28T13:44:16Z

Failed to run cherry-pick command. I executed the below command:

git cherry-pick -x cd455277bf16c14d2f56cd1efec5b68d37b0f90a 711949d999520d41f2e45f5d1912c45b255e7f1b 8f83991236f209513a8e1e292d7bc0bc9037d9ca 8e01dd02678263b6fc4e46b4619af88292fe477d 179969311f7f249e90722f0c26c9651bc3a556d1 b2467b331364f5f6da21b049cf1155b246c49f8d d83a975e5ec44294c78451e4a3c88ef43a45c59c

Workflow run logs.

vshtokman · 2023-04-28T13:52:02Z

/backport v23.1.x

vbotbuildovich · 2023-04-28T13:53:02Z

Failed to run cherry-pick command. I executed the below command:

git cherry-pick -x cd455277bf16c14d2f56cd1efec5b68d37b0f90a 711949d999520d41f2e45f5d1912c45b255e7f1b 8f83991236f209513a8e1e292d7bc0bc9037d9ca 8e01dd02678263b6fc4e46b4619af88292fe477d 179969311f7f249e90722f0c26c9651bc3a556d1 b2467b331364f5f6da21b049cf1155b246c49f8d d83a975e5ec44294c78451e4a3c88ef43a45c59c

Workflow run logs.

vshtokman · 2023-04-28T19:16:23Z

/backport v22.2.x

vbotbuildovich · 2023-04-28T19:17:17Z

Failed to run cherry-pick command. I executed the below command:

git cherry-pick -x cd455277bf16c14d2f56cd1efec5b68d37b0f90a 711949d999520d41f2e45f5d1912c45b255e7f1b 8f83991236f209513a8e1e292d7bc0bc9037d9ca 8e01dd02678263b6fc4e46b4619af88292fe477d 179969311f7f249e90722f0c26c9651bc3a556d1 b2467b331364f5f6da21b049cf1155b246c49f8d d83a975e5ec44294c78451e4a3c88ef43a45c59c

Workflow run logs.

vshtokman · 2023-04-28T19:18:25Z

Please ignore the /backport v22.2.x command. I was testing out the backport bot.

[backport] [v23.1.x] More detailed partition reconfiguration tracking #10201

github-actions bot added the area/redpanda label Apr 19, 2023

mmaslankaprv changed the title ~~Partition moving progress api~~ More detailed partition reconfiguration tracking Apr 19, 2023

mmaslankaprv force-pushed the partition-moving-progress-api branch from 0cf1b95 to fc87bd2 Compare April 19, 2023 15:50

mmaslankaprv force-pushed the partition-moving-progress-api branch from fc87bd2 to e282113 Compare April 19, 2023 17:41

mmaslankaprv force-pushed the partition-moving-progress-api branch from e282113 to d82ad13 Compare April 19, 2023 17:43

mmaslankaprv requested review from dotnwat, bharathv and ztlpn April 19, 2023 19:32

dotnwat reviewed Apr 22, 2023

View reviewed changes

mmaslankaprv added 2 commits April 24, 2023 14:38

admin_server: extracted reconfigurations handler to separate method

cd45527

Signed-off-by: Michal Maslanka <michal@redpanda.com>

mmaslankaprv force-pushed the partition-moving-progress-api branch from d82ad13 to 95d38c0 Compare April 24, 2023 12:38

mmaslankaprv requested a review from dotnwat April 24, 2023 15:17

ztlpn reviewed Apr 24, 2023

View reviewed changes

bharathv reviewed Apr 24, 2023

View reviewed changes

src/v/cluster/controller_api.cc Outdated Show resolved Hide resolved

mmaslankaprv added 3 commits April 25, 2023 10:58

c/controller_backend: store last error in controller backend delta queue

8f83991

Signed-off-by: Michal Maslanka <michal@redpanda.com>

c/controller_api: add more information to backend_operation

8e01dd0

Added revision, last error and retry count to backend operation. The information will be used to track partition reconfiguration progress. Signed-off-by: Michal Maslanka <michal@redpanda.com>

mmaslankaprv force-pushed the partition-moving-progress-api branch from 0630381 to 69bbf8e Compare April 25, 2023 08:58

mmaslankaprv requested review from bharathv and ztlpn April 25, 2023 08:58

ztlpn previously approved these changes Apr 25, 2023

View reviewed changes

bharathv previously approved these changes Apr 26, 2023

View reviewed changes

mmaslankaprv added 2 commits April 26, 2023 09:03

tests: added basic test for new reconfigurations api

d83a975

Signed-off-by: Michal Maslanka <michal@redpanda.com>

mmaslankaprv dismissed stale reviews from bharathv and ztlpn via d83a975 April 26, 2023 07:03

mmaslankaprv force-pushed the partition-moving-progress-api branch from 69bbf8e to d83a975 Compare April 26, 2023 07:03

mmaslankaprv requested review from ztlpn and bharathv April 26, 2023 10:04

bharathv approved these changes Apr 26, 2023

View reviewed changes

mmaslankaprv merged commit 60079f3 into redpanda-data:dev Apr 27, 2023

mmaslankaprv deleted the partition-moving-progress-api branch April 27, 2023 05:35

vbotbuildovich mentioned this pull request Apr 28, 2023

[v23.1.x] More detailed partition reconfiguration tracking #10434

Closed

vbotbuildovich mentioned this pull request Apr 28, 2023

[v22.2.x] More detailed partition reconfiguration tracking #10464

Closed

bharathv mentioned this pull request May 9, 2023

[backport] [v23.1.x] More detailed partition reconfiguration tracking #10201 #10630

Merged

7 tasks

bharathv added a commit that referenced this pull request May 10, 2023

Merge pull request #10630 from bharathv/backport-10201

c9364d8

[backport] [v23.1.x] More detailed partition reconfiguration tracking #10201

daisukebe mentioned this pull request Aug 18, 2023

rpk: Enhance partition management commands #9205

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

More detailed partition reconfiguration tracking #10201

More detailed partition reconfiguration tracking #10201

mmaslankaprv commented Apr 19, 2023 •

edited

Loading

emaxerrno commented Apr 19, 2023

emaxerrno commented Apr 19, 2023

mmaslankaprv commented Apr 19, 2023

dotnwat commented Apr 21, 2023

dotnwat left a comment

dotnwat Apr 22, 2023

mmaslankaprv Apr 24, 2023

dotnwat Apr 22, 2023

mmaslankaprv Apr 24, 2023

ztlpn Apr 24, 2023

mmaslankaprv Apr 25, 2023

ztlpn Apr 24, 2023

mmaslankaprv Apr 25, 2023

bharathv left a comment

mmaslankaprv commented Apr 25, 2023

vshtokman commented Apr 28, 2023

vbotbuildovich commented Apr 28, 2023

vshtokman commented Apr 28, 2023

vbotbuildovich commented Apr 28, 2023

vshtokman commented Apr 28, 2023

vbotbuildovich commented Apr 28, 2023

vshtokman commented Apr 28, 2023

vbotbuildovich commented Apr 28, 2023

vshtokman commented Apr 28, 2023

More detailed partition reconfiguration tracking #10201

More detailed partition reconfiguration tracking #10201

Conversation

mmaslankaprv commented Apr 19, 2023 • edited Loading

Backports Required

Release Notes

Improvements

emaxerrno commented Apr 19, 2023

emaxerrno commented Apr 19, 2023

mmaslankaprv commented Apr 19, 2023

dotnwat commented Apr 21, 2023

dotnwat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bharathv left a comment

Choose a reason for hiding this comment

mmaslankaprv commented Apr 25, 2023

vshtokman commented Apr 28, 2023

vbotbuildovich commented Apr 28, 2023

vshtokman commented Apr 28, 2023

vbotbuildovich commented Apr 28, 2023

vshtokman commented Apr 28, 2023

vbotbuildovich commented Apr 28, 2023

vshtokman commented Apr 28, 2023

vbotbuildovich commented Apr 28, 2023

vshtokman commented Apr 28, 2023

mmaslankaprv commented Apr 19, 2023 •

edited

Loading