Skip to content

Commit

Permalink
rpk group: fix describe
Browse files Browse the repository at this point in the history 
Commit 15df1b3 broke describe.
Previously, if a group had no active members, describe would list
offsets for *all* topics and partitions in the cluster. This was
wasteful and would be slow on huge clusters. CalculateGroupLag would
then filter for only topics that have commits, and then print only what
was necessary.

Describe was changed to only list offsets on groups that were consuming.
Unfortunately, this broke empty groups.

We now describe a union of all topics that have commits, and all topics
that a group is consuming. This is the minimal amount of topics needed
to satisfy the group lag calculation and does not waste extra resources.

We bump kadm to take advantage of a new API to keep this code nice.
  • Loading branch information
@twmb
twmb committed Aug 5, 2022
1 parent d3bb917 commit ab6a991
Show file tree
Hide file tree
Showing 3 changed files with 12 additions and 13 deletions.
6 changes: 3 additions & 3 deletions src/go/rpk/go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@ require (
github.com/stretchr/testify v1.7.0
github.com/tklauser/go-sysconf v0.3.10
github.com/twmb/franz-go v1.6.0
github.com/twmb/franz-go/pkg/kadm v1.1.1
github.com/twmb/franz-go/pkg/kmsg v1.1.0
github.com/twmb/franz-go/pkg/kadm v1.2.0
github.com/twmb/franz-go/pkg/kmsg v1.2.0
github.com/twmb/tlscfg v1.2.0
github.com/twmb/types v1.1.6
golang.org/x/sync v0.0.0-20220601150217-0de741cfad7f
Expand All @@ -53,7 +53,7 @@ require (
github.com/inconshreveable/mousetrap v1.0.0 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
github.com/klauspost/compress v1.15.8 // indirect
github.com/klauspost/compress v1.15.9 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/mattn/go-colorable v0.1.12 // indirect
github.com/mattn/go-isatty v0.0.14 // indirect
Expand Down
15 changes: 6 additions & 9 deletions src/go/rpk/go.sum
View file
Original file line number Diff line number Diff line change
Expand Up @@ -270,10 +270,9 @@ github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 h1:Z9n2FFNU
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51/go.mod h1:CzGEWj7cYgsdH8dAjBGEr58BoE7ScuLd+fwFZ44+/x8=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/compress v1.15.2/go.mod h1:PhcZ0MbTNciWF3rruxRgKxI5NkcHHrHUDtV4Yw2GlzU=
github.com/klauspost/compress v1.15.4/go.mod h1:PhcZ0MbTNciWF3rruxRgKxI5NkcHHrHUDtV4Yw2GlzU=
github.com/klauspost/compress v1.15.8 h1:JahtItbkWjf2jzm/T+qgMxkP9EMHsqEUA6vCMGmXvhA=
github.com/klauspost/compress v1.15.8/go.mod h1:PhcZ0MbTNciWF3rruxRgKxI5NkcHHrHUDtV4Yw2GlzU=
github.com/klauspost/compress v1.15.9 h1:wKRjX6JRtDdrE9qwa4b/Cip7ACOshUI4smpCQanqjSY=
github.com/klauspost/compress v1.15.9/go.mod h1:PhcZ0MbTNciWF3rruxRgKxI5NkcHHrHUDtV4Yw2GlzU=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/kr/fs v0.1.0/go.mod h1:FFnZGqtBN9Gxj7eW1uZ42v5BccTP0vu6NEaFoC2HwRg=
Expand Down Expand Up @@ -390,14 +389,13 @@ github.com/tklauser/go-sysconf v0.3.10/go.mod h1:C8XykCvCb+Gn0oNCWPIlcb0RuglQTYa
github.com/tklauser/numcpus v0.4.0/go.mod h1:1+UI3pD8NW14VMwdgJNJ1ESk2UnwhAnz5hMwiKKqXCQ=
github.com/tklauser/numcpus v0.5.0 h1:ooe7gN0fg6myJ0EKoTAf5hebTZrH52px3New/D9iJ+A=
github.com/tklauser/numcpus v0.5.0/go.mod h1:OGzpTxpcIMNGYQdit2BYL1pvk/dSOaJWjKoflh+RQjo=
github.com/twmb/franz-go v1.5.3/go.mod h1:eqHYpAuvlTArOdZ1XtPYyOQ1uUb40CSZwbpL3ccjibI=
github.com/twmb/franz-go v1.6.0 h1:yri7YsVBe/k1LKcoZSLILgUI3U14e82qtD9i4VOcs9c=
github.com/twmb/franz-go v1.6.0/go.mod h1:xdMwpUIQL/JDKKwerc5qJQG8TU1SNIddfjKJJyqRJIg=
github.com/twmb/franz-go/pkg/kadm v1.1.1 h1:tqcJt9ChdqiY+Vi3F13z8/XRAJozbJNj0/7VPnrdTQA=
github.com/twmb/franz-go/pkg/kadm v1.1.1/go.mod h1:Ly8COloKx7pbwBdlP4qTYKdEVcNvk7D3+in3ujv3x/M=
github.com/twmb/franz-go/pkg/kmsg v1.0.0/go.mod h1:SxG/xJKhgPu25SamAq0rrucfp7lbzCpEXOC+vH/ELrY=
github.com/twmb/franz-go/pkg/kmsg v1.1.0 h1:csckTxG48q7Tem7ZwMxe2jAb0ehDNglxZccGnpqe4RU=
github.com/twmb/franz-go/pkg/kadm v1.2.0 h1:3jtgdCi04TPGSCiaIBe2nxBNkHoCZRm/YljSFEh7BHA=
github.com/twmb/franz-go/pkg/kadm v1.2.0/go.mod h1:izleX4EttZwes7MRiHFmPWPSqDuGr4VN6p7jCIYoU0g=
github.com/twmb/franz-go/pkg/kmsg v1.1.0/go.mod h1:SxG/xJKhgPu25SamAq0rrucfp7lbzCpEXOC+vH/ELrY=
github.com/twmb/franz-go/pkg/kmsg v1.2.0 h1:jYWh2qFw5lDbNv5Gvu/sMKagzICxuA5L6m1W2Oe7XUo=
github.com/twmb/franz-go/pkg/kmsg v1.2.0/go.mod h1:SxG/xJKhgPu25SamAq0rrucfp7lbzCpEXOC+vH/ELrY=
github.com/twmb/tlscfg v1.2.0 h1:WCzLHtmnVJ94+veAO4TLTB1ENx7TPYLkTl4Q6WFF4Vo=
github.com/twmb/tlscfg v1.2.0/go.mod h1:GameEQddljI+8Es373JfQEBvtI4dCTLKWGJbqT2kErs=
github.com/twmb/types v1.1.6 h1:PnQYJ8fMHkjPR4mgJkzqX1lYhfamNl5I2zuVBSZwLuE=
Expand Down Expand Up @@ -425,7 +423,6 @@ golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPh
golang.org/x/crypto v0.0.0-20210421170649-83a5a9bb288b/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20211108221036-ceb1ce70b4fa/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20220427172511-eb4f295cb31f/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/crypto v0.0.0-20220518034528-6f7dac969898/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d h1:sK3txAijHtOK88l68nt020reeT1ZdKLIYetKl95FzVY=
golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
Expand Down
4 changes: 3 additions & 1 deletion src/go/rpk/pkg/cli/cmd/group/describe.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,9 @@ information about the members.
}

var listed kadm.ListedOffsets
if topics := described.AssignedPartitions().Topics(); len(topics) > 0 {
listPartitions := described.AssignedPartitions()
listPartitions.Merge(fetched.CommittedPartitions())
if topics := listPartitions.Topics(); len(topics) > 0 {
listed, err = adm.ListEndOffsets(ctx, topics...)
out.HandleShardError("ListOffsets", err)
}
Expand Down

0 comments on commit ab6a991

Please sign in to comment.