Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tests: update offline_log_viewer and use in ducktape tests #5315

Merged
merged 22 commits into from
Dec 16, 2022
Merged

tests: update offline_log_viewer and use in ducktape tests #5315

merged 22 commits into from
Dec 16, 2022

Commits on Dec 15, 2022

  1. tools: write offline_log_viewer kvstore/controller output to stdout 

    On stderr it gets mixed with logs.  On stdout it is
straightforward to capture and decode json for use in tests.
    @jcsp
    jcsp committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    e168164 View commit details
    Browse the repository at this point in the history

  2. tools: make offline_log_viewer aware of clean_segment kvstore key

    @jcsp
    jcsp committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    9f00c02 View commit details
    Browse the repository at this point in the history

  3. tools: offline_log_viewer don't replay pre-snapshot kvstore messages 

    This was wasteful: all messages from before the snapshot should
already be accounted for in the state we loaded from the snapshot.
    @jcsp
    jcsp committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    963fc3d View commit details
    Browse the repository at this point in the history

  4. tools/metadata_viewer: handle kvstore deletions 

    A None value is a deletion.  Previously, we ignored
these and thereby the kvstore dump would include
all keys ever created, even if they had since
been deleted.
    @jcsp
    jcsp committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    7f666f7 View commit details
    Browse the repository at this point in the history

  5. tools: handle short header read in offline_log_viewer

    @jcsp
    jcsp committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    b492385 View commit details
    Browse the repository at this point in the history

  6. tools/offline_log_viewer: added decode_adl_or_serde to dispatch decod… 

    …ing functions
    @andijcr @jcsp
    andijcr authored and jcsp committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    37b76a1 View commit details
    Browse the repository at this point in the history

  7. tools/offline_log_viewer: fixed variable used to read BatchType (was … 

    …batch.header.type instead of batch.type)
    @andijcr @jcsp
    andijcr authored and jcsp committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    18328b1 View commit details
    Browse the repository at this point in the history

  8. tools/offline_log_viewer: added peek_int8, read_serde_enum 

    ... and augmented Reader.read_envelope to accept a type_read Callable[rdr: Reader, version: Int]: Dict and a max_version parameter.

 If a type_read is used, it is invoked with reader and envelope if envelope.version <= max_version parameter.
 the result is joined with a dict {'envelope': envelope} and returned, otherwise an error dict is returned.

 If no type_read is used, envelope is returned immediately
    @andijcr @jcsp
    andijcr authored and jcsp committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    4bbd0c5 View commit details
    Browse the repository at this point in the history

  9. tools/offline_log_viewer: added serde decoding for topic_commands 

    the new code tries to mimic the structure of the adl decoding functions.
the new code is aware of the version number embedded in the binary stream and will skip the field if the version is not supported, while reporting the error.
    @andijcr @jcsp
    andijcr authored and jcsp committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    021317a View commit details
    Browse the repository at this point in the history

  10. tools/offline_log_viewer: added decode_user_command for serde

    @andijcr @jcsp
    andijcr authored and jcsp committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    ce43d16 View commit details
    Browse the repository at this point in the history

  11. tools/offline_log_viewer: added decode_acl_command for serde

    @andijcr @jcsp
    andijcr authored and jcsp committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    8b924de View commit details
    Browse the repository at this point in the history

  12. tools/offline_log_viewer: added decode_config_command for serde for c… 

    …luster_config_command
    @andijcr @jcsp
    andijcr authored and jcsp committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    29f9e7e View commit details
    Browse the repository at this point in the history

  13. tools/offline_log_viewer: decode_feature_command, decode_node_managem… 

    …ent_command
    @andijcr @jcsp
    andijcr authored and jcsp committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    98e4df6 View commit details
    Browse the repository at this point in the history

  14. tools/offline_log_viewer: added method to get number of unprocessed b… 

    …ytes
    @andijcr @jcsp
    andijcr authored and jcsp committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    8fe0654 View commit details
    Browse the repository at this point in the history

  15. tools/offline_log_viewer: added check that all the bytes of a record … 

    …are processed

in case of unread byte, an 'unread': {'k': ..., 'v': ...} field is added and an error is logged

this check is not exact but is a good proxy to know if the decoding code is correct
    @andijcr @jcsp
    andijcr authored and jcsp committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    ba26cd7 View commit details
    Browse the repository at this point in the history

  16. tools/offline_log_viewer: support v=4 group_configuration

    @jcsp
    jcsp committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    d8d85c4 View commit details
    Browse the repository at this point in the history

  17. tools: fix kvstore reads in offline_log_viewer

    @jcsp
    jcsp committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    a9ab9fa View commit details
    Browse the repository at this point in the history

  18. tests: include offline_log_viewer in Dockerfile

    @jcsp
    jcsp committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    8557ac4 View commit details
    Browse the repository at this point in the history

  19. tests: add clients.offline_log_viewer

    @jcsp
    jcsp committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    a7e6983 View commit details
    Browse the repository at this point in the history

  20. tests: validate kvstore cleanup in topic_delete_test 

    Partitions have various kvstore items, stored by
the raft + storage layers.  When the topic is
deleted, they should be deleted too.

Use our offline log decoder tool to inspect
the kvstore out-of-band and validate the keys
go away when the topic is deleted.
    @jcsp
    jcsp committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    fac7d51 View commit details
    Browse the repository at this point in the history

  21. tests: check controller log decode after multi-version upgrade

    @jcsp
    jcsp committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    1813135 View commit details
    Browse the repository at this point in the history

  22. tools: disable broken controller command decode 

    This needs fixing, but better to comment it out and have
the rest of the command work.
    @jcsp
    jcsp committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    189e475 View commit details
    Browse the repository at this point in the history