Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tools/metadata_viewer: Refactor metadata_viewer and add transaction metadata support #5294

Merged
merged 6 commits into from
Jul 1, 2022

Commits on Jun 30, 2022

  1. tools[log_viewer]: cleanup metadata_viewer

    * renames to offline_log_viewer since it is not limited to metadata
    anymore and we can parse actual record data.
    * tools/storage.py is obselete.
    bharathv committed Jun 30, 2022
    Configuration menu
    Copy the full SHA
    60e28ae View commit details
    Browse the repository at this point in the history
  2. tools[log_viewer]: Expand batch attrs

    Includes compression type and whether the batch is
    transactional/control type.
    bharathv committed Jun 30, 2022
    Configuration menu
    Copy the full SHA
    e9e2ece View commit details
    Browse the repository at this point in the history
  3. tools[log_viewer]: support kafka_internal ns

    Some topics are in kafka_internal ns like
    tx (transaction coordinator) and id_allocator.
    bharathv committed Jun 30, 2022
    Configuration menu
    Copy the full SHA
    9b51495 View commit details
    Browse the repository at this point in the history
  4. tools[log_viewer]: Map record header types to enum

    Example:
    
    INFO:viewer:{
      "header_crc": 1126531443,
      "batch_size": 90,
      "base_offset": 3,
      "type": 9,
      "crc": 1766141844,
      "attrs": 32,
      "delta": 0,
      "first_ts": 1656554802864,
      "max_ts": 1656554802864,
      "producer_id": 1002,
      "producer_epoch": 1,
      "base_seq": -1,
      "record_count": 1,
      "type_name": "tx_prepare", <====
      "expanded_attrs": {
        "compression": "none",
        "transactional": false,
        "control_batch": true,
        "timestamp_type": false
      }
    }
    bharathv committed Jun 30, 2022
    Configuration menu
    Copy the full SHA
    8d96c0c View commit details
    Browse the repository at this point in the history
  5. tools[log_viewer]: Populate txn control markers

    Moves the logic in KafkaLog like other methods,
    cleans up duplicate code.
    
    Introduces KafkaControlRecordType
    
    For transactional && control records we extract
    the exact control record type(commit/abort).
    bharathv committed Jun 30, 2022
    Configuration menu
    Copy the full SHA
    45f4288 View commit details
    Browse the repository at this point in the history

Commits on Jul 1, 2022

  1. tools/log_viewer: Switch to BatchType enum

    Switch to the enum in various other places for
    readability.
    bharathv committed Jul 1, 2022
    Configuration menu
    Copy the full SHA
    64f1b39 View commit details
    Browse the repository at this point in the history