ktail is a small project intended to dump content of a Kafka topic from a given time.
ktail is provided as rpm packages (Sorry, only this packaging is currently provided. Contribution welcome), on the release pages.
Once installed, basic usage is the following:
# ktail --brokers "brk1.my.domain.com:9092,brk2.my.domain.com:9092,brk3.my.domain.com:9092" --list
or
# ktail --brokers "brk1.my.domain.com:9092,brk2.my.domain.com:9092,brk3.my.domain.com:9092" --topic t1
Here all all the parameters
Option (* = required) Description
--------------------- -----------
* --brokers <br1:9092,br2:9092> Comma separated values of Source Kafka brokers
--forceProperties Force unsafe properties
--from <Starting point in time> Start from (In Iso date format or in XX[d|h|m|s] notation)
--help Display this usage message
--list list available topics
--max <Long: count> Max record count (default: 9223372036854775807)
--pattern <topic1> Display pattern. ex: %p-%t-%o-%k-%v for <partition#>-<timestamp>-<offset>-<key>-<value> (default: %v)
--property <prop=val> Producer property (May be specified several times)
--to <Ending point in time> Up to (In Iso date format or in XX[d|h|m|s] notation)
--topic <topic1> Source topic
ktail --brokers "brk1.my.domain.com:9092" --topic t1 --from 2m
List all messages from topic t1
produced in the last 2 minutes and wait for new messages.
ktail --brokers "brk1.my.domain.com:9092" --topic t1 --from 2m --to 0s
List all messages from topic t1
produced in the last 2 minutes and exit
ktail --brokers "brk1.my.domain.com:9092" --topic t1 --from 2017-01-01T12:00:00 --to 2017-01-01T13:00:00 --pattern "%k [%t]"
List all messages key and timestamp from topic t1
produced between 12H and 13H on January 1rst 2017
If kerberos is activated, you will need to define a jaas configuration file as java option.
This can easily be achieved by uncomment this following line in /etc/ktail/setenv.sh
JOPTS="$JOPTS -Djava.security.auth.login.config=/etc/ktail/kafka_client_jaas.conf"
You can of course modify the kafka_client_jaas.conf
file to adjust to your needs or target another existing one.
But, keep in mind, you must also perform a kinit
command, with a principal granting read access to target topic
You will find an Ansible role at this location.
This role can be used as following;
- hosts: kafka_brokers
- hosts: cmd_node
vars:
kdescribe_rpm_url: https://github.com/Kappaware/ktail/releases/download/v0.2.0/ktail-0.2.0-1.noarch.rpm
kerberos: true
kafka_port: 9092
roles:
- kappatools/ktail
Note:
- hosts: kafka_brokers
at the beginning, which force ansible to grab info about the hosts in the[kafka_brokers]
group, to be able to fulfill this info into ktail configuration. Of course, such a group must be defined in the inventory. In this case, brokers information will no longer be required on the command line.
Just clone this repository and then:
$ gradlew rpm
This should build everything. You should be able to find generated packages in build/distribution folder.
Copyright (C) 2017 BROADSoftware
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.