Building connectors for Apache Kafka is hard. Chances are that you just read the previous sentence, and you subconsciously nooded with your head. The reason this happens is that Kafka Connect, which is the runtime platform behind the executing connectors, uses a not so trivial software architecture. There is also a lack of a proper documentation that teaches how the development framework works, how it connects with the runtime, and which best practices you must follow.
For situations like this, your best bet is to get hold of an existing code and try to do the same, hoping your own connector will be written the best way possible. By all means, this is an excellent strategy. But most of the time, it is hard to understand the code, as the connector you took as an example might have more code focused on solving the technical problem that the connector is aiming to solve than only the code part related to building custom connectors.
This is the reason this project exists. This is a minimalistic repository that contains a source connector, whose focus is to show how the development framework from Kafka Connect works. From this repository, you can easily derive your own connector and write only the code that matter for your technical use case. This project also provides you with the resources to build, deploy, test, and debug the code on-premises, as well as deploying it in the cloud.
The first thing you need to do to use this connector is to build it.
- Install the following dependencies:
- Build the Kafka Connect connector file.
mvn clean package💡 A file named target/my-first-kafka-connector-1.0.jar will be created. This is your connector for Kafka Connect.
With the connector properly built, you need to have a local environment to test it. This project includes a Docker Compose file that can spin up container instances for Apache Kafka and Kafka Connect.
- Install the following dependencies:
- Start the containers using Docker Compose.
docker compose up -dWait until the containers kafka and connect are started and healthy.
Nothing is actually happening since the connector hasn't been deployed. Once you deploy the connector, it will start generating sample data from an artificial source and write this data off into three Kafka topics.
- Deploy the connector.
curl -X POST -H "Content-Type:application/json" -d @examples/my-first-kafka-connector.json http://localhost:8083/connectors- Check if the connector is producing data to Kafka topics.
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic source-1 --from-beginningkafka-console-consumer.sh --bootstrap-server localhost:9092 --topic source-2 --from-beginningkafka-console-consumer.sh --bootstrap-server localhost:9092 --topic source-3 --from-beginning💡 All three topics should have sample data continuously generated for them.
This is actually an optional step, but if you wish to debug the connector code to learn its behavior by watching the code executing line by line, you can do so by using remote debugging. The Kafka Connect container created in the Docker Compose file was changed to rebind the port 8888 to enable support for JDWP. The instructions below assume that you are using Visual Studio Code for debugging. However, most IDEs for Java should provide support for JDWP. Please check their documentation manuals about how to attach their debugger to the remote process.
- Create a file named
.vscode/launch.jsonwith the following content:
{
"version": "0.2.0",
"configurations": [
{
"name": "Debug Connector",
"type": "java",
"request": "attach",
"hostName": "localhost",
"port": 8888
}
]
}- Set one or multiple breakpoints throughout the code.
- Launch a new debugging session to attach to the container.
- Play with the connector to trigger the live debugging.
Use the following command to undeploy the connector from Kafka Connect:
curl -X DELETE http://localhost:8083/connectors/my-first-kafka-connector