Support other operating systems

[aneudecker]

The R package should be functional on all common operating systems (main linux distributions, macOS, Windows).

Hints:

• Create Makevars file dynamically to allow for different compiler arguments
• Check for other Rcpp package for examples: https://github.com/RcppCore/RcppArmadillo/, https://github.com/eddelbuettel/rcppgsl/
• Use GNU Autoconf to create configure file https://www.gnu.org/software/autoconf/

[aneudecker]

We can download librdkafka with nuget (https://www.nuget.org/downloads) nuget.exe install librdkafka.redist

• in python this is done via: https://github.com/confluentinc/confluent-kafka-python/blob/a27d11356caf8d2cac399a0a031259361fcca43d/tools/windows-install-librdkafka.bat and https://github.com/confluentinc/confluent-kafka-python/blob/a27d11356caf8d2cac399a0a031259361fcca43d/tools/windows-copy-librdkafka.bat

[wahani]

We currently have

══ Failed tests ════════════════════════════════════════════════════════════════
── Error ('test-commit.R:21:3'): Committing throws no error ────────────────────
<Rcpp::exception/C++Error/error/condition>
Error: Error: Failed commit: Local: No offset stored
Backtrace:
    ▆
 1. └─consumer$commit(async = FALSE) at test-commit.R:21:3

[ FAIL 1 | WARN 0 | SKIP 0 | PASS 8 ]
Error: Error: Test failures
Execution halted

in https://github.com/INWTlab/r-kafka-client/actions/runs/7033691872/job/19140093058?pr=9

On MacOS.

[aneudecker]

It was possible to compile the cpp into .o files using the following Makevars.win and putting the header files (rdkafkacpp.h) into inst/include/librdkafka:

PKG_CXXFLAGS = -I../inst/include

However the last step threw a lot of errors like

   [...]/consumer.cpp:16: undefined reference to `__imp__ZN7RdKafka13KafkaConsumer6createEPKNS_4ConfERNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE'

This makes sense as the library was not included yet. But all attempts to link to the library were not successful, i.e. putting the *.lib files into the lib folder and running

PKG_LIBS = -L../lib -llibrdkafkacpp

[aneudecker]

The problem is that the nuget package is compiled with visual studio which is not compatible with the mingw compiler included in Rtools. I got it partially working by downloading the binaries from here: https://packages.msys2.org/package/mingw-w64-ucrt-x86_64-librdkafka?repo=ucrt64 and adjusting the Makevars.win file to

PKG_CXXFLAGS = -I../include
PKG_LIBS = -L../lib -lrdkafka++

Now the building succeeds but the library cannot be loaded because there are other runtime dependencies (i.e. libcurl, libzstd, libcrypt, liblz4, libssl) which cannot be found. This could be debugged by using Dependency Walker in Windows.

I installed the msys2 helper from https://www.msys2.org/ and installed the library with pacman -S mingw-w64-ucrt-x86_64-librdkafka. This also downloaded all dependencies into C:\msys64\ucrt64\bin

I added this folder to the path (Settings -> Edit Environment Variables for your Account) and loading worked 🎉

msys2 is also included in Rtools. I need to test if we can install librdkafka this way and do not need to adjust the path

[aneudecker]

We can use the msys2 binary from Rtools to install librdkafka using the command

pacman -S mingw-w64-ucrt-x86_64-librdkafka

Then we can compile and use the package with the following Makevars.win

PKG_CXXFLAGS = -I/C/RBuildTools/4.3/ucrt64/include
PKG_LIBS = -L/C/RBuildTools/4.3/ucrt64/lib -lrdkafka++

[aneudecker]

It's now possible to run the package on windows, potentially with some manual steps involved. As we need to change the process anyway when publishing to CRAN this should be okay for now.

[aneudecker]

We can now support all main operating systems (windows, macos, linux). Pipeline tests however are currently running under linux