cargo install a library as .dll / .so file

[fschutt]

Describe the problem you are trying to solve

I have a rather large library (azul-dll), which can be compiled to a cdylib. The library itself has 200+ dependencies (already tried to optimize it) and rebuilding code that uses that library takes ~30s, which is slow.

My plan to speed up compile time was to do something like this:

# 1. let the user run an installation command
# optionally, the user can also install the library via apt-get and link it dynamically (not the default)
# compiles and installs the library as a dll into the ./cargo directory
# standardized (?) path for outputting the compiled dll / .so file
cargo install azul-dll -- --release --target=cdylib --output-path="./cargo/installed-libs/azul.so"

Then, in the main azul library, include and load the pre-compiled DLL:

// 2. include the .so file as bytes, so that there are no deployment problems (still one binary file)
// if the dll is not installed, this line will simply fail to compile
#[cfg(feature = "include-dll-in-binary")]
const LIBRARY_BYTES: &[u8] = include_bytes!("./cargo/installed-libs/azul.so");

mod dll {
    pub struct AzulDll {
        lib: Box<Library>,
        az_css_empty: Symbol<extern fn() -> AzCssPtr>,
    }

    fn load_library(bytes: &'static [u8]) -> AzulDll {
        fs::write("/temp/azul.so", bytes); // workaround, since dlopen only accepts file paths
        let lib = Library::new("/temp/azul.so").unwrap();
        AzulDll {
            lib,
            az_css_empty = unsafe { lib.get::<extern fn() -> AzCssPtr>(b"az_css_empty").unwrap() }
        }
    }
}

lazy_static! {
    // initialize the library at startup
    let LIB: AzulDll = load_library(LIBRARY_BYTES);
}

fn main() {
    let css = LIB.az_css_empty();
    println!("{:?}", css);
}

This solution would especially speed up the link time, which is often greater than the compile time itself.

For a release build, the user could specify via feature-flags whether the DLL is loaded from the system and deployed alongside the binary or in a separate package (for example, to work with things like apt on Linux) or if the DLL should be included in the binary for deployment on systems that have little to no library dependency management (Windows), with the disadvantage of slight binary bloat.

Describe the solution you'd like

Currently you can only install binaries from crates.io, it would be great if you could also install DLL / .so files directly from crates.io if they have a crate-type=cdylib in their Cargo.toml.

I could maybe hack around this with giving azul-dll a main.rs file, which the bootstraps itself, and builds and installs azul-dll in the correct directory, but it would be better to have a standardized solution for installing DLLs, especially for security purposes.

Lastly, if something like that would be considered, I've noticed that there is a strip command for cargo now - could the command / rustc be modified to only strip debug information, not function names (which are important for dynamically loading a library)? Thanks in advance for any help.

[FFY00]

@ehuss do you need any help with this? I am interested in getting it to work, though I had a slightly different implementation in mind, something that is useful for Linux distributions and similar users. I would like a way to build and install projects as dynamic libraries, and a way to tell cargo to link against dynamic libraries on the system instead of building the crates.
Currently, distros are essentially forced to static link everything, which has a few issues like being hard to track and deploy fixes for security issues.

I would be happy to throw some time into this if needed 😊

[cgranade]

Just to chime in, I'd find this really useful for myself as well. There's a lot of times where I want to gradually introduce Rust into a non-Rust codebase by using a crate-type = ["cdylib"] crate as the interface between Rust and C/C++.

For breaking dependencies between CI builds, it'd be really helpful if I could have the C build run cargo install foo to get the sources for and build foo.{so,dylib,dll} without having to use cargo-download or git submodules.

[aviramha]

I find this useful as well.

[weihanglo]

Semi-related. Cargo gets an unstable feature Z bindeps, which allows your package to depend on bin, cdylib, and staticlib on compile time. Hope it helps to some extent.

https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#artifact-dependencies

[aviramha]

Semi-related. Cargo gets an unstable feature Z bindeps, which allows your package to depend on bin, cdylib, and staticlib on compile time. Hope it helps to some extent.

https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#artifact-dependencies

Oh wow! That's exactly what I need! Thank you. Is it possible to get the cdylib of the same package? for example package pkg has lib and bin. I want to embed lib -> bin as an artifact.

[weihanglo]

Is it possible to get the cdylib of the same package? for example package pkg has lib and bin. I want to embed lib -> bin as an artifact.

The interface to define an artifact dependency is specifying dependencies in [dependencies] in Cargo.toml, so I guess it may not work for your scenario. Though I believe it works if you separate bin and lib into two different packages.

[uuhan]

Any updates?
I does not find anyway to install the cdylib library.
The use case is, I have a lua shared library, but I can not copy out the built library.
The library built in macos with file extension .dylib, in linux it is .so, but I only need the correct name with .so extension and install it into some path.