Architecture-specific folders like runtimes/<rid>/native/ outside of NuGet packages [nativeinterop]

[KiruyaMomochi]

Problem

I am trying to build a project with native libraries P/Invoke.
Since the project is for multiple architectures, I hope dotnet can find the right library automatically after I include them.

However, the only way I can find is to create a NuGet package, putting these files in architecture-specific folders
(RID-specfic folders), then add the package reference to the project. If I directly put runtimes/<rid>/native/ in the project or add project reference to another project containing this folder, dotnet cannot find the library.

When architecture-specific folders take effect, path to these native libraries can be found at <AssemblyName>.deps.json.

Side story: Xamarin and Blazor WebAssembly

In Xamarin, both iOS/Catalyst and Android supports architecture-specific folders in NuGet packages now. Out of NuGet package, their behavior is the same as dotnet. However, they do provide a MSBuild item to include a native library: PackageReference for iOS/Catalyst (not documented yet), AndroidNativeLibrary for Android.

In WebAssembly, these dependencies aren't referenced automatically and must be referenced manually as NativeFileReferences. For NuGet package, author can choose to include a .props file in the package.

Suggested Solution

Goal

Taken from #19929, the following goals should be achieved:

• User can define architecture-specifics asset in a library or application project without packaging (not NuGet package)
• Project reference should be able to consume library with architecture-specific assets correctly
• Such specific assets from all sources (app project itself, project references and package references) are
  • correctly copied to the output (for portable apps into runtime subdirectory, for RID specific apps into the root) and
  • the correct .deps.json is produced.

This behavior may not be default for a new project, but developers can make use of it by setting some properties.

MSBuild file

Reference native library

To support architecture-specific native libraries, we should at least provide a MSBuild item, which can be used to reference a library. For example, NativeFileReferences.

<ItemGroup>
  <NativeFileReferences Include="runtimes/win-x64/native/MyGrimoire.dll" RuntimeIdentifier="win-x64" />
  <NativeFileReferences Include="runtimes/<rid>/native/<AssemblyName>.dll" RuntimeIdentifier="<rid>" />
</ItemGroup>

For a unified experience, this item should also be available to Xamarin.iOS, Xamarin.Android and Blazor WebAssembly, although they can also choose to use their own item when some special feature is required.

Implictly include all architecture-specific items

To support architecture-specific folders, we may provide a true/false property. For example, a property called EnableArchitectureSpecificItems.

<PropertyGroup>
  <EnableArchitectureSpecificItems>true</EnableArchitectureSpecificItems>
</PropertyGroup>

When the property is set, all files in architecture-specific folders will behave as if referencing a package containing these files.

CLI support

We should also make our CLI supports adding these native references.

dotnet add reference 'cool.dylib' --rid 'osx-x64'

When invoking this command, NativeFileReferences will be added to the project.

Documention

We should document it in Native Interoperability part of .NET Docs, explaining this convention and how to use them. We may also remind user possible divergence between other platforms like Xamarin or WebAssembly.

Additional Context

There are many releated issues (some are not active), and it's hard to understand how rid-specific folder works without knowing NuGet, MSBuild and difference between package/project references. Adding Xamarin and WebAssembly, it is much harder.

That's why I created this feature request, trying to collect all these discussions and try to make it clear. Below are some discussions I have found.

• how to include p/invoke needed native library in csproj and be able to publish the project? runtime#65420

  This is a known limitation right now (you could call it a design flaw). There's no good way to include native dependencies in a project reference, the only way which really works is package reference. - vitek-karas

• Architecture-specific folders #16381

  Mentioned in discussion above. John0King suggested to implement <NativeBinaryReference /> item

  <NativeBinaryReference Include="native.dll" Rid="win-x64" />

  which can be use to both genereate nuget package and also for project reference. Or to make the metadata PackagePath to be supported both on <PackageReference /> and <ProjectReference />.

• Resolution of native libraries when using DllImport #19929

  This has been around for a time, and been transformed between dotnet/msbuild, dotnet/runtime, and our dotnet/sdk.

  I think the native binary resolution is just a symptom of a larger problem, which is that when consuming a project that builds a NuGet package as a ProjectReference, none of the package behaviors are brought in. That includes not just RID-specific binaries but also .props and .targets. Perhaps the answer is that things that are expressed by convention (file name/path, etc) in NuGet packages need to be able to be made explicit in the project file so they can be consumed by the SDK. That could also drive the package definition so that less of it is based on magic folder names. - saucecontrol

• Add a way to list native assets that a project will load from the app directory (list them in deps.json) #11373

  Issue author safern has managed to find a workaround using RuntimeTargetsCopyLocalItems, but this

  • requires a NuGetPackageId which has to be an actual package that app references.
  • is not transitive. Another project that references the project with RuntimeTargetsCopyLocalItems will not have the native assets included in the output.

• How to consume native dependencies via ProjectReference? #10575

  This issue is transfered from dotnet/cli. It received twenty thumbs up but zero replies...

• Runtime native dll not copied beetween local projects #8645

  It suggested a <RuntimeAsset /> item like

  <RuntimeAsset Include="cool.dylib" RuntimeIdentifier="osx-x64" />

• Including native libraries in dotnet projects without nuget package #7017

  The earlist issue I have found, also transfered from dotnet/cli. To my surprise, this issue is closed. But the author said he will try if <Content> works then didn't update any progress.

• Where to put Native libraries inside a Project and how to reference them? runtime#11404

  Copy libraies by CopyToOutputDirectory is suggested but it won't work for linux.

• Actually, even how these folders work in a NuGet package is not fully documented. Users are asking. djee-ms opened a issue Guide for packaging C# library using P/Invoke to per-architecture and/or per-platform C++ native DLLs NuGet/Home#8623 to collect these problems, while csproj: document how to properly pack platform-specific native assemblies NuGet/Home#6645 and https://github.com/NuGet/docs.microsoft.com-nuget/issues/238 are tracking NuGet's documentation problem.

• As searching further, I also found talks about adding TFM support in Add a WinRT rid runtime#17263. These talks lead to NuGet should add support for /runtimes/{rid}/nativeassets/{txm}/ NuGet/Home#2782 and NuGet now also supports /runtimes/{rid}/nativeassets/{txm}/.

Interesting discovery

I don't know if it's expected behavior or not, but if there are some NuGet libraries already using these rid-specific folders (like SkiaSharp.NativeAssets.*), and you put your own library in these folder too, then your libraries can also be found.

[asmichi]

Some more links:

• Copy runtime libraries from references project #765
• Allow runtime-specific assets in the project #1088