Native dependency resolution problem for #r nuget #10136
Comments
|
Repros on RC1 for me |
update torch getting started to workaround dotnet/fsharp#10136
|
@dsyme, this is really cool. I will see what I can do. |
|
This repros quite nicely on Windows: |
|
So ...transitive native dependencies are not resolvable using the native resolution event handler. I am fairly confident that only works for managed code that has native dependencies. It looks to me like the design was primarily designed to make pinvoke work. |
|
With the AssemblyLoadContext.ResolvingUnmanagedDll event handler, we are not being notified for native dll load attempts that are caused by a native dependency of a native library on either windows or linux. Is that "ByDesign" or is there a mechanism we can use that will allow us to detect attempts to transitively load native .dlls. In our example above we have a managed library We get notified to locate the torchsharp dependency but not the torch_cpu.dll one. Not that it will be much help but our handler is here: https://github.com/dotnet/fsharp/blob/main/src/fsharp/Microsoft.DotNet.DependencyManager/NativeDllResolveHandler.fs#L89 |
|
It is by design. The native library loader is part of OS. It does not expose events to resolve dependencies like this. Different OSes provide assorted OS-specific mechanisms to help with this scenarios. For example, there is SetDllDirectoryW on Windows or RPATH on Unix. |
|
@jkotas , thanks mate, that was what I expected. |
|
@dsyme - if you are okay bundling the libtorch native libs with torchsharp then it will work fine: It produced this. I won't submit a pr, i'm not sure how your build works. |
|
Closing as external |
|
OK, thank you, I'll find some kind of resolution. |
|
@cartermp The packages work OK with applications. From what I see there is nothing wrong with the packages as such - the problem is with our dynamic loader, which doesn't handle transitive native references. (Applications handle this by copying all native DLLs to the one directory on build) I'm not saying it's easy to fix, but it feels like the problem is with us, and could hit us with any packages that rely on transitive native dependencies, so I'll reopen the bug if that's ok. That said I will try to find a workaround to arrange the TorchSharp native packages so they are non-transitive.
@KevinRansom Given that in F#/.NET Interactive we are loading DLLs directly from the package directories, it does kind of feel like we should be using these mecahnisms to augment the native loader load paths. Hard to see any other systematic way to solve this @jkotas Did you mean AddDllDirectory?
@KevinRansom Unfortunately this is not a practical solution.
Tricky problem |
dsyme
added
Area-FSI
Bug
Impact-Medium
|
I've documented this from the TorchSharp perspective here: dotnet/TorchSharp#169 (I can see that we're not going to make this a high-priority thing for .NET Interactive and F# Interactive unless we hit other packages that have transitive native references.) |
|
Yes, this feels like a very niche thing that is low severity |
You are right. |
|
@dsyme, it shouldn't be too hard to make a change to also use this mechanism, l will put something together, hopefully over the weekend. You can let me know if it works for. So ... my linux is not great, however, it seems that rpath is a string embedded into the library that has a dependency. This would require TorchSharp to embed this string for Linux, and to the best of my knowledge the Windows dll loader has no equivalent, so we would still need a windows solution. The linux equivalent of AddDllDirectory is probably LD_LIBRARY_PATH. Which I can set after package resolution, but before dll load. Because it is an environment variable, if developers use fsi to spawn new processes they are also going to see this variable, which is somewhat dll hellish. Although I suppose I could swap it in before we do the load, and back out afterwards. Given that it is a dll load operation, that is bound to be vastly more expensive than swapping out an environment variable. @jkotas , @dsyme could I ask you both to check my PR, if an when I implement it, and see if it is not too terrible. Thanks Kevin |
It's ok, don't worry. I've come to the conclusion that all these native DLLs need to be in the same directory anyway. They register the "torch implementation directory" in a common registry in some way, and it looks like they all have to be in the same place otherwise we get whacky errors like "Key already registered with the same priority: GroupSpatialSoftmax" I'll think about what to do. Awkward but hey |
|
Okay mate, may I close this issue? |
Also, setting process environment variables is not thread safe on Unix that comes with its own set of problems... |
|
For really off-the-wall shenanigans, from what I'm reading you could use |
|
@baronfel, lol. That would be super cool but we would prefer not to copy files in #r, if we copied files, we would shove them in the same directory, and wouldn't have an issue. I am sort of thinking of adding an option that will do that for these really tricky scenarios. However, Don doesn't need it anymore so I'm not going to rush to do something, even real cool nerdy stuff :-) |
This is still, I think, in some sense a bug in the F# and .NET Interactive loading experience of packages. I think we can only close the issue if we document the limitation. Are these docs up-to-date? They look a bit dated at first glance? https://github.com/fsharp/fslang-design/blob/master/tooling/FST-1027-fsi-references.md |
|
They don't discuss native dependency resolution at all. If we think it's a bug, I can prepare a fix, I would rather fix it now, than in two years time, when I've forgotten all of this stuff. |
|
Could you please update the specs to include information on #r for packages with native dependencies ? The specs should say what is meant to work and what isn't - right now it's a little hard to tell what the intended spec is. Here's an approximate spec, maybe you can work from this? Spec: Dynamic loading of packages containing native DLLsDynamic loading of packages containing native DLLs is supported by adding an event handler to This handler consults current architecture and platform settings plus resolved package metadata and files across all dynamically referenced packages to look for a matching native DLL and then dynamically loads that DLL using an internal NativeAssemblyLoadContext that implements This process is not triggered for transitive native-to-native references, which are resolved with respect to the native DLL using standard rules of the operating system. Normally this means any transitive native dependencies must sit next to the native DLL at time of load. That spec seems pretty clear to me and I'm pretty sure we shouldn't rush to use any native library loading functionality that .NET doesn't provide (even if that means we can't reasonable support transitive loading of native components with native-to-native references across multiple packages). It's just a total can of worms. I'm just going to have to work out an approach that works for these horrific Torch native components. Most the complexity is to do with the vast size of the native components involved. So much AI. |
|
@dsyme, I can add a switch that copies files to a single directory on resolution. Sort of like publish lite. That would take care of your issues. And mean you don't have to deal with the size issue. It wouldn't be the default it would be opt-in so normally scripts wouldn't have to deal with the issues. It would also mean that we have an approach for transitive package dependencies, for when project build works and scripting fails. What do you think? Kevin |
For my use cases the problem is that this doesn't deal with size - we would end up consuming 1-2GB of copy and storage each .NET/F# interactive invocation (when running on the GPU - less for CPU Torch binaries), which is a significant pause time in itself. These native binaries are just vast (even if they don't all get paged in). I think this is such a special case that we should just settle on where we are at the moment - with the spec above - and find some workaround for TorchSharp. |
|
OK |
|
So for now this is by design, and Don updated the rfc to not the transitive native dependency limitation. |
|
One workaround I have used in the past is to add the native dlls to the system.environment "Path" variable, dynamically in the script. Here is a snippet I have used to reference native dlls for ML.Net in the past: open System
let path = Environment.GetEnvironmentVariable("path")
let path' =
path
+ ";" + "c:\users\admin\.nuget\packages\microsoft.ml\1.5.2\runtimes\win-x64\native\LdaNative.dll"
+ ";" + "c:\users\admin\.nuget\packages\microsoft.ml.cpumath\1.5.2\runtimes\win-x64\nativeassets\netstandard2.0\CpuMathNative.dll"
+ ";" + "c:\users\admin\.nuget\packages\microsoft.ml.fasttree\1.5.2\runtimes\win-x64\native\FastTreeNative.dll"
+ ";" + "c:\users\admin\.nuget\packages\microsoft.ml.mkl.components\1.5.2\runtimes\win-x64\native\SymSgdNative.dll"
+ ";" + "c:\users\admin\.nuget\packages\microsoft.ml.recommender\0.17.2\runtimes\win-x64\native\MatrixFactorizationNative.dll"
Environment.SetEnvironmentVariable("path",path')Note that adding the full path to the dll works fine. Maybe just adding directories would also be ok but I have not tested. Also now with "nuget: ..." style references I don't have to do the above except for one dll that is under the "nativeassets\netstandard2.0" directory. |
|
@fwaris thanks, nativeassets was a new one on me. I will update probing to support it: NuGet/Home#2782 |
|
@KevinRansom FYI I just built a recommender model in ML.Net with FSI packaged with vs2019 preview. FSI could not find the MatrixFactorizationNative.dll. I had to add the directory to the "PATH" variable. Here is the script to load the packages and set the environment that worked for me: #r "nuget: Microsoft.ML.AutoML, Version=0.17.2"
#r "nuget: Microsoft.ML.Recommender"
let userProfile = System.Environment.GetEnvironmentVariable("UserProfile")
let packageRoot = $@"{userProfile}\.nuget\packages"
let nativeLib = $@"{packageRoot}\microsoft.ml.cpumath\1.5.2\runtimes\win-x64\nativeassets\netstandard2.0"//CpuMathNative.dll"
let nativeLib2 = $@"{packageRoot}\microsoft.ml.recommender\0.17.2\runtimes\win-x64\native"//MatrixFactorizationNative.dll"
let path = System.Environment.GetEnvironmentVariable("path")
let path' = path + ";" + nativeLib + ";" + nativeLib2
System.Environment.SetEnvironmentVariable("path",path') |
|
Thanks for the information. |
Native DLLs are not being found in
libtorch-cpupackage which is referenced transitively from TorchSharp and DiffSharpAnalysis
Possible causes either
There is no managed DLL in the
libtorch-cpupackage (and hence the native resolution logic decides it doesn't need to probe around in that package)There is a problem with transitive native dependency - in this repro, there are two native DLLs ("libLibTorchSharp.so" from TorchSharp and
libtorch.sofromlibtorch-cpu)./ The first is being found but the load is failing due to the transitive reference on the second.The relevant resolved transitive package versions for the repro are:
Repro steps
A simpler repro might be this (though I'm not certain NativeLibrary.Load triggers resolution using the handlers)
Expected behavior
This works
Actual behavior
Known workarounds
This is a workaround to force the load of the native DLL that is not being found:
On Linux:
Put together these are:
Related information
It's possible there is something wrong with the packages but this works when referenced from a project.
The text was updated successfully, but these errors were encountered: