Detribits (ForTrilinos as a standalone library) #271
Conversation
|
@aprokop: can you take a look at the NOX wrappers? (I'm building against Trilinos 12.18.1.) I can't figure out how the ambiguous call error is happening, but I can't say I'm surprised given the Group overload and the number of default arguments 🙄 Thanks! @jwillenbring FYI for our meeting in 15 minutes |
I assume that you are talking about a standalone build, and not about failing checks here? I'll take a look. |
|
Yep, failing stand-alone build. I’ve got a spack environment under “script” that you can use if you don’t have it installed |
|
I can reproduce the error: |
|
Hmm, so far this does not make any sense. The file was changed in trilinos/Trilinos@2403907 with --- a/packages/nox/src-thyra/NOX_Thyra_Group.H
+++ b/packages/nox/src-thyra/NOX_Thyra_Group.H
@@ -109,12 +109,14 @@ namespace NOX {
\param[in] model ModelEvaluator
\param[in] weightVector Optional diagonal weighting vector for the model.
\param[in] rightWeightVector Optional solution vector weighting
+ \param[in] inv_rightWeightVector Optional inverse solution vector weighting
\param[in] rightScalingFirst Optional bool to select if right scaling should be applied before left scaling
*/
Group(const NOX::Thyra::Vector& initialGuess,
const Teuchos::RCP<const ::Thyra::ModelEvaluator<double> >& model,
const Teuchos::RCP<const ::Thyra::VectorBase<double> >& weightVector = Teuchos::null,
const Teuchos::RCP<const ::Thyra::VectorBase<double> >& rightWeightVector = Teuchos::null,
+ const Teuchos::RCP<::Thyra::VectorBase<double> >& inv_rightWeightVector = Teuchos::null,
const bool rightScalingFirst = false);
/** \brief Power user constructor that takes explicit linear solver objects to handle different combinations.
@@ -133,6 +135,7 @@ namespace NOX {
\param[in] precFactory (Optional) Factory for updating the precondiitoner. If set to Teuchos::null and there is a non-null prec_op, then the preconditioner will be updated using the model evaluator as long as the ModelEvaluator::outArgs supports W_prec.
\param[in] weightVector (Optional) diagonal weighting vector for the model.
\param[in] rightWeightVector Optional solution vector weighting
+ \param[in] inv_rightWeightVector Optional inverse solution vector weighting
\param[in] rightScalingFirst Optional bool to select if right scaling should be applied before left scaling
\param[in] updatePreconditioner Optional bool to select if the Group should auotmatically update the preconditioner matrix values between Newton iterations
\param[in] jacobianIsEvaluated Optional bool, if true this means that the input Jacobian operator (linearOp) has been evaluated externally and is consistent with the initialGuess. In this case, the isJacobian() flag is initialized to true.
@@ -145,6 +148,7 @@ namespace NOX {
const Teuchos::RCP< ::Thyra::PreconditionerFactoryBase<double> >& precFactory,
const Teuchos::RCP<const ::Thyra::VectorBase<double> >& weightVector = Teuchos::null,
const Teuchos::RCP<const ::Thyra::VectorBase<double> >& rightWeightVector = Teuchos::null,
+ const Teuchos::RCP<::Thyra::VectorBase<double> >& inv_rightWeightVector = Teuchos::null,
const bool rightScalingFirst = false,
const bool updatePreconditioner = true,
const bool jacobianIsEvaluated = false); |
|
I pushed a "fix". I still cannot compile on my workstation, but that's due to |
|
@sethrj I'm struggling to compile it with Cuda backend and the whole |
|
I haven’t tried, I’m afraid. Ive never compiled trilinos with it to be honest... |
|
One thing that I noticed in this PR is that somehow it does not pick up my MPI compilers, and uses stuff like |
|
It links using normal mph flags and libraries instead of the compiler wrapper. I have a fix for the fortran mpi failure |
|
OK @aprokop I've got the build working (though using a non-cuda-enabled trilinos). I might wait till spack/spack#17900 goes through to test with Kokkos CUDA, or do you think I should start now and look at ArborX's standalone system to see how it built? |
|
I tried building this PR with Trilinos built just with Serial. I'm getting linking errors I haven't looked at how you find the necessary libraries to put into linking. Wonder why it works for you. |
|
Does your install of Trilinos have the correct install rpath? |
I'm not even sure how to check that. This is Trilinos installed from source using scripts, not Spack. Thus, if spack does something with rpath, it may not be done with independent install |
|
https://gitlab.kitware.com/cmake/community/-/wikis/doc/cmake/RPATH-handling |
|
Yep, |
|
The problem is with the installation of trilinos, not the downstream usage of it. You'll often see "not found" type errors on systems not configured with rpaths, where it expects you to manually set it or set it up via exporting Shared libraries that depend on other shared libraries either need to set the rpath correctly at install time, or require the user to know where the dependencies are and manually set See https://tribits.org/doc/TribitsBuildReference.html#setting-install-rpath for how to modify your cmake install command to work. As it stands, your trilinos installation won't work correctly with any downstream software without manual intervention. |
Sure. My point is that I think it's very common to not configure Trilinos with this options, and we may need to work with this fact. Many projects building against trilinos import few variables that contain the full list of Trilinos libraries. So, instead of linking just with |
|
That may be true; but usually on systems that I've seen with missing RPATHs the sysadmin will at least hardcode the RPATH into the It would be interesting to find out if most trilinos installations are indeed like that -- because in my case (far predating spack) I've had to add extra arguments to cmake code, and then extra RPATH configuration inside of trilinos-based projects, to get installs working correctly. Incidentally, I think the ForTrilinos CMake is linking against not just although since rpaths are for runtime and this is at link time, not quite the same as the problem with missing shared libraries. |
|
Hey @aprokop , I've got everything working and tested except for a failing anasazi solve in the 4-processor The failing line is after Anasazi::ReturnType r = solver_->solve();I tried updating the MueLu solver verbosity to see if that would help shed light on the error, but no luck there. |
```
In file included from ../src/fortrilinos/generated/fortrilinosFORTRAN_wrap.cxx:613:
In file included from ../src/fortrilinos/nox_solver.hpp:39:
../src/fortrilinos/nox_solver_def.hpp:44:29: error: call to constructor of 'NOX::Thyra::Group' is ambiguous
nox_group = rcp(new NOX::Thyra::Group(*(model_->initial_guess), model_,
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/fortrilinos/generated/fortrilinosFORTRAN_wrap.cxx:2079:15: note: in instantiation of member function 'ForTrilinos::NOXSolver<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::setup' requested here
(arg1)->setup(*arg2);
^
/usr/local/spack/var/spack/environments/fortrilinos/.spack-env/view/include/NOX_Thyra_Group.H:115:7: note: candidate constructor
Group(const NOX::Thyra::Vector& initialGuess,
^
/usr/local/spack/var/spack/environments/fortrilinos/.spack-env/view/include/NOX_Thyra_Group.H:143:7: note: candidate constructor
Group(const NOX::Thyra::Vector& initialGuess,
^
```
|
Closing this as its functionality was split into two separate prs. |
This changes ForTrilinos to a standalone project that depends on Trilinos as a third-party library. Among other benefits, this will allow integration into system environments that already have Trilinos installed.