Help needed with demos/simpleBuildAgainstTrilinos. TPL_LIST empty

[abodado]

I am trying to compile the simpleBuildAgainstTrilinos example. I think I was successful in building Trilinos, but some of the build output confuses me. My config file was as follows:

#!/bin/bash
export CC=gcc
export OMPI_CC=gcc
export CXX=g++
export OMPI_CXX=g++
export FC=gfortran
export OMPI_FC=gfortran

export TRILINOS_ROOT=/s/mySpace/trilinos
export TRILINOS_HOME=${TRILINOS_ROOT}/Trilinos-master-27Aug2024
export TrilinosBuildDir=${TRILINOS_HOME}/buildTrilinos
export TrilinosInstallDir=${TRILINOS_HOME}/__INSTALL
mkdir ${TrilinosBuildDir}
mkdir ${TrilinosInstallDir}

cd ${TrilinosBuildDir}
export MPI_ROOT=/apps/mpi/openmpi/openmpi-4.0.2
export BLASPATH=${TRILINOS_ROOT}/lapack-3.10.1/INSTALL/lib64

cmake \
-D CMAKE_INSTALL_PREFIX=${TrilinosInstallDir} \
-D TPL_ENABLE_MPI=ON \
-D TPL_ENABLE_BLAS:BOOL=ON \
-D TPL_ENABLE_LAPACK:BOOL=ON \
-D MPI_BASE_DIR=${MPI_ROOT} \
-D BLAS_LIBRARY_DIRS=${BLASPATH} \
-D TPL_BLAS_LIBRARIES="-L/${BLASPATH} -lblas -L/usr/lib64/;-lgfortran;-lgomp;-lm" \
-D LAPACK_LIBRARY_DIRS=${BLASPATH} \
-D Trilinos_ENABLE_Tpetra=ON \
-D CMAKE_C_COMPILER=$(which mpicc) \
-D CMAKE_CXX_COMPILER=$(which mpicxx) \
${TRILINOS_HOME}
make -j4 install

The output related to TPLs was:

Getting information for all enabled external packages/TPLs ...

Processing enabled external package/TPL: MPI (enabled explicitly, disable with -DTPL_ENABLE_MPI=OFF)
-- MPI_LIBRARY_NAMES=''
Processing enabled external package/TPL: BLAS (enabled explicitly, disable with -DTPL_ENABLE_BLAS=OFF)
-- BLAS_LIBRARY_NAMES='blas blas_win32'
-- TPL_BLAS_LIBRARIES='-L/s/mySpace/trilinos/lapack-3.10.1/INSTALL/lib64 -lblas -L/usr/lib64/;-lgfortran;-lgomp;-lm'
Processing enabled external package/TPL: LAPACK (enabled explicitly, disable with -DTPL_ENABLE_LAPACK=OFF)
-- LAPACK_LIBRARY_NAMES='lapack lapack_win32'
-- Must find at least one lib in each of the lib sets "lapack lapack_win32"
-- Searching for libs in LAPACK_LIBRARY_DIRS='/s/mySpace/trilinos/lapack-3.10.1/INSTALL/lib64'
-- Searching for a lib in the set "lapack lapack_win32":
--   Searching for lib 'lapack' ...
--     Found lib '/s/mySpace/trilinos/lapack-3.10.1/INSTALL/lib64/liblapack.a'
-- TPL_LAPACK_LIBRARIES='/s/mySpace/trilinos/lapack-3.10.1/INSTALL/lib64/liblapack.a'
Processing enabled external package/TPL: DLlib (enabled explicitly, disable with -DTPL_ENABLE_DLlib=OFF)
-- Attempting to tentatively enable TPL 'DLlib' ...
-- DLlib_LIBRARY_NAMES='dl'
-- Must find at least one lib in each of the lib sets "dl"
-- Searching for libs in DLlib_LIBRARY_DIRS=''
-- Searching for a lib in the set "dl":
--   Searching for lib 'dl' ...
--     Found lib '/usr/lib64/libdl.so'
-- TPL_DLlib_LIBRARIES='/usr/lib64/libdl.so'
-- Attempt to tentatively enable TPL 'DLlib' passed!

I noticed that MPI_LIBRARY_NAMES is blank in the output. Also there is no confirmation that BLAS or LAPACK "passed" as was the case for "DLlib".

To build the sample app, demos/simpleBuildAgainstTrilinos/app.cpp, I created a new folder called myApp and created a config file there:

#!/bin/bash

export CC=gcc
export OMPI_CC=gcc
export CXX=g++
export OMPI_CXX=g++

export TRILINOS_MAIN=/s/mySpace/trilinos
export TRILINOS_HOME=${TRILINOS_MAIN}/Trilinos-master-27Aug2024
export TrilinosInstallDir=${TRILINOS_HOME}/__INSTALL

export myApp_HOME=${TRILINOS_MAIN}/myApp
export myAppBuildDir=${myApp_HOME}/buildMyApp
mkdir ${myAppBuildDir}
cd ${myAppBuildDir}

cmake \
-D CMAKE_PREFIX_PATH=${TrilinosInstallDir} \
-D CMAKE_C_COMPILER=$(which mpicc) \
-D CMAKE_CXX_COMPILER=$(which mpicxx) \
${TRILINOS_HOME}/demos/simpleBuildAgainstTrilinos
make -j4

This process uses the CMakeLists.txt file found in .../Trilinos-master-27Aug2024/demos/simpleBuildAgainstTrilinos/. The update I made to the CMakeLists.txt is the CMake version. Updated to 3.27.0 minimum version.

The output from the build (from running that config file was):

-- The C compiler identification is GNU 8.5.0
-- The CXX compiler identification is GNU 8.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /apps/mpi/openmpi/openmpi-4.0.2/bin/mpicc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /apps/mpi/openmpi/openmpi-4.0.2/bin/mpicxx - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Enabled Kokkos devices: SERIAL

Found Trilinos!  Here are the details: 
   Trilinos_DIR = /s/mySpace/trilinos/Trilinos-master-27Aug2024/__INSTALL/lib64/cmake/Trilinos
   Trilinos_VERSION = 16.1.0
   Trilinos_PACKAGE_LIST = Tpetra;TpetraCore;TpetraTSQR;KokkosKernels;Teuchos;TeuchosKokkosComm;TeuchosKokkosCompat;TeuchosRemainder;TeuchosNumerics;TeuchosComm;TeuchosParameterList;TeuchosParser;TeuchosCore;Kokkos
   Trilinos_LIBRARIES = Teuchos::all_libs;Tpetra::all_libs
   Trilinos_INCLUDE_DIRS = /s/mySpace/trilinos/Trilinos-master-27Aug2024/__INSTALL/include
   Trilinos_TPL_LIST = 
   Trilinos_TPL_LIBRARIES = 
   Trilinos_BUILD_SHARED_LIBS = FALSE
End of Trilinos details

-- Configuring done (0.6s)
-- Generating done (0.0s)
-- Build files have been written to: /s/mySpace/trilinos/myApp/buildMyApp
[ 50%] Building CXX object CMakeFiles/MyApp.dir/app.cpp.o
[100%] Linking CXX executable MyApp
/s/mySpace/trilinos/Trilinos-master-27Aug2024/__INSTALL/lib64/libkokkoskernels.a(KokkosBlas_Host_tpl.cpp.o): In function `KokkosBlas::Impl::HostBlas<float>::scal(int, float, float*, int)':
KokkosBlas_Host_tpl.cpp:(.text+0x25): undefined reference to `sscal_'

There were many more link errors related to KokkosBlas_Host_tpl. Do the link errors have anything to do with the "Trilinos_TPL_LIST" and "Trilinos_TPL_LIBRARIES" being empty in the above output? Or what could be the reason that Kokkos does not find the blas libraries?

[jhux2]

@abodado Could you enable Tpetra's tests to verify that your build is ok?

cmake \
-D CMAKE_INSTALL_PREFIX=${TrilinosInstallDir} \
-D TPL_ENABLE_MPI=ON \
-D TPL_ENABLE_BLAS:BOOL=ON \
-D TPL_ENABLE_LAPACK:BOOL=ON \
-D MPI_BASE_DIR=${MPI_ROOT} \
-D BLAS_LIBRARY_DIRS=${BLASPATH} \
-D TPL_BLAS_LIBRARIES="-L/${BLASPATH} -lblas -L/usr/lib64/;-lgfortran;-lgomp;-lm" \
-D LAPACK_LIBRARY_DIRS=${BLASPATH} \
-D Trilinos_ENABLE_Tpetra=ON \
-D Tpetra_ENABLE_TESTS=ON \                                          <<<<<<<<<<<<<<<<<<<<  enables tests
-D CMAKE_C_COMPILER=$(which mpicc) \
-D CMAKE_CXX_COMPILER=$(which mpicxx) \
${TRILINOS_HOME}

[jhux2]

So reconfigure, rebuild, and then in the build directory, run ctest.

[abodado]

Well, that's interesting. Just by adding the line "-D Tpetra_ENABLE_TESTS=ON " I run into issues with the Trilinos build. It already does not find the Blas functions.

[ 31%] Linking CXX executable TpetraTSQR_Combine.exe
../../../teuchos/numerics/src/libteuchosnumerics.a(Teuchos_BLAS.cpp.o): In function `Teuchos::BLAS<int, float>::ROTG(float*, float*, float*, float*) const':
Teuchos_BLAS.cpp:(.text+0xd): undefined reference to `srotg_'

I believe it has something to do with the following directive in my config file:
-D TPL_BLAS_LIBRARIES="-L/${BLASPATH} -lblas -L/usr/lib64/;-lgfortran;-lgomp;-lm" \

When I simply specified
-D BLAS_LIBRARY_DIRS=${BLASPATH} \

The build output shows that libblas.a was found. I believed this was progress as this was not the case previously. But then I got link errors associated with LAPACK not finding gfortran libraries. So I updated the config file to have:

-D TPL_LAPACK_LIBRARIES="-L/usr/lib64/;-lgfortran;-lgomp;-lm" \

But I still get errors and lapack library is not found:

Processing enabled external package/TPL: MPI (enabled explicitly, disable with -DTPL_ENABLE_MPI=OFF)
-- MPI_LIBRARY_NAMES=''
Processing enabled external package/TPL: BLAS (enabled explicitly, disable with -DTPL_ENABLE_BLAS=OFF)
-- BLAS_LIBRARY_NAMES='blas blas_win32'
-- Must find at least one lib in each of the lib sets "blas blas_win32"
-- Searching for libs in BLAS_LIBRARY_DIRS='/s/mySpace/trilinos/lapack-3.10.1/INSTALL/lib64'
-- Searching for a lib in the set "blas blas_win32":
--   Searching for lib 'blas' ...
--     Found lib '/s/mySpace/trilinos/lapack-3.10.1/INSTALL/lib64/libblas.a'
-- TPL_BLAS_LIBRARIES='/s/mySpace/trilinos/lapack-3.10.1/INSTALL/lib64/libblas.a'
Processing enabled external package/TPL: LAPACK (enabled explicitly, disable with -DTPL_ENABLE_LAPACK=OFF)
-- LAPACK_LIBRARY_NAMES='lapack lapack_win32'
-- TPL_LAPACK_LIBRARIES='-L/usr/lib64/;-lgfortran;-lgomp;-lm'

I am adding the directive "-D TPL_LAPACK[or BLAS]_LIBRARIES="-L/usr/lib64/;-lgfortran;-lgomp;-lm" " due to comments on
#8632

I will keep playing around with this and update on any progress. But any additional comments/hints/suggestions from the Trilinos team are more than welcome!

[jhux2]

@trilinos/framework Should this work as specified in the demo directory? In particular, is there anything missing regarding TPLs?

[abodado]

I have made some changes to the config script for both the trilinos build and the app build. These changes seemed to have worked and the build is successful.

My new config file to build Trilinos:

#!/bin/bash
export CC=gcc
export OMPI_CC=gcc
export CXX=g++
export OMPI_CXX=g++
export FC=gfortran
export OMPI_FC=gfortran
export TRILINOS_ROOT=/s/mySpace/trilinos
export TRILINOS_HOME=${TRILINOS_ROOT}/Trilinos-master-27Aug2024
export TrilinosBuildDir=${TRILINOS_HOME}/buildTrilinos
export TrilinosInstallDir=${TRILINOS_HOME}/__INSTALL
mkdir ${TrilinosBuildDir}
mkdir ${TrilinosInstallDir}
cd ${TrilinosBuildDir}
export MPI_ROOT=/apps/mpi/openmpi/openmpi-4.0.2
export BLASPATH=${TRILINOS_ROOT}/lapack-3.10.1/INSTALL/lib64

cmake \
-D CMAKE_INSTALL_PREFIX=${TrilinosInstallDir} \
-D TPL_ENABLE_MPI=ON \
-D TPL_ENABLE_BLAS:BOOL=ON \
-D TPL_ENABLE_LAPACK:BOOL=ON \
-D MPI_BASE_DIR=${MPI_ROOT} \
-D BLAS_LIBRARY_DIRS=${BLASPATH} \
-D LAPACK_LIBRARY_DIRS=${BLASPATH} \
-D CMAKE_EXE_LINKER_FLAGS:STRING="-lgfortran -lgomp -lm" \
-D Trilinos_ENABLE_Tpetra=ON \
-D Tpetra_ENABLE_TESTS=ON \
-D CMAKE_C_COMPILER=$(which mpicc) \
-D CMAKE_CXX_COMPILER=$(which mpicxx) \
${TRILINOS_HOME}
make -j4 install

I believe the important change was to move the library references (that I previously had listed under TPL_BLAS_LIBRARIES) to the following:
-D CMAKE_EXE_LINKER_FLAGS:STRING="-lgfortran -lgomp -lm" \

This was mentioned in #8632, but it wasn't clear that that was the final solution for that build issue. But I tried it, and for me, it worked. As can be seen in the trilinos build output, there are explicit messages that both blas and lapack are found (whereas previously, this had not been the case).

Processing enabled external package/TPL: MPI (enabled explicitly, disable with -DTPL_ENABLE_MPI=OFF)
-- MPI_LIBRARY_NAMES=''
Processing enabled external package/TPL: BLAS (enabled explicitly, disable with -DTPL_ENABLE_BLAS=OFF)
-- BLAS_LIBRARY_NAMES='blas blas_win32'
-- Must find at least one lib in each of the lib sets "blas blas_win32"
-- Searching for libs in BLAS_LIBRARY_DIRS='/s/mySpace/trilinos/lapack-3.10.1/INSTALL/lib64'
-- Searching for a lib in the set "blas blas_win32":
--   Searching for lib 'blas' ...
--     Found lib '/s/mySpace/trilinos/lapack-3.10.1/INSTALL/lib64/libblas.a'
-- TPL_BLAS_LIBRARIES='/s/mySpace/trilinos/lapack-3.10.1/INSTALL/lib64/libblas.a'
Processing enabled external package/TPL: LAPACK (enabled explicitly, disable with -DTPL_ENABLE_LAPACK=OFF)
-- LAPACK_LIBRARY_NAMES='lapack lapack_win32'
-- Must find at least one lib in each of the lib sets "lapack lapack_win32"
-- Searching for libs in LAPACK_LIBRARY_DIRS='/s/mySpace/trilinos/lapack-3.10.1/INSTALL/lib64'
-- Searching for a lib in the set "lapack lapack_win32":
--   Searching for lib 'lapack' ...
--     Found lib '/s/mySpace/trilinos/lapack-3.10.1/INSTALL/lib64/liblapack.a'
-- TPL_LAPACK_LIBRARIES='/s/mySpace/trilinos/lapack-3.10.1/INSTALL/lib64/liblapack.a'

My updated config script for the build of app.cpp is now:

#!/bin/bash
export CC=gcc
export OMPI_CC=gcc
export CXX=g++
export OMPI_CXX=g++
export TRILINOS_MAIN=/s/mySpace/trilinos
export TRILINOS_HOME=${TRILINOS_MAIN}/Trilinos-master-27Aug2024
export TrilinosInstallDir=${TRILINOS_HOME}/__INSTALL
export myApp_HOME=${TRILINOS_MAIN}/myApp
export myAppBuildDir=${myApp_HOME}/buildMyApp
mkdir ${myAppBuildDir}
cd ${myAppBuildDir}
cmake \
-D CMAKE_PREFIX_PATH=${TrilinosInstallDir} \
-D CMAKE_C_COMPILER=$(which mpicc) \
-D CMAKE_CXX_COMPILER=$(which mpicxx) \
-D CMAKE_EXE_LINKER_FLAGS:STRING="-lgfortran -lgomp -lm" \
${TRILINOS_HOME}/demos/simpleBuildAgainstTrilinos
make -j4

where (as in the config script for trilinos) I have added the directive "-D CMAKE_EXE_LINKER_FLAGS:STRING="-lgfortran -lgomp -lm" ".

I hope this is helpful to anyone facing a similar issue. Thank you to @jhux2 for the suggestion of building with Tpetra tests which helped me realize the problem was further upstream than I had thought!

[jhux2]

@abodado Glad you figured it out. In the first configure line in the discussion, I should have noticed that the formatting of the BLAS line looks suspicious:

-D TPL_BLAS_LIBRARIES="-L/${BLASPATH} -lblas -L/usr/lib64/;-lgfortran;-lgomp;-lm"

According to the Trilinos Configure, Build, Test, and Install Reference Guide, cmake should detect BLAS/LAPACK if the option is instead

-D TPL_BLAS_LIBRARIES="-L/${BLASPATH};-lblas;-L/usr/lib64/;-lgfortran;-lgomp;-lm"

[abodado]

@jhux2 , Thank you! I see your point. I tried it, and it works. Though I don't get the confirmation that "blas" was found, the sample app did build without link errors. So this is another viable solution.