Trilinos

• Register and download the Trilinos packages from trilinos.sandia.gov, we have tested against version 12.6.x and recommend downloading latest release.

• Extract the downloaded file, and create a build directory inside the newly-created folder.

• Copy the do-configure.sh script from the TeaLeaf repository into the build folder.

• Edit the do-configure.sh script and change the variables

  • CMAKE_INSTALL_PREFIX (if you would like to install to a custom directory)
  • CMAKE_CXX_COMPILER
  • CMAKE_C_COMPILER
  • CMAKE_FORTRAN_COMPILER as necessary.

• If you don't edit the CMAKE_INSTALL_PREFIX variable, Trilinos will be installed in the libs/trilinos folder of the TeaLeaf repository.

• Run the do-configure.sh script, and then make and make install to build and install Trilinos.

• If you would prefer to use a custom build of Trilinos, note that TeaLeaf depends on the packages: Tpetra, Belos and MueLu.

TeaLeaf

Compling

• If you didn't install Trilinos to libs/trilinos, edit the TeaLeaf Makefile and specify the directory that you have installed Trilinos in.
• In many case just typing make in the required software version will work.

If the MPI compilers have different names then the build process needs to notified of this by defining two environment variables, MPI_COMPILER and C_MPI_COMPILER.

For example on some Intel systems:

make MPI_COMPILER=mpiifort C_MPI_COMPILER=mpiicc

Or on Cray systems:

make MPI_COMPILER=ftn C_MPI_COMPILER=cc

OpenMP Build

All compilers use different arguments to invoke OpenMP compilation. A simple call to make will invoke the compiler with -O3. This does not usually include OpenMP by default. To build for OpenMP for a specific compiler a further variable must be defined, COMPILER that will then select the correct option for OpenMP compilation.

For example with the Intel compiler:

make COMPILER=INTEL

Which then append the -openmp to the build flags.

Other supported compiler that will be recognise are:-

• CRAY
• SUN
• GNU
• IBM
• PATHSCALE
• PGI

The default flags for each of these is show below:-

• INTEL: -O3 -ipo
• SUN: -fast
• GNU: -ipo
• XL: -O5
• PATHSCLE: -O3
• PGI: -O3 -Minline
• CRAY: -em Note: that by default the Cray compiler with pick the optimum options for performance.

Other Flags

The default compilation with the COMPILER flag set chooses the optimal performing set of flags for the specified compiler, but with no hardware specific options or IEEE compatability.

To produce a version that has IEEE compatiblity a further flag has to be set on the compiler line.

make COMPILER=INTEL IEEE=1

This flag has no effect if the compiler flag is not set because IEEE options are always compiler specific.

For each compiler the flags associated with IEEE are shown below:-

• INTEL: -fp-model strict –fp-model source –prec-div –prec-sqrt
• CRAY: -hpflex_mp=intolerant
• SUN: -fsimple=0 –fns=no
• GNU: -ffloat-store
• PGI: -Kieee
• PATHSCALE: -mieee-fp
• XL: -qstrict –qfloat=nomaf

Note that the MPI communications have been written to ensure bitwise identical answers independent of core count. However under some compilers this is not true unless the IEEE flags is set to be true. This is certainly true of the Intel and Cray compiler. Even with the IEEE options set, this is not guarantee that different compilers or platforms will produce the same answers. Indeed a Fortran run can give different answers from a C run with the same compiler, same options and same hardware.

Extra options can be added without modifying the makefile by adding two further flags, OPTIONS and C_OPTIONS, one for the Fortran and one for the C options.

make COMPILER=INTEL OPTIONS=-xavx C_OPTIONS=-xavx

Finally, a DEBUG flag can be set to use debug options for a specific compiler.

make COMPILER=PGI DEBUG=1

These flags are also compiler specific, and so will depend on the COMPILER environment variable.

So on a system without the standard MPI wrappers, for a build that requires OpenMP, IEEE and AVX this would look like so:-

make COMPILER=INTEL MPI_COMPILER=mpiifort C_MPI_COMPILER=mpiicc IEEE=1 \
OPTIONS="-xavx" C_OPTIONS="-xavx"

Running TeaLeaf with the Trilinos solver

TeaLeaf is controlled through the tea.in, which defines the problem being run. Setting the option use_trilinos involves the Trilinos solver.

The Trilinos solver is controlled through the Options.xml file, which controls the Krylov method provided by Belos package and the preconditioner provided by the MueLu package. Various sample xml files are provided, but for details on the syntax of the options in the "Belos Solver" and "MueLu Preconditioner" parameter lists you should consult the documentation of the associated Trilinos packages.

The only custom options specific to TeaLeaf are in the "Solver Parameters" parameter list. These are:

<Parameter        name="Belos Solver"                         type="string"   value="CG"/>

which allows the user to set the Krylov method and:

<Parameter        name="MueLu Preconditioner"                 type="bool"     value="true"/>

which enables/disables the MueLu preconditioner.