Check MPI error codes

[IgorBaratta]

In the following code MPI_Allreduce fails silently (but an error code is actually returned).

#include <dolfinx.h>
#include <dolfinx/fem/petsc.h>
#include <iostream>

using namespace dolfinx;

int main(int argc, char* argv[])
{
  dolfinx::init_logging(argc, argv);
  PetscInitialize(&argc, &argv, nullptr, nullptr);

  {
    MPI_Comm comm{MPI_COMM_WORLD};
    MPI_Comm_dup(MPI_COMM_WORLD, &comm);

    [[maybe_unused]] int rank = dolfinx::MPI::rank(comm);
    [[maybe_unused]] int ierr0 = MPI_Comm_free(&comm);
    // dolfinx::MPI::assert_and_throw(MPI_COMM_WORLD, ierr0);

    int size = 0;
    [[maybe_unused]] int ierr1
        = MPI_Allreduce(&rank, &size, 1, MPI_INT, MPI_SUM, comm);
    // dolfinx::MPI::assert_and_throw(MPI_COMM_WORLD, ierr1);

    assert(size == dolfinx::MPI::size(MPI_COMM_WORLD));
  }

  PetscFinalize();
  return 0;
}

This PR adds a function to check whether an error code returned by an MPI function is equal to MPI_SUCCESS. If the check fails then it prints a useful error message and aborts.

Fixes #2058.

[francesco-ballarin]

Related to #2058 ?

[IgorBaratta]

Related to #2058 ?

Yes, it is. If merged, I think it would close #2058.

[garth-wells]

I don't think the function name is quite right. assert might imply that the check is skipped in non-debug mode, and the function doesn't 'throw' an exception; it just aborts.

[garth-wells]

'runtime error' is a bit vague. Does it just 'abort'?

[IgorBaratta]

Yes, I need to review documentation and function name.
I was throwing a runtime error before, but it can cause a deadlock when MPI_Abort fails.
This can happen when the communicator is NULL (reusing a communicator that has been freed) followed by a barrier with MPI_COMM_WORLD.
So I think a sensible solution is to forcibly abort the execution while still printing the error message.