Allow extra control for mesh partitioning from python

[jorgensd]

Currently, you cannot chose a partitioner (say scotch, parmetis, kahip) and select GhostMode.shared_facet
from Python.

This is possible from C++, as stated in: #2591.

This PR adds this option to Python.
Default behavior will be to use GhostMode.none for any input partitioner, and would therefore not cause confusion for
custom partitioners (which has to handle the ghosting themselves).

MWE:

import ipyparallel
from dolfinx.mesh import GhostMode


def create_mesh(ghost_mode):
    from mpi4py import MPI
    import numpy as np
    import ufl
    import dolfinx

    if MPI.COMM_WORLD.rank == 0:
        quadrilaterals = np.array([], dtype=np.int64)
        quad_points = np.array([[0, 0], [0.3, 0]], dtype=np.float64)
    elif MPI.COMM_WORLD.rank == 1:
        quadrilaterals = np.array(
            [[0, 1, 3, 4], [1, 2, 4, 5]], dtype=np.int64)
        quad_points = np.array(
            [[1, 0], [0, 1], [0.5, 1], [1, 1]], dtype=np.float64)
    else:
        quad_points = np.empty((0, 2), dtype=np.float64)
        quadrilaterals = np.empty((0, 4), dtype=np.int64)

    ufl_quad = ufl.Mesh(ufl.VectorElement(
        "Lagrange", ufl.quadrilateral, 1))

    partitioner = dolfinx.mesh.create_cell_partitioner(
        dolfinx.graph.partitioner_scotch(), ghost_mode)
    quad_mesh = dolfinx.mesh.create_mesh(
        MPI.COMM_WORLD, quadrilaterals, quad_points, ufl_quad, partitioner=partitioner)
    output = f"{quad_mesh.comm.rank}: {ghost_mode}, "
    output += f"num cells local {quad_mesh.topology.index_map(2).size_local}, "
    output += f"num ghosts {quad_mesh.topology.index_map(2).num_ghosts}"
    return output


with ipyparallel.Cluster(engines="mpi", n=2) as rc:
    view = rc.broadcast_view()
    for ghost_mode in [GhostMode.none, GhostMode.shared_facet]:
        output = view.apply_sync(create_mesh, ghost_mode=ghost_mode)
        print("\n".join(output))

yields:

0: GhostMode.none, num cells local 1, num ghosts 0
1: GhostMode.none, num cells local 1, num ghosts 0
0: GhostMode.shared_facet, num cells local 1, num ghosts 1
1: GhostMode.shared_facet, num cells local 1, num ghosts 1

default behavior is still sending in GhostMode.none which means that when using custom-partitioners, we are not sending in the wrong ghost mode.

[jorgensd]

@chrisrichardson @garth-wells any thoughts? Or are we fine to merge?