Core Halo Exchange

Overview

Halo exchange allows data uniquely owned by one rank to be shared with other ranks as ghost data. In Cabana, the communication plan for halo exchange is encapsulated in the Halo class and is updated with the gather and scatter functions.

In the figure below all ghosted neighbors (red) from the left MPI rank are copied to the right MPI rank (yellow):

Implementation

Header File: Cabana_Halo.hpp

Usage: Set up communication plan for sharing particle data

Interface

template<class DeviceType>
class Halo

Template Parameters

• DeviceType: The device type for which the halo data will be allocated and where parallel execution will occur

Examples

Example: Halo Exchange Tutorial

Example: Halo Exchange Unit Test

// Get MPI parameters
MPI_Comm_rank( MPI_COMM_WORLD, &comm_rank );
MPI_Comm_size( MPI_COMM_WORLD, &comm_size );

// Define particle data 
using DataTypes = Cabana::MemberTypes<double[3],int>;
using DeviceType = Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>;
Cabana::AoSoA<DataTypes,DeviceType> aosoa( num_tuple );
slice_0 = Cabana::slice<0>( aosoa );

// Define MPI ranks to share to and particle IDs to share
Kokkos::View<int*,DeviceType> export_ranks( "export_ranks", local_num_send );
Kokkos::deep_copy(export_ranks, next_rank);
Kokkos::View<int*,DeviceType> export_ids( "export_ids", local_num_send );
for ( int i = 0; i < local_num_send; ++i )
    export_ids( i ) = i;

// Define unique neighboring MPI ranks
std::vector<int> neighbors = { previous_rank, comm_rank, next_rank };

// Create the halo
Cabana::Halo<DeviceType> halo( MPI_COMM_WORLD, num_tuple, export_ids, export_ranks, neighbors );


// Gather the data for the ghosts from the current rank (could be done for a slice instead)
Cabana::gather( halo, aosoa );

// Scatter the data from the ghosts back to the owning rank
Cabana::scatter( halo, slice_0 );

This is part of the Programming Guide series