Cartesian Topologies in MPI

MPI provides a way to arrange processes in a Cartesian grid, which is especially useful for applications with structured grids, such as finite difference or finite volume solvers.

Key Features

  • Define a Cartesian topology for processes in a communicator.
  • Simplify neighbor communication by using relative shifts in grid dimensions.
  • Enable wrap-around (periodic) boundaries for dimensions.

Main Functions

1. MPI_Cart_create: Create a Cartesian Topology

This function creates a new communicator with a Cartesian process topology.

Function Signature:

int MPI_Cart_create(MPI_Comm old_comm, int ndims, const int dims[], const int periods[], int reorder, MPI_Comm *new_comm);

Parameters:

  • old_comm: Existing communicator (usually MPI_COMM_WORLD).
  • ndims: Number of dimensions in the Cartesian grid.
  • dims: Array specifying the number of processes in each dimension.
  • periods: Array specifying whether each dimension is periodic (1) or not (0).
  • reorder: Allows process ranks to be reordered (1) or not (0).
  • new_comm: New communicator with the Cartesian topology.

Example:

int dims[2] = {2, 2};          // 2x2 grid
int periods[2] = {1, 0};       // Periodic in one dimension
MPI_Comm cart_comm;
MPI_Cart_create(MPI_COMM_WORLD, 2, dims, periods, 0, &cart_comm);

2. MPI_Cart_coords: Get Coordinates of a Process

This function retrieves the Cartesian coordinates of a process in the grid.

Function Signature:

int MPI_Cart_coords(MPI_Comm comm, int rank, int maxdims, int coords[]);

Parameters:

  • comm: Cartesian communicator.
  • rank: Rank of the process in the communicator.
  • maxdims: Maximum number of dimensions.
  • coords: Array to store the coordinates.

Example:

int coords[2];
MPI_Cart_coords(cart_comm, rank, 2, coords);
printf("Rank %d has coordinates (%d, %d)\n", rank, coords[0], coords[1]);

3. MPI_Cart_rank: Get Rank from Coordinates

This function computes the rank of a process given its Cartesian coordinates.

Function Signature:

int MPI_Cart_rank(MPI_Comm comm, const int coords[], int *rank);

Parameters:

  • comm: Cartesian communicator.
  • coords: Cartesian coordinates of the process.
  • rank: Rank of the process in the communicator.

Example:

int rank;
int coords[2] = {1, 0};
MPI_Cart_rank(cart_comm, coords, &rank);
printf("Coordinates (1,0) correspond to rank %d\n", rank);

4. MPI_Cart_shift: Determine Neighbors

This function finds the ranks of neighboring processes in a specific dimension.

Function Signature:

int MPI_Cart_shift(MPI_Comm comm, int direction, int disp, int *source, int *dest);

Parameters:

  • comm: Cartesian communicator.
  • direction: Dimension in which to find neighbors.
  • disp: Displacement (e.g., 1 for next neighbor).
  • source: Rank of the source neighbor.
  • dest: Rank of the destination neighbor.

Example:

int source, dest;
MPI_Cart_shift(cart_comm, 0, 1, &source, &dest);
printf("Rank %d: source in dim 0 = %d, dest in dim 0 = %d\n", rank, source, dest);

Full Example: 2D Cartesian Grid

#include <mpi.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
    MPI_Init(&argc, &argv);

    int rank, size;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    int dims[2] = {3, 2};
    MPI_Dims_create(size, 2, dims);  // Automatically calculate grid dimensions

    int periods[2] = {1, 0};  // Periodic in the first dimension
    MPI_Comm cart_comm;
    MPI_Cart_create(MPI_COMM_WORLD, 2, dims, periods, 0, &cart_comm);

    int coords[2];
    MPI_Cart_coords(cart_comm, rank, 2, coords);
    printf("Rank %d -> Coordinates (%d, %d)\n", rank, coords[0], coords[1]);

    int source, dest;
    MPI_Cart_shift(cart_comm, 0, 1, &source, &dest);
    printf("Rank %d -> Source = %d, Dest = %d\n", rank, source, dest);

    MPI_Comm_free(&cart_comm);
    MPI_Finalize();
    return 0;
}

Summary Table

FunctionPurpose
MPI_Cart_createCreates a Cartesian topology communicator.
MPI_Cart_coordsRetrieves Cartesian coordinates of a rank.
MPI_Cart_rankConverts coordinates to a rank.
MPI_Cart_shiftFinds neighbors in a specific direction.