Message Passing Interface (MPI)

We start from the serial code we used in the previous chapters, calculates the integral of a function over a range .

To parallelize the trapezoidal integration with MPI, we can divide the integration range among multiple processors. Each processor computes the integral over a smaller sub-range, and then the results from all processors are combined to obtain the final integral. This approach leverages the distributed processing power of the cluster to speed up the computation.

We first recall the serial code:

#include <stdio.h>
#include <math.h>

// Define the function to integrate: f(x) = x^3
double f(double x) {
    return pow(x, 3); // pow(a,b) computes a^b
}

// Trapezoidal rule for numerical integration
double trapezoidal_rule(double (*func)(double), double a, double b, int n) {
    double p = (b - a) / n;                  // Width of each trapezoid
    double sum = 0.5 * (func(a) + func(b));  // End points contribution

    for (int i = 1; i < n; i++) {
        double x = a + i * p;
        sum += func(x);
    }

    return sum * p;
}

int main() {
	double a = 0.0;  // Lower limit of integration
	double b = 1.0;  // Upper limit of integration
	int n = 1000;    // Number of trapezoids (higher n for better accuracy)
	
	printf("This program performs numerical integration of f(x) = x^3 from a = %.2f to b = %.2f using %d trapezoids.\n", a, b, n);
	
	// Check if n is a valid number of trapezoids
	if (n > 0) {
		printf("The number of trapezoids is positive.\n");
	} else if (n < 0) {
		printf("Error: The number of trapezoids is negative.\n");
	} else {
		printf("Error: The number of trapezoids is zero.\n");
	}
	
	// Perform numerical integration
	double result = trapezoidal_rule(f, a, b, n);
	
	printf("The integral of f(x) = x^3 from %.2f to %.2f is approximately: %.5f\n", a, b, result);
	
	return 0;
}

The MPI-parallelised version of the code is:

#include <stdio.h>
#include <math.h>
#include <mpi.h>

// Define the function to integrate: f(x) = x^3
double f(double x) {
    return pow(x, 3); // pow(a,b) computes a^b
}

// Trapezoidal rule for numerical integration over a sub-interval
double trapezoidal_rule(double (*func)(double), double a, double b, int n) {
    double p = (b - a) / n;                  // Width of each trapezoid
    double sum = 0.5 * (func(a) + func(b));  // End points contribution

    for (int i = 1; i < n; i++) {
        double x = a + i * p;
        sum += func(x);
    }

    return sum * p;
}

int main(int argc, char *argv[]) {
    int rank, size;
    double a = 0.0;  // Lower limit of integration
    double b = 1.0;  // Upper limit of integration
    int n = 1000;    // Total number of trapezoids (higher n for better accuracy)
    double local_a, local_b;  // Local limits for each process
    int local_n;  // Number of trapezoids for each process
    double local_result, total_result;  // Local and total integral results

    // Initialize MPI
    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    // Check if n is a positive number
    if (rank == 0) {
        if (n <= 0) {
            printf("Error: The number of trapezoids must be positive.\n");
            MPI_Abort(MPI_COMM_WORLD, 1);
            return 1;
        }
    }

    // Divide the interval [a, b] among processes
    double h = (b - a) / size;  // Width of each sub-interval
    local_a = a + rank * h;
    local_b = local_a + h;
    local_n = n / size;

    // Each process computes the integral over its sub-interval
    local_result = trapezoidal_rule(f, local_a, local_b, local_n);

    // Use MPI_Reduce to sum up the results from all processes
    MPI_Reduce(&local_result, &total_result, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);

    // The root process (rank 0) prints the final result
    if (rank == 0) {
        printf("The integral of f(x) = x^3 from %.2f to %.2f is approximately: %.5f\n", a, b, total_result);
    }

    // Finalize MPI
    MPI_Finalize();
    return 0;
}

How To Compile and Run

To compile and run an MPI program, you need an MPI compiler (like mpicc for C and mpif90 for Fortran) and an MPI runtime to execute the code across multiple processes. Here’s a step-by-step guide for compiling and running the code.

Compile

Use mpicc, the MPI C compiler, to compile the code. Assuming the code is saved in a file named mpi_integration.c, the command is:

mpicc -o mpi_integration mpi_integration.c -lm

In Fortran, assuming the code is saved in a file named mpi_integration.f90:

mpif90 -o mpi_integration mpi_integration.f90

Run

You can use the mpirun or mpiexec command to run the executable with multiple processes.

For example, to run the program with 4 processes:

mpirun -np 4 ./mpi_integration
  • mpirun: Launches the MPI job.
  • -np 4: Specifies the number of processes to use.
  • ./mpi_integration: Runs the compiled executable.

MPI_Init and MPI_Finalize

MPI_Init and MPI_Finalize are essential functions that manage the lifecycle of an MPI application. They are used to initialize and finalize the MPI environment, allowing multiple processes to communicate with each other through message-passing.

The function MPI_Init initializes the MPI environment and must be called before any other MPI function. It prepares the program to use the MPI library by setting up the required resources and creating a "communicator", which establishes a context for all participating processes to communicate.

The MPI_Finalize function shuts down the MPI environment. It releases resources, terminates MPI communication, and ensures a clean exit for the parallel application. All processes in the communicator must reach MPI_Finalize for the program to end correctly.

The signatures in C are:

int MPI_Init(int *argc, char ***argv);
int MPI_Finalize(void);
  • MPI_Init: The function takes pointers to argc and argv from main, allowing MPI to process command-line arguments.
  • MPI_Finalize: Takes no parameters and simply finalizes the MPI environment.

The signature in Fortran are:

CALL MPI_INIT(ierr)
CALL MPI_FINALIZE(ierr)
  • MPI_INIT: Takes a single argument, ierr, which is an integer to hold the error status (0 for success).
  • MPI_FINALIZE: Similarly takes an ierr argument for error handling.

[!NOTE]

MPI_Init

  • It initializes MPI and allows all processes to participate in the parallel computation.
  • This function must be the first MPI function called, as no MPI operations are allowed before the MPI environment is set up.

MPI_Finalize

  • It acts as a synchronization point, as all processes in the communicator must reach this function before the program can terminate.
  • It ensures that all ongoing MPI communications are completed and all resources allocated by the MPI library are freed.
  • No MPI function can be called after MPI_Finalize, making it the final MPI operation in the program.

MPI_Comm_rank

The MPI_Comm_rank function is a core MPI command that allows each process in a distributed system to determine its unique identifier, or “rank,” within a given communicator. This rank is an integer that typically starts at zero for the first process and increments by one for each additional process in the communicator. Knowing the rank of each process is essential for controlling how tasks are distributed and managed in parallel computation.

The signatures in C are:

int MPI_Comm_rank(MPI_Comm comm, int *rank);

The signature in Fortran are:

CALL MPI_COMM_RANK(comm, rank, ierr)
  • comm: The communicator for which the rank is being queried, usually MPI_COMM_WORLD, which represents all processes initiated by MPI_Init.
  • rank: An integer variable where the rank of the calling process will be stored.
  • ierr (Fortran only): An integer error code. Zero indicates success.

[!NOTE]

  • The rank values range from 0 to size-1, where size is the total number of processes in the communicator.
  • MPI_COMM_WORLD is the default communicator encompassing all processes. Custom communicators can be defined to group subsets of processes.

[!TIP]

To ensure a section of code is executed by only one process, use a conditional check on the rank, commonly rank == 0. For example, if you want a specific action (such as printing output or managing I/O) to be done by only one process, add a check for rank == 0. In this way, only the process with rank 0 —- often called the “master” or “root” process —- will execute that block, avoiding redundancy and reducing communication overhead.

In our example:

// Check if n is a positive number
   if (rank == 0) {
       if (n <= 0) {
           printf("Error: The number of trapezoids must be positive.\n");
           MPI_Abort(MPI_COMM_WORLD, 1);
           return 1;
       }
   }

MPI_Comm_size

The MPI_Comm_size function allows you to determine the total number of processes participating in a given communicator, typically MPI_COMM_WORLD. Knowing the number of processes (size) is essential for distributing tasks evenly and dynamically adapting workloads in parallel applications. This is especially useful in dividing up iterations or data among processes.

The signatures in C are:

MPI_Comm_size(MPI_Comm comm, int *size);

The signature in Fortran are:

CALL MPI_Comm_size(comm, size, ierr)
  • comm: The communicator for which the rank is being queried, usually MPI_COMM_WORLD, which represents all processes initiated by MPI_Init.
  • size: [(C only) A pointer to] an integer where the number of processes will be stored.
  • ierr (Fortran only): An integer error code. Zero indicates success.

In our example, we used size to divide the integration interval among the processes:

// Divide the interval [a, b] among processes
double h = (b - a) / size;  // Width of each sub-interval

MPI_Abort

MPI_Abort provides a way to terminate all processes in a communicator if a critical error occurs, making it a helpful tool for immediate shutdown in MPI programs. When called, MPI_Abort stops all processes within the specified communicator (usually MPI_COMM_WORLD) and returns an error code to indicate the reason for termination. This is particularly useful in situations where continuing execution would lead to incorrect or unpredictable behavior.

The signatures in C are:

MPI_Abort(MPI_Comm comm, int errorcode);

The signature in Fortran are:

CALL MPI_Abort(comm, errorcode, ierr)
  • comm: The communicator for which the rank is being queried, usually MPI_COMM_WORLD, which represents all processes initiated by MPI_Init.
  • errorcode: The integer error code to return, indicating the reason for aborting.
  • ierr (Fortran only): An integer error code. Zero indicates success.

[!NOTE] The difference between MPI_Abort and MPI_Finalize lies in how they terminate an MPI program and the context in which each should be used. MPI_Finalize is used to gracefully shut down an MPI program after all processes have completed their tasks and the program is ready to terminate normally. MPI_Abort is used to immediately terminate all processes within a communicator if a critical error or unexpected condition is detected that prevents the program from continuing safely. In particular, MPI_Abort provides an error code for debugging purposes, while MPI_Finalize does not.

MPI_Reduce

MPI_Reduce is a collective communication function that performs a reduction operation across all processes within a communicator. It combines data from each process and reduces it to a single result, which is stored in one specified process, typically the root process.

The signatures in C are:

int MPI_Reduce(const void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm);

The signature in Fortran are:

CALL MPI_Reduce(sendbuf, recvbuf, count, datatype, op, root, comm, ierror)
  • sendbuf: The starting address of the data buffer to be reduced from each process. Each process contributes its data stored here.
  • recvbuf: The starting address of the buffer where the root process stores the result. (Ignored by non-root processes).
  • count: The number of elements in the data buffer from each process.
  • datatype: The data type of elements in sendbuf and recvbuf (e.g., MPI_INT, MPI_FLOAT, MPI_DOUBLE).
  • op: The reduction operation to be applied, like MPI_SUM, MPI_MAX, MPI_MIN, or custom operations.
  • root: The rank of the process that will store the result.
  • comm: The communicator within which the operation is performed (usually MPI_COMM_WORLD).
  • ierr (Fortran only): An integer error code. Zero indicates success.

[!Warning] MPI_Reduce collects data from all processes, applies the reduction operation, and stores the result in the buffer of the designated root process. Other processes will not receive the result directly; only the root process gets the reduced value.

MPI_Allreduce

The MPI_Allreduce function in MPI is similar to MPI_Reduce but with one key difference: it performs a reduction operation across all processes and then shares the result with all processes, not just a designated root process. This makes MPI_Allreduce particularly useful when every process needs access to the result of the reduction.

The signatures in C are:

int MPI_Allreduce(const void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);

The signature in Fortran are:

CALL MPI_Allreduce(sendbuf, recvbuf, count, datatype, op, comm, ierror)
  • sendbuf: The starting address of the data buffer to be reduced from each process. Each process contributes its data stored here.
  • recvbuf: The starting address of the buffer where the root process stores the result. (Ignored by non-root processes).
  • count: The number of elements in the data buffer from each process.
  • datatype: The data type of elements in sendbuf and recvbuf (e.g., MPI_INT, MPI_FLOAT, MPI_DOUBLE).
  • op: The reduction operation to be applied, like MPI_SUM, MPI_MAX, MPI_MIN, or custom operations.
  • comm: The communicator within which the operation is performed (usually MPI_COMM_WORLD).
  • ierr (Fortran only): An integer error code. Zero indicates success.

MPI_Allreduce collects data from all processes, performs the specified reduction operation, and then distributes the result back to all participating processes. Each process will have the final reduced value in its recvbuf after the call.