Message-Passing Parallel Programming

Message-passing parallel programming coordinates processes on distributed-memory machines by explicit communication, the dominant model for large-scale high-performance computing.

Definition

In message-passing parallel programming, each process has its own private memory and processes cooperate solely by sending and receiving messages; the Message Passing Interface (MPI) is the standard library that provides point-to-point and collective communication primitives for this model.

Scope

This topic covers the message-passing paradigm for distributed-memory parallel computers: point-to-point send/receive operations and their blocking and non-blocking variants, collective operations (broadcast, scatter/gather, reduction, all-to-all), process groups and communicators, and the de facto MPI standard. It also covers structured cost models such as the Bulk Synchronous Parallel (BSP) model that guide algorithm design and performance reasoning.

Core questions

• How are computations partitioned across distributed-memory processes that share no address space?
• When should collective operations replace explicit point-to-point communication?
• How can the communication cost of a parallel algorithm be modeled and minimized?

Key theories

Point-to-point and collective communication

MPI structures parallel computation around explicit messages between processes and optimized collective patterns—broadcasts, reductions, and gathers—whose efficient implementations are central to scalable distributed-memory programs.

Bulk Synchronous Parallel model

The BSP model structures computation into supersteps of local computation, communication, and barrier synchronization, giving a clean cost model that bridges algorithm design and message-passing hardware.

Communication-aware algorithm design

Because communication often dominates cost on distributed-memory machines, parallel algorithms are designed and analyzed to minimize message volume and latency, balancing computation against communication.

Clinical relevance

MPI is the backbone of scientific simulation on supercomputers—climate modeling, computational fluid dynamics, molecular dynamics—and of any computation that must scale across the many distributed-memory nodes of a cluster.

History

Valiant's 1990 BSP model gave a bridging abstraction for parallel computation; the MPI Forum standardized message passing in 1994, and successive MPI versions plus reference texts by Gropp, Snir, Dongarra, and colleagues made it the lingua franca of high-performance computing.

Key figures

• William Gropp
• Jack Dongarra
• Marc Snir
• Leslie Valiant

Seminal works

• gropp2014
• valiant1990
• snir1998

Frequently asked questions

Why is message passing preferred for very large parallel machines?

Large parallel computers are physically distributed-memory: there is no single shared memory to scale. Message passing matches this hardware directly and forces the programmer to manage data locality explicitly, which is what enables scaling to thousands or millions of cores.