Parallel Programming Models
Parallel programming models are the abstractions through which programmers express and coordinate computation across many processors to gain speed.
Definition
A parallel programming model is an abstraction of a parallel computer that defines how computation is decomposed into concurrent tasks, how those tasks communicate and synchronize, and how data is shared or distributed among processors, enabling a problem to be solved faster by using multiple processing units.
Scope
This area covers the principal parallel programming models—shared-memory and data parallelism (threads, OpenMP), message passing (MPI), and accelerator/GPU programming (CUDA, OpenCL)—together with Flynn's taxonomy of parallel architectures, the design of parallel algorithms, and the performance laws (Amdahl's and Gustafson's) and metrics (speedup, efficiency, scalability) that bound and measure their benefit. It is the parallel-computing complement to the distributed-systems side of the subfield.
Sub-topics
Core questions
- How should a computation be decomposed into tasks and mapped onto processors?
- Which programming model—shared memory, message passing, or accelerator—fits a given architecture and problem?
- What limits the speedup achievable from parallelism, and how is scalability measured?
Key theories
- Flynn's taxonomy
- Parallel architectures are classified by their instruction and data streams into SISD, SIMD, MISD, and MIMD, a framework that still organizes thinking about vector, GPU, and multicore machines.
- Amdahl's and Gustafson's laws
- Amdahl's law bounds speedup by the program's serial fraction for a fixed problem size, while Gustafson's reframing notes that larger problems can keep more processors usefully busy, together framing the limits and promise of parallelism.
- Parallel design patterns
- Recurring strategies—task and data decomposition, pipelines, geometric decomposition, and master-worker—provide a structured vocabulary for designing parallel programs across models.
Clinical relevance
Parallel programming models are the basis of high-performance and scientific computing, machine-learning training on GPUs, and the everyday exploitation of multicore processors; the choice of model determines achievable performance and portability across hardware.
History
Flynn's 1972 taxonomy and Amdahl's 1967 speedup argument framed parallel computing from the start; standardized models emerged with MPI and OpenMP in the 1990s and GPU computing in the 2000s, while textbooks such as Grama and colleagues' codified the design and analysis of parallel algorithms.
Debates
- Shared memory versus message passing as the default model
- Shared-memory models are easier to program but scale less readily and risk subtle data races, whereas message passing scales to large clusters at the cost of explicit communication; hybrid models combine both, and the right default remains workload- and architecture-dependent.
Key figures
- Gene Amdahl
- Michael Flynn
- Vipin Kumar
- John Gustafson
Related topics
Seminal works
- grama2003
- amdahl1967
- flynn1972
Frequently asked questions
- Why can't adding more processors make any program arbitrarily fast?
- Amdahl's law shows that the serial portion of a computation—work that cannot be parallelized—bounds the total speedup. If even a small fraction is inherently sequential, it dominates as processor counts grow, capping the achievable acceleration.