How does model parallelism differ from data parallelism?

Model parallelism and data parallelism are two different techniques used to train large machine learning and deep learning models efficiently across multiple devices such as GPUs or TPUs. The main difference lies in what is divided across the devices.

Data Parallelism

In data parallelism:

• Every GPU contains an identical copy of the entire neural network.
• The training dataset is divided into smaller mini-batches.
• Each GPU processes a different mini-batch independently.
• After computing gradients, all GPUs synchronize them before updating the model weights.

Example

Suppose you have:

• 4 GPUs
• A model with 100 million parameters
• Batch size = 256

The batch is divided as:

• GPU 1 → 64 images
• GPU 2 → 64 images
• GPU 3 → 64 images
• GPU 4 → 64 images

Each GPU performs forward and backward propagation independently, and the gradients are averaged before updating the model.

Advantages

• Simple to implement.
• Provides nearly linear speedup with more GPUs.
• Widely supported by frameworks like TensorFlow and PyTorch.

Disadvantages

• Every GPU must have enough memory to store the complete model.
• Gradient synchronization can become a communication bottleneck.

Model Parallelism

In model parallelism:

• The model itself is divided among multiple GPUs.
• Each GPU stores only part of the neural network.
• Input data flows sequentially through the different GPUs.

Example

Consider a neural network with eight layers.

Instead of storing all layers on one GPU:

• GPU 1 → Layers 1–2
• GPU 2 → Layers 3–4
• GPU 3 → Layers 5–6
• GPU 4 → Layers 7–8

The computation proceeds like this:

Input
  ↓
GPU 1 (Layers 1–2)
  ↓
GPU 2 (Layers 3–4)
  ↓
GPU 3 (Layers 5–6)
  ↓
GPU 4 (Layers 7–8)
  ↓
Output

Each GPU only performs computations for its assigned layers.

Advantages

• Enables training models that cannot fit into the memory of a single GPU.
• Essential for extremely large models such as modern large language models.

Disadvantages

• Requires communication of intermediate activations between devices.
• More complex to implement and optimize.
• Devices may be idle if the workload is not balanced.

Key Differences

Data Parallelism

• Replicates the entire model on every device.
• Splits the dataset.
• Synchronizes gradients.
• Primarily improves training throughput.

Model Parallelism

• Splits the model across devices.
• Processes the same input through different model partitions.
• Transfers activations between devices.
• Primarily addresses memory limitations for very large models.

When to Use Each

• Use data parallelism when the model fits comfortably on a single GPU but the dataset is large and you want faster training.
• Use model parallelism when the model is too large to fit into the memory of one GPU.

Many modern large-scale systems combine both approaches. For example, training large language models often uses hybrid parallelism, which may include data parallelism, model (tensor or pipeline) parallelism, and optimizer state sharding to efficiently scale training across hundreds or thousands of accelerators.