How Does Instruction Pipelining Increase CPU Throughput?

Dividing instruction execution into fetch, decode, execute, memory, and writeback stages increases throughput through a technique called instruction pipelining. Instead of waiting for one instruction to finish all its steps before starting the next, a processor overlaps the execution of multiple instructions simultaneously. Each stage handles a different instruction during every clock cycle, allowing the CPU to complete one instruction per cycle once the pipeline is full.

Understanding the Five-Stage Pipeline

In a classic five-stage RISC pipeline, every instruction passes through five distinct hardware units:

  • Fetch (IF): Retrieves the next instruction from instruction memory using the program counter.
  • Decode (ID): Interprets the instruction opcode and reads the required source registers.
  • Execute (EX): Performs arithmetic or logic operations in the ALU, or calculates memory addresses.
  • Memory (MEM): Reads from or writes to data memory if the instruction is a load or store.
  • Writeback (WB): Writes the final result back into the destination register.

Overlapping Execution vs. Sequential Execution

Without pipelining, a processor must complete all five stages for a single instruction before fetching the next one. If each stage takes \(1\) clock cycle, a single-cycle or non-pipelined multi-cycle design completes at most one instruction every \(5\) cycles, giving a throughput of \(0.2\) instructions per cycle (IPC).

With pipelining, each stage is separated by pipeline registers that hold intermediate results between clock ticks. As soon as Instruction 1 moves from Fetch to Decode, Instruction 2 enters Fetch. By the fifth clock cycle, all five stages are active at the same time, each working on a different instruction.

The Math Behind the Speedup

In an ideal \(k\)-stage pipeline where each stage takes time \(t\), the clock period is determined by the longest stage rather than the sum of all stages. For \(n\) instructions:

\[\text{Non-pipelined time} = n \cdot k \cdot t\]

\[\text{Pipelined time} = (k + n - 1) \cdot t\]

The theoretical speedup \(S\) as the number of instructions \(n\) grows large is:

\[S = \lim_{n \to \infty} \frac{n \cdot k \cdot t}{(k + n - 1) \cdot t} = k\]

For a five-stage pipeline (\(k = 5\)), ideal throughput increases by a factor of \(5\), approaching \(1\) instruction completed per clock cycle (\(\text{IPC} \approx 1\)) rather than \(1\) every \(5\) cycles. Note that pipelining does not reduce the latency of an individual instruction—it still takes \(5\) cycles from start to finish—but it dramatically increases the rate at which instructions finish.

Practical Limits to Pipeline Throughput

Real-world processors rarely sustain the ideal speedup of \(k\) due to pipeline hazards:

  • Structural hazards: Occur when two instructions need the same hardware resource in the same cycle.
  • Data hazards: Happen when an instruction depends on the result of a previous instruction that has not yet reached writeback.
  • Control hazards: Arise from branch and jump instructions that change the program counter, potentially invalidating instructions already fetched into the pipeline.

Techniques such as data forwarding, branch prediction, and out-of-order execution help minimize these stalls and keep the five stages as busy as possible.