Linux Policy Routing Using Multiple Routing Tables

Linux manages advanced packet forwarding through Policy-Based Routing (PBR), a mechanism that determines packet paths based on custom attributes rather than relying solely on the destination IP address. By combining the Routing Policy Database (RPDB) with multiple independent routing tables, the Linux kernel allows administrators to steer traffic according to source address, incoming interface, firewall marks, or protocol type. This article details the underlying architecture of Linux policy routing, how the kernel evaluates routing rules, and how multiple tables are configured and processed.

The Architecture: RPDB and Multiple Routing Tables

Traditional IP routing is destination-based: the kernel consults a single routing table and matches the packet's destination against subnet prefixes. Linux extends this model using the Routing Policy Database (RPDB). The RPDB acts as a selector that sits in front of multiple routing tables, dictating which table the kernel must consult for any given packet.

Linux supports up to 2^32 routing tables (historically 255), identified by a numeric ID between 0 and 4,294,967,295. By default, the system initializes with predefined tables:

local (ID 255): Contains routes for local and broadcast addresses. The kernel automatically maintains this table, and it has the highest precedence.
main (ID 254): The standard routing table used for traditional destination-based routes (e.g., standard gateways and interface subnets).
default (ID 253): Reserved for post-processing and default routes, though rarely used by modern tooling.

Custom tables are assigned numeric identifiers or friendly aliases mapped within /etc/iproute2/rt_tables.

The Packet Evaluation Process

When a network packet enters the network stack—either received from a network interface or generated locally—the kernel follows a deterministic pipeline to select its path:

Rule Traversal: The kernel examines the RPDB sequentially, ordering rules from lowest priority number (highest precedence) to highest priority number.
Selector Matching: Each rule specifies selectors (criteria) such as source IP prefix, destination IP prefix, Type of Service (TOS), firewall mark (fwmark), or input interface (iif).
Table Lookup: When a packet matches a rule's selector, the evaluation jumps to the routing table designated by that rule.
Route Match or Fall-Through:
- If the designated table contains a route that matches the packet's destination, the kernel uses that route and stops processing.
- If no matching route is found in that specific table, the kernel returns to the RPDB and resumes evaluation at the next priority rule.
Action Execution: In addition to standard lookups, rules can specify actions such as unreachable, prohibit, or blackhole to drop matching packets immediately.

Defining Routing Tables

Custom routing tables are typically declared in /etc/iproute2/rt_tables to give numeric IDs human-readable names. For example:

200 isp2

Once defined, routes specific to that table are populated independently of the standard main table using the ip route utility:

# Add a default gateway specifically to table 'isp2'
ip route add default via 192.168.2.1 dev eth1 table isp2

Routes in this table remain isolated and will not affect regular system traffic until linked to an RPDB rule.

Steering Traffic with `ip rule`

The ip rule command configures the RPDB selectors that direct traffic to specific tables.

Source-Based Routing: Routes packets originating from a specific IP address or subnet through a separate table:
```
ip rule add from 192.168.2.100/32 table isp2 priority 1000
```
Interface-Based Routing: Routes packets entering a specific interface via a distinct gateway:
```
ip rule add iif eth1 table isp2 priority 1010
```

Mark-Based Routing (fwmark): Leverages iptables or nftables to mark packets based on Layer 4 attributes (such as port numbers or protocols) before routing evaluation:

# Mark outgoing HTTPS packets with mark 1
iptables -t mangle -A OUTPUT -p tcp --dport 443 -j MARK --set-mark 1

# Route marked packets using table 'isp2'
ip rule add fwmark 1 table isp2 priority 1020

Cache and State Management

Linux maintains a routing cache to speed up lookups for active connections. When modifications are made to the RPDB or routing tables via ip rule or ip route, the changes take effect immediately, but existing path calculations may persist until the cache is refreshed. Running ip route flush cache invalidates stale cache entries, forcing the kernel to re-evaluate active flows against the updated rule hierarchy.