Linux Watchdog Timer: Automatic Reboot on System Hang

In the Linux operating system, high availability and fault recovery are managed through watchdog timers that automatically reboot the system during a complete freeze or kernel lockup. This mechanism relies on a continuous heartbeat system: an active timer counts down toward zero, requiring the operating system or a dedicated user-space daemon to periodically "pet" or reset the counter. If the operating system hangs, becomes deadlocked, or crashes, the counter reaches zero and triggers a hardware or software reset, returning the machine to an operational state without manual intervention.

The Watchdog Mechanism

The core operation of a Linux watchdog relies on a simple countdown timer. Under normal operating conditions, a process regularly writes to the watchdog device to reset the expiration timer—an action commonly referred to as "pinging," "kicking," or "petting" the dog.

When a critical failure occurs, such as a kernel panic, out-of-memory deadlock, or hard CPU lockup, the running processes are halted. Consequently, the reset signal is never delivered. Once the countdown expires (typically after a user-defined threshold, such as 30 or 60 seconds), the watchdog hardware asserts the system's reset line, simulating a physical push of the power or reset button.

Hardware vs. Software Watchdogs

Linux supports both hardware-based and software-emulated watchdog solutions:

• Hardware Watchdogs: These utilize dedicated microcontrollers or integrated chipsets (such as Intel TCO, AMD SP5100, Raspberry Pi BCM2835 watchdog, or IPMI baseboard management controllers). Because the timer runs entirely independent of the CPU and main memory, a hardware watchdog will reliably reboot the machine even if the CPU completely locks up, overheats, or experiences a power anomaly.
• Software Watchdog (softdog): Linux provides a kernel module named softdog. This emulates a watchdog device using kernel timers. While useful for systems lacking dedicated hardware, softdog has a significant limitation: if the kernel itself encounters a hard lockup that disables interrupts or freezes the timer subsystem, the software watchdog will also freeze and fail to reboot the machine.

The Linux Kernel Watchdog Subsystem

The Linux kernel standardizes watchdog management through its core driver framework located in drivers/watchdog/. The system exposes these devices to user space through special character devices, primarily /dev/watchdog (legacy single-open interface) and /dev/watchdog0 (supporting modern sysfs attributes).

Key driver behaviors include:

• File Operations: Opening /dev/watchdog starts the timer. Writing any data to this file resets the countdown.
• Magic Close Feature: To prevent accidental reboots when a monitoring process stops intentionally (such as during a clean shutdown), the driver implements a "Magic Close" feature. The controlling process must write the character 'V' to /dev/watchdog before closing the file descriptor. If the file is closed without this character, the kernel assumes the daemon crashed and lets the timer expire to force a reboot.
• Pre-timeout Interrupts: Many hardware drivers support a pre-timeout interrupt. This generates a Non-Maskable Interrupt (NMI) shortly before the system resets, allowing the kernel to capture diagnostic data, print stack traces, or trigger a kdump for post-mortem analysis.

User-Space Integration

The kernel provides the interface, but user-space software typically manages the heartbeat logic:

• Systemd Watchdog: Modern Linux distributions utilizing systemd can natively manage the watchdog. By setting RuntimeWatchdogSec= in /etc/systemd/system.conf, systemd continuously pings /dev/watchdog. If the init system hangs, the hardware resets the host. Systemd also offers RebootWatchdogSec= to protect against hangs during system shutdown sequences.
• The watchdog Daemon: Traditional environments use the standalone watchdog daemon. This tool does not merely ping the timer at fixed intervals; it evaluates system health metrics before each ping. It can monitor memory exhaustion, process table saturation, interface availability, and critical process states. If any specified health check fails, the daemon deliberately stops pinging the device, triggering a reboot.

NMI Watchdogs for Lockup Detection

For multi-core systems, Linux includes an internal NMI (Non-Maskable Interrupt) watchdog managed by the kernel. It couples high-resolution performance counters with NMIs to detect when a single CPU core is stuck inside an interrupt handler or spinlock for more than a set duration (typically 10 to 20 seconds). When combined with the sysctl setting kernel.panic_on_io_nmi = 1 or kernel.panic = 10, the kernel will detect the stuck core, induce a panic, and automatically reboot the machine after 10 seconds.