How Telegraf Collects System Metrics in Linux

This article provides an overview of how the Linux operating system implements and executes the Telegraf agent to monitor core system performance. It explores Telegraf's operational lifecycle as a system daemon, details how its input plugins query Linux kernel interfaces and virtual file systems—such as /proc and /sys—and explains how collected metrics are structured, buffered, and transmitted to storage backends.

Daemon Architecture and Execution

Telegraf is a lightweight, plugin-driven agent written in Go, compiled as a single static binary. On Linux distributions, it is typically installed via package managers (apt, yum, or dnf) and registered as a background service managed by systemd.

Upon boot or service start, systemd invokes Telegraf under a dedicated, unprivileged service account (usually telegraf). The service configuration, found at /lib/systemd/system/telegraf.service, handles automated restarts, resource limits (cgroups), and process isolation. Because Telegraf operates in user space without requiring root privileges for standard operations, it accesses system health metrics exclusively through standardized Linux kernel interfaces and POSIX APIs.

Metric Gathering via Pseudo-Filesystems

Linux exposes kernel state and hardware statistics to user space through pseudo-filesystems generated dynamically in RAM. Telegraf uses Go libraries (such as gopsutil) to read and parse these text-based files at defined intervals.

• CPU Metrics (inputs.cpu): Telegraf parses /proc/stat to track the aggregate amount of time the CPU spends in various states (user, system, idle, iowait, irq, softirq, steal). By computing the delta between consecutive polling cycles, it derives real-time usage percentages.
• Memory Metrics (inputs.mem): Telegraf reads /proc/meminfo to gather values such as MemTotal, MemFree, MemAvailable, Buffers, and Cached. This provides an accurate breakdown of memory allocation without invoking heavy kernel system calls.
• Disk I/O Metrics (inputs.diskio): Input plugins read /proc/diskstats and files under /sys/block/ to determine read/write operations completed, sectors read/written, and time spent doing I/O.
• Filesystem Metrics (inputs.disk): Telegraf issues the statvfs system call via the C standard library/POSIX interface across all mounted filesystems listed in /proc/mounts to calculate total, free, and used disk space.
• Network Metrics (inputs.net): Network interface traffic (bytes sent/received, packets, errors, drops) is parsed directly from /proc/net/dev.
• System Load (inputs.system): Average system load over 1, 5, and 15 minutes is collected from /proc/loadavg, while system uptime is derived from /proc/uptime.

Configuration and Collection Lifecycle

Telegraf is controlled via a TOML configuration file located at /etc/telegraf/telegraf.conf, along with drop-in configuration snippets placed in /etc/telegraf/telegraf.d/.

The execution cycle operates on a synchronous polling loop governed by internal timers:

1. Polling (Interval): At every configured interval (e.g., interval = "10s"), Telegraf invokes the Gather() method for each enabled input plugin concurrently.
2. Normalization: The gathered metrics are converted into an internal data model consisting of a measurement name, tags (indexed key-value pairs like host or device), fields (unindexed values like usage_idle = 95.2), and an accurate UNIX timestamp.
3. Aggregation and Filtering: Optional processor and aggregator plugins modify or downsample metrics directly in memory according to defined rules.

Buffering and Ingestion Pipeline

To minimize I/O overhead on the host Linux system, Telegraf does not transmit metrics immediately upon retrieval. Instead, metrics are placed into an in-memory ring buffer (metric_buffer_limit).

A separate flushing routine runs based on the flush_interval directive (e.g., flush_interval = "10s"). During each flush cycle, Telegraf packages batches of metrics into the format required by the configured output plugins (such as InfluxDB Line Protocol, Prometheus, Kafka, or OpenTelemetry) and writes them over network sockets via TCP, UDP, or HTTP. If a downstream network failure occurs, the buffer retains metrics locally up to its configured ceiling, preventing data loss during temporary network partitions.