Linux SAR: Collect, Report, and Save System Activity

The System Activity Report (sar) utility is one of the most comprehensive performance monitoring tools available in the Linux operating system. Part of the sysstat package, sar enables system administrators to collect performance metrics automatically at set intervals, store the resulting data in compact binary files, and generate detailed historical or real-time reports covering CPU, memory, I/O, paging, and network subsystems.

How sar Collects System Information

The collection process does not originate directly from the interactive sar command itself. Instead, it relies on an underlying backend utility named sadc (System Activity Data Collector).

1. Kernel Interfaces: At scheduled intervals, sadc queries the Linux kernel interfaces, specifically reading virtual pseudo-filesystems such as /proc (e.g., /proc/stat, /proc/meminfo, /proc/diskstats, /proc/net/dev) and /sys.
2. Scheduling Daemons: Modern Linux systems automate this collection using either systemd timers (sysstat-collect.timer) or standard cron jobs (/etc/cron.d/sysstat). Typically, a collection script runs every 10 minutes (or at custom defined frequencies) to sample current operating metrics.
3. Internal Timers: For immediate data collection, executing sar directly with an interval and count parameter (such as sar 2 5) prompts the utility to take immediate snapshots directly from /proc, calculating the delta between intervals to determine real-time utilization rates.

How sar Saves System Data

To minimize storage overhead and processing impact on production servers, data is saved in a raw, optimized binary format rather than plain text.

• Storage Location: Collected data is written directly to the /var/log/sysstat/ or /var/log/sa/ directory (depending on the distribution).
• Naming Conventions:
  • Files named saDD (where DD represents the day of the month) store raw binary data captured by sadc.
  • Files named sarDD represent pre-generated daily text reports produced during log rotation.
• Log Rotation: The sysstat service maintains a configurable retention period (defined in /etc/sysstat/sysstat). Old log files are automatically overwritten or purged once they exceed the retention window, preventing uncontrolled disk usage.

How sar Reports System Metrics

The sar front-end command reads the generated binary data and translates it into human-readable tables, allowing users to query either live data or past performance metrics.

• Historical Reporting: By reading daily binary files using the -f flag, administrators can inspect historical conditions down to the minute:

  sar -u -f /var/log/sysstat/sa15

  This command extracts and prints CPU metrics specifically for the 15th day of the current month.
• Subsystem Filtering: Specific flags isolate distinct resources:
  • -u: CPU utilization (user, system, iowait, idle).
  • -r: Memory and swap usage.
  • -d: Disk block device read/write rates.
  • -n DEV: Network interface throughput, packet transfer, and error rates.
  • -q: System load averages and process run-queue lengths.
• Time Slicing: Users can restrict reports to specific incidents using -s (start time) and -e (end time) switches formatted as hh:mm:ss.
• Structured Exporting: While sar primarily outputs plain text tables, companion utilities like sadf can read the same binary log files to output data in structured formats such as CSV, XML, JSON, or SVG for external graphing and analysis pipelines.