Linux Log Forwarding Using Rsyslog Daemon

This article provides an overview of how the Linux operating system handles remote log forwarding through the native rsyslog daemon. It covers the core architecture of log processing, the transport protocols used to transmit data across networks, the configuration syntax required to define remote targets, and the buffering mechanisms that ensure log reliability during network interruptions.

The Rsyslog Processing Pipeline

The rsyslog daemon operates as an event-driven processing pipeline divided into inputs, filtering engines, and outputs (actions). Log messages are first ingested from local system sources, such as /dev/log, the systemd-journald service via the imjournal module, or application files via imfile. Once ingested, rsyslog parses the message properties (facility, severity, hostname, timestamp, and message body). The engine evaluates these properties against user-defined rulesets to determine whether a log should be written to local storage, discarded, or forwarded across the network.

Transport Protocols

When forwarding logs to a centralized collector or SIEM, rsyslog supports three primary transport mechanisms:

• UDP (User Datagram Protocol): Denoted by a single @ symbol preceding the destination host (e.g., @192.168.1.50:514). UDP offers low overhead and minimal latency, but lacks delivery confirmation, packet retransmission, and flow control. Messages may be dropped during network congestion.
• TCP (Transmission Control Protocol): Denoted by two @ symbols (e.g., @@192.168.1.50:514). TCP provides connection-oriented transport, ensuring ordered delivery and retransmission of lost packets.
• RELP (Reliable Event Logging Protocol): Handled via the omrelp module. RELP addresses the "loss at close" scenario inherent to standard TCP, where an abrupt connection drop can cause unacknowledged packets to be lost in transit. RELP provides application-level acknowledgments to guarantee zero-loss log delivery.

Configuration Methods

Log forwarding is configured in /etc/rsyslog.conf or modular files within /etc/rsyslog.d/. Modern deployments utilize RainerScript syntax rather than legacy directives for improved readability and conditional logic.

A basic TCP forwarding rule using RainerScript looks like this:

action(type="omfwd"
       target="logs.example.com"
       port="514"
       protocol="tcp"
       template="RSYSLOG_SyslogProtocol23Format")

Filters can restrict forwarding to specific applications or severity levels. For example, to forward only authentication logs with an error level or higher:

if ($programname == 'sshd' and $syslogseverity <= 3) then {
    action(type="omfwd" target="siem.example.com" port="514" protocol="tcp")
}

Queue Management and Reliability

To prevent system lockups and data loss when a remote destination is unreachable, rsyslog uses action queues. By default, actions operate in direct (in-memory, synchronous) mode. When forwarding over a network, administrators can configure a disk-assisted memory queue:

action(type="omfwd"
       target="logs.example.com"
       port="514"
       protocol="tcp"
       queue.filename="fwd_queue"
       queue.type="LinkedList"
       queue.size="100000"
       queue.maxdiskspace="1g"
       queue.saveonshutdown="on"
       action.resumeRetryCount="-1")

In this mode, messages are held in a memory buffer (LinkedList). If the remote collector goes offline, messages spool to disk (queue.filename), and rsyslog continuously retries the connection (action.resumeRetryCount="-1"). Once connectivity is restored, the queue flushes the stored records in order without dropping active events.

Encryption via TLS

By using the lmnsd_gtls or lmnsd_ossl network stream drivers, rsyslog can encrypt log transmissions via Transport Layer Security (TLS). This prevents eavesdropping and tampering of log data over untrusted networks. Mutual TLS (mTLS) authentication can also be enforced, requiring both the forwarder and the receiver to present valid X.509 certificates before establishing the logging pipeline.