PyZMQ Pub-Sub and Request-Reply in Python

This article provides a practical guide to implementing two foundational messaging architectures—Publish-Subscribe (Pub-Sub) and Request-Reply—using pyzmq, the Python bindings for ZeroMQ. You will learn the mechanics behind each pattern, the specific socket types involved, and how to implement working clients and servers for distributed messaging without needing a centralized message broker.

Understanding ZeroMQ and pyzmq

ZeroMQ is a high-performance asynchronous messaging library that provides message-oriented sockets across various transports (such as TCP, IPC, and in-process). Unlike traditional broker-based systems like RabbitMQ or Kafka, pyzmq applications are typically brokerless; sockets connect directly to one another, drastically lowering latency and complexity.

The Request-Reply Pattern

The Request-Reply pattern is used for synchronous remote procedure calls (RPC) and client-server communication. It enforces a strict alternating lockstep: a client must send a request before receiving a response, and a server must receive a request before sending a reply.

Socket Types

• zmq.REQ: Used by the client to send requests and receive responses.
• zmq.REP: Used by the server to receive requests and send responses.

Implementation

Server (rep_server.py):

import zmq

context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:5555")

while True:
    message = socket.recv_string()
    print(f"Received request: {message}")
    socket.send_string(f"Echo: {message}")

Client (req_client.py):

import zmq

context = zmq.Context()
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:5555")

socket.send_string("Hello")
response = socket.recv_string()
print(f"Received reply: {response}")

If the alternating send-receive order is violated (for instance, calling send_string() twice consecutively on a REQ socket), pyzmq raises a zmq.ZMQError.

The Publish-Subscribe Pattern

The Publish-Subscribe pattern facilitates one-to-many asynchronous data distribution. Publishers emit messages without knowing who the receivers are, and subscribers receive only the messages they have explicitly registered interest in via topic filtering.

Socket Types

• zmq.PUB: Used by the publisher to distribute messages.
• zmq.SUB: Used by subscribers to listen for messages matching specific prefixes.

Topic Filtering

In pyzmq, filtering occurs on the subscriber side. By default, a newly created zmq.SUB socket ignores all incoming messages. A subscriber must subscribe to a topic prefix using setsockopt_string(zmq.SUBSCRIBE, ...). An empty string ("") subscribes to all incoming messages.

Implementation

Publisher (pub_server.py):

import time
import zmq

context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.bind("tcp://*:5556")

# Brief pause to allow subscribers to connect
time.sleep(1)

topics = ["SPORTS", "WEATHER", "TECH"]

for i in range(10):
    topic = topics[i % len(topics)]
    payload = f"Update {i}"
    socket.send_string(f"{topic} {payload}")
    time.sleep(0.5)

Subscriber (sub_client.py):

import zmq

context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect("tcp://localhost:5556")

# Subscribe specifically to the "TECH" topic
socket.setsockopt_string(zmq.SUBSCRIBE, "TECH")

while True:
    message = socket.recv_string()
    print(f"Subscriber received: {message}")

Key Differences and Considerations

• State Enforcement: REQ/REP sockets operate as state machines enforcing sequential request-response cycles. PUB/SUB sockets are unidirectional; PUB sockets can only send, while SUB sockets can only receive.
• Late Joiner Problem: In Pub-Sub, subscribers that connect after a publisher has begun transmitting will miss messages sent prior to connection establishment.
• Scalability: Both patterns support multiple endpoints. A single REP socket can handle requests from thousands of REQ clients, and a single PUB socket can fan out messages to thousands of SUB sockets without architectural changes.