Event-Driven Microservices with Node.js and Kafka

This article provides a practical guide on how to implement an event-driven microservice architecture using Node.js and Apache Kafka. You will learn how to set up a local Kafka broker, write a Node.js producer service to publish events, and create a Node.js consumer service to process those events in real-time. By the end of this guide, you will have a working prototype of decoupled, scalable microservices communicating asynchronously.

Understanding the Architecture

In an event-driven microservices architecture, services communicate by producing and consuming events rather than making direct HTTP requests.

Step 1: Setting Up Apache Kafka

The fastest way to run Apache Kafka locally is using Docker Compose. Create a file named docker-compose.yml with the following configuration to spin up Zookeeper and Kafka:

version: '3'
services:
  zookeeper:
    image: wurstmeister/zookeeper:3.4.6
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka:2.13-2.8.1
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Run the following command in your terminal to start the services:

docker-compose up -d

Step 2: Initializing the Node.js Projects

Create a directory for your project and initialize two separate Node.js services: a producer (e.g., an Order Service) and a consumer (e.g., a Notification Service).

mkdir kafka-microservices && cd kafka-microservices
mkdir producer consumer

In both the producer and consumer directories, initialize a Node.js project and install kafkajs, the popular, modern Kafka client for Node.js:

npm init -y
npm install kafkajs

Step 3: Implementing the Node.js Producer

The producer service will connect to the Kafka broker and send message payloads to a specific topic. Save the following code as index.js inside the producer folder.

const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'order-service',
  brokers: ['localhost:9092']
});

const producer = kafka.producer();

const sendOrderEvent = async () => {
  await producer.connect();
  console.log('Producer connected successfully');

  const orderData = {
    orderId: 'ORD-99281',
    customerEmail: 'user@example.com',
    amount: 129.99,
    status: 'CREATED'
  };

  await producer.send({
    topic: 'order-events',
    messages: [
      { value: JSON.stringify(orderData) }
    ],
  });

  console.log('Event published: Order Created');
  await producer.disconnect();
};

sendOrderEvent().catch(console.error);

Step 4: Implementing the Node.js Consumer

The consumer service runs continuously, listening for new messages on the order-events topic. Save the following code as index.js inside the consumer folder.

const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'notification-service',
  brokers: ['localhost:9092']
});

const consumer = kafka.consumer({ groupId: 'notification-group' });

const runConsumer = async () => {
  await consumer.connect();
  console.log('Consumer connected successfully');

  await consumer.subscribe({ topic: 'order-events', fromBeginning: true });

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      const order = JSON.parse(message.value.toString());
      console.log(`Received order event for Order ID: ${order.orderId}`);
      console.log(`Sending confirmation email to ${order.customerEmail}...`);
      // Add notification sending logic here
    },
  });
};

runConsumer().catch(console.error);

Step 5: Testing the Ecosystem

  1. Start your consumer microservice first so it is ready to listen for incoming events:

    node consumer/index.js

  2. In a separate terminal window, run your producer microservice to publish a new event:

    node producer/index.js

Upon running the producer, the console output of the consumer will instantly update, showing that it received the payload and initiated the notification sequence.

Scaling and Best Practices

To scale this architecture in a production environment: