Run Background Workers with BullMQ and Node.js

Offloading resource-intensive or time-consuming tasks to background workers is essential for maintaining high performance in Node.js applications. This article provides a straightforward guide on how to set up and run a background worker in Node.js using BullMQ, a robust, Redis-backed queue system, ensuring your main application remains fast and responsive.

Prerequisites

To implement a BullMQ background worker, you need: * Node.js installed on your system. * A running Redis server (BullMQ uses Redis to manage jobs and states).

Step 1: Install BullMQ

First, initialize a Node.js project if you haven’t already, and install the bullmq package.

npm init -y
npm install bullmq

Step 2: Configure the Redis Connection

BullMQ requires a connection to Redis to manage queue data. Define your Redis connection configuration in your application code:

const redisConnection = {
  host: '127.0.0.1',
  port: 6379,
};

Step 3: Create the Background Worker

A worker listens to a specific queue, fetches waiting jobs, and executes them in the background. Create a file named worker.js and add the following code:

import { Worker } from 'bullmq';

// Name of the queue this worker will listen to
const queueName = 'email-queue';

// Define the worker and the processing logic
const worker = new Worker(
  queueName,
  async (job) => {
    console.log(`Processing job ${job.id} of type ${job.name}...`);
    
    // Simulate a time-consuming background task (e.g., sending an email)
    await new Promise((resolve) => setTimeout(resolve, 3000));
    
    console.log(`Email successfully sent to: ${job.data.email}`);
    return { status: 'completed', recipient: job.data.email };
  },
  {
    connection: redisConnection,
    concurrency: 5 // Process up to 5 jobs simultaneously
  }
);

// Event listeners for monitoring
worker.on('completed', (job, result) => {
  console.log(`Job ${job.id} completed successfully. Result:`, result);
});

worker.on('failed', (job, err) => {
  console.log(`Job ${job.id} failed with error: ${err.message}`);
});

console.log('Worker is running and waiting for jobs...');

Step 4: Add Jobs to the Queue (Producer)

To test the worker, you need to push jobs into the queue. Create a file named producer.js to add a job:

import { Queue } from 'bullmq';

const emailQueue = new Queue('email-queue', {
  connection: redisConnection,
});

async function addEmailJob() {
  const job = await emailQueue.add('send-welcome-email', {
    email: 'user@example.com',
    userId: 101,
  });
  
  console.log(`Job added to queue with ID: ${job.id}`);
}

addEmailJob();

Running the System

1. Ensure your Redis server is running:

   redis-server

2. Start the background worker process in your terminal:

   node worker.js

3. In a separate terminal, trigger the producer to queue a task:

   node producer.js

The worker terminal will immediately detect the new job, process it in the background, and output the success logs without blocking your main application loop.