Read and Write Large Files with Node fs.promises

Working with large files in Node.js can easily overwhelm system memory if not handled correctly. This article explains how to safely read and write massive files using the Node.js fs.promises API, exploring why standard buffer-based methods fail and how to implement memory-efficient streaming and chunk-based processing alternatives.

The Problem with Standard fs.promises Methods

Methods like fs.promises.readFile() and fs.promises.writeFile() are convenient but dangerous for massive files. They load the entire file into memory (V8 heap) all at once. If you try to read a 4 GB file on a system with a 2 GB Node.js default heap limit, your application will crash with an “Out of Memory” error.

To process large files safely, you must handle them in smaller, manageable chunks. The fs.promises API provides FileHandle objects that allow you to read and write files incrementally.

Safe Reading with FileHandles and Async Iterators

Instead of loading the entire file into memory, you can open the file using fs.promises.open() and read it sequentially using an async iterator. This ensures that only a small chunk of the file resides in memory at any given time.

import { open } from 'fs/promises';

async function readLargeFile(filePath) {
  let fileHandle;
  try {
    // Open the file in read-only mode
    fileHandle = await open(filePath, 'r');

    // Create a read stream from the file handle
    const stream = fileHandle.createReadStream();

    for await (const chunk of stream) {
      // Process each chunk (Buffer) here safely
      console.log(`Received chunk of size: ${chunk.length} bytes`);
    }
  } catch (error) {
    console.error('Error reading file:', error);
  } finally {
    // Always close the file handle to prevent memory leaks
    if (fileHandle) {
      await fileHandle.close();
    }
  }
}

Safe Writing with Stream Pipelines

When writing massive amounts of data, writing everything at once will also exhaust your memory. Using stream/promises alongside fs.promises allows you to pipe data safely. Node’s pipeline automatically manages “backpressure,” preventing data from buffering in memory if the disk write speed cannot keep up with the data source.

import { open } from 'fs/promises';
import { pipeline } from 'stream/promises';
import { Readable } from 'stream';

async function writeLargeFile(filePath, dataSourceIterator) {
  let fileHandle;
  try {
    // Open the file in write mode
    fileHandle = await open(filePath, 'w');

    // Create a readable stream from your data source generator
    const readStream = Readable.from(dataSourceIterator);
    const writeStream = fileHandle.createWriteStream();

    // Safely pipe the data to the destination file
    await pipeline(readStream, writeStream);
    console.log('File write completed successfully.');
  } catch (error) {
    console.error('Error writing file:', error);
  } finally {
    if (fileHandle) {
      await fileHandle.close();
    }
  }
}

// Example data generator yielding chunks of data
async function* dataGenerator() {
  for (let i = 0; i < 1000000; i++) {
    yield `Line ${i}: This is some data to write to our massive file.\n`;
  }
}

Processing and Transforming Large Files

You can combine reading, transforming, and writing into a single memory-efficient pipeline. This approach is ideal for tasks like parsing massive CSV logs, modifying them, and writing them to a new location.

import { open } from 'fs/promises';
import { pipeline } from 'stream/promises';

async function transformLargeFile(sourcePath, destinationPath) {
  let sourceHandle;
  let destHandle;

  try {
    sourceHandle = await open(sourcePath, 'r');
    destHandle = await open(destinationPath, 'w');

    const readStream = sourceHandle.createReadStream();
    const writeStream = destHandle.createWriteStream();

    await pipeline(
      readStream,
      async function* (source) {
        for await (const chunk of source) {
          // Convert chunk to string, uppercase it, and yield
          const transformed = chunk.toString().toUpperCase();
          yield transformed;
        }
      },
      writeStream
    );

    console.log('File transformation complete.');
  } catch (error) {
    console.error('Transformation failed:', error);
  } finally {
    if (sourceHandle) await sourceHandle.close();
    if (destHandle) await destHandle.close();
  }
}