fs.readFile vs fs.createReadStream in Node.js

When working with the file system in Node.js, choosing the right method to read files is crucial for application performance and scalability. This article explains the key differences between fs.readFile and fs.createReadStream, detailing how they handle memory, process data, and when you should use each approach in your applications.

How They Work

The fundamental difference between these two methods lies in how they load file data into your computer’s memory.

fs.readFile (Buffer-based)

fs.readFile reads the entire contents of a file into memory asynchronously. Node.js allocates a single buffer large enough to hold the whole file. Once the entire file is successfully read and loaded into memory, the callback function is triggered, or the promise resolves with the complete file contents.

const fs = require('fs');

fs.readFile('large-file.txt', 'utf8', (err, data) => {
  if (err) throw err;
  console.log(data); // The entire file content is in memory here
});

fs.createReadStream (Stream-based)

fs.createReadStream reads the file in small, sequential chunks (by default, 64KB per chunk). Instead of waiting for the entire file to be read, it returns a Readable Stream object. This stream emits events (data, end, error) as chunks of data become available, allowing you to process data sequentially as it arrives.

const fs = require('fs');

const stream = fs.createReadStream('large-file.txt', { encoding: 'utf8' });

stream.on('data', (chunk) => {
  console.log('Received chunk of data:', chunk); // Processing data chunk-by-chunk
});

stream.on('end', () => {
  console.log('Finished reading file.');
});

Key Differences

1. Memory Consumption

• fs.readFile: Highly memory-intensive. Reading a 2GB file requires allocating at least 2GB of RAM. If multiple users request large files simultaneously, this can easily crash your Node.js process with an “Out of Memory” error.
• fs.createReadStream: Highly memory-efficient. Regardless of the file size (whether it is 10MB or 10GB), the memory footprint remains extremely low because only a small chunk of the file is loaded into memory at any given time.

2. Processing Speed and Latency

• fs.readFile: High latency. Your application must wait for the disk to finish reading the entire file before any data processing can begin.
• fs.createReadStream: Low latency. You can start processing, transforming, or sending the first chunk of data to a client almost immediately, without waiting for the rest of the file to load.

3. Maximum File Size Limits

• fs.readFile: Limited by the maximum buffer size in Node.js (typically around 2GB on 64-bit systems). Attempting to read a file larger than this limit will result in an error.
• fs.createReadStream: No size limitations. It can read files of any size because it never attempts to load the entire file into memory at once.

When to Use Which?

Use fs.readFile when:

• You are dealing with small files (e.g., configuration files, small JSON files, or metadata).
• You need the complete file contents in memory before doing any operations (such as parsing a small JSON configuration).
• Code simplicity is your primary concern and performance is not critical.

Use fs.createReadStream when:

• You are dealing with large files (e.g., video files, database backups, or massive log files).
• You want to stream data directly to another destination, such as uploading to a cloud storage service or piping a file directly to an HTTP response (stream.pipe(res)).
• You are building high-throughput, scalable web servers where memory usage must remain predictable under heavy traffic.