Node.js Readable.from: Convert Async Generators to Streams

This article explains how the Readable.from() utility in Node.js bridges the gap between modern asynchronous generators and traditional streams. We will explore how this method works, its internal mechanics regarding backpressure, and why it is a crucial tool for efficient, memory-friendly data processing in JavaScript.

Understanding the Role of Readable.from()

In modern JavaScript, async generators (async function*) provide an elegant way to produce data sequentially and asynchronously. However, older Node.js APIs and ecosystem libraries rely heavily on Node.js Streams for data handling.

The Readable.from() utility, introduced in Node.js 12.3.0, acts as an adapter. It wraps any iterable or async iterable—including async generators—and turns it into a fully functional Node.js Readable stream. This allows you to write clean generator functions while maintaining compatibility with stream-based destinations, such as file writers (fs.createWriteStream) or HTTP responses.

Key Benefits and Functions

1. Automatic Iteration and Buffering

When you pass an async generator to Readable.from(), Node.js automatically manages the iteration process. Under the hood, the stream consumer triggers the next() method of the generator. The utility handles the resolution of promises returned by the generator, wrapping yielded values into data chunks and pushing them down the pipeline.

2. Built-in Backpressure Management

One of the hardest parts of writing custom streams is handling backpressure—ensuring a fast data producer does not overwhelm a slow consumer. Readable.from() manages this automatically. If a downstream writable stream is busy and pauses the readable stream, Readable.from() pauses pulling data from your async generator until the stream is ready to receive more data.

3. Compatibility and Interoperability

By converting an async generator to a stream, you unlock the ability to use standard stream methods like .pipe() and .pipeline(). This makes it simple to stream data directly from database cursors, API pagination loops, or large file parsers into standard Node.js destinations.

Practical Implementation Example

Converting an async generator into a stream requires very little code. Here is a basic implementation:

const { Readable } = require('stream');
const fs = require('fs');

// 1. Define an async generator that yields data over time
async function* dataProducer() {
  const items = ['Line 1\n', 'Line 2\n', 'Line 3\n'];
  for (const item of items) {
    // Simulate async delay (e.g., waiting for an API response)
    await new Promise((resolve) => setTimeout(resolve, 100));
    yield item;
  }
}

// 2. Convert the generator to a Readable stream using Readable.from()
const readableStream = Readable.from(dataProducer());

// 3. Pipe the resulting stream to a destination file
const writeStream = fs.createWriteStream('output.txt');
readableStream.pipe(writeStream);

readableStream.on('end', () => {
  console.log('Streaming completed successfully.');
});

In this example, the generator yields strings after a simulated delay. Readable.from() converts this generator into a stream, which is then piped directly to a file. The conversion ensures that memory usage remains low, as data is written to the disk chunk-by-chunk rather than being accumulated in RAM first.