How to Stream Large Datasets with Axios in Node.js

Fetching massive payloads using the default Axios configuration loads the entire response into memory at once, which frequently leads to ERR_BUFFER_OUT_OF_BOUNDS or JavaScript heap out-of-memory errors. The best and most memory-efficient way to handle large datasets in Axios is by configuring the responseType to stream, allowing Node.js to consume, parse, and write incoming chunks incrementally while managing backpressure.

1. Configure Axios with responseType: 'stream'

By default, Axios sets responseType to json or text. When working with large responses in Node.js, explicitly set responseType: 'stream' to receive a standard Node.js ReadableStream instead of a fully buffered JavaScript object.

const axios = require('axios');

async function getStream() {
  const response = await axios({
    method: 'get',
    url: 'https://api.example.com/large-dataset.json',
    responseType: 'stream'
  });

  return response.data; // This is a Readable stream
}

2. Pipe Directly to Storage to Avoid Memory Overhead

If the objective is to save the large payload directly to disk or forward it to another service, pipe the Axios stream to a WritableStream using Node's stream/promises pipeline. This approach handles backpressure and cleans up resources on error.

const fs = require('fs');
const { pipeline } = require('stream/promises');
const axios = require('axios');

async function downloadLargeFile(url, outputPath) {
  const response = await axios({
    method: 'get',
    url: url,
    responseType: 'stream'
  });

  const fileStream = fs.createWriteStream(outputPath);
  
  // pipeline automatically manages backpressure and error handling
  await pipeline(response.data, fileStream);
  console.log('Download complete without memory bloat.');
}

3. Process Chunk-by-Chunk for JSON Lines (NDJSON)

When processing newline-delimited JSON (NDJSON), split the stream using a transform stream or line reader. Processing records line-by-line ensures memory consumption remains flat regardless of whether the dataset is 10 MB or 100 GB.

const readline = require('readline');
const axios = require('axios');

async function processNdjsonStream() {
  const response = await axios({
    method: 'get',
    url: 'https://api.example.com/logs.ndjson',
    responseType: 'stream'
  });

  const rl = readline.createInterface({
    input: response.data,
    crlfDelay: Infinity
  });

  for await (const line of rl) {
    if (line.trim()) {
      const record = JSON.parse(line);
      // Process individual record
      await handleRecord(record);
    }
  }
}

async function handleRecord(record) {
  // Business logic (e.g., insert into DB)
}

4. Parse Large Standard JSON Arrays

If the response is a single, massive JSON array (e.g., [ {...}, {...}, ... ]), JSON.parse will fail if loaded all at once. Use a streaming JSON parser like stream-json to emit individual array items on the fly.

const axios = require('axios');
const { parser } = require('stream-json');
const { streamArray } = require('stream-json/streamers/StreamArray');

async function processLargeJsonArray() {
  const response = await axios({
    method: 'get',
    url: 'https://api.example.com/massive-array.json',
    responseType: 'stream'
  });

  const pipeline = response.data
    .pipe(parser())
    .pipe(streamArray());

  for await (const { value } of pipeline) {
    // Process each object in the array individually
    await handleRecord(value);
  }
}

Best Practices for Streaming with Axios