Handling API Pagination with Axios and Recursion

Fetching large datasets from paginated REST APIs requires iterating over multiple pages until all records are retrieved. This guide demonstrates how to handle pagination flows by implementing recursive functions using the Axios HTTP client. You will learn how to implement recursion for both page-number and cursor-based pagination strategies, handle termination conditions, and apply best practices such as rate limiting and error handling.

Understanding Recursive Pagination

Recursive pagination works by having an asynchronous function call itself to request the next page of results as long as a specific continuation condition is met (e.g., a nextPage index exists or a nextCursor token is returned). Once the API indicates that no further pages exist, the function terminates and returns the accumulated dataset.

1. Page-Number Based Pagination

In page-number or offset-based pagination, the API expects a page query parameter and typically returns metadata such as totalPages or the current page index.

const axios = require('axios');

/**
 * Recursively fetches all pages from a page-based API endpoint.
 *
 * @param {string} url - The base API endpoint.
 * @param {number} page - The current page to fetch.
 * @param {Array} accumulated - The accumulated results array.
 * @returns {Promise<Array>} - Resolves to the combined data from all pages.
 */
async function fetchByPage(url, page = 1, accumulated = []) {
  try {
    const response = await axios.get(url, {
      params: { page, limit: 50 }
    });

    const { data, totalPages } = response.data;
    const combinedData = accumulated.concat(data);

    // Base condition: Stop if we have reached or exceeded the total pages
    if (page >= totalPages || data.length === 0) {
      return combinedData;
    }

    // Recursive call for the next page
    return await fetchByPage(url, page + 1, combinedData);
  } catch (error) {
    console.error(`Failed fetching page ${page}:`, error.message);
    throw error;
  }
}

// Example usage:
// fetchByPage('https://api.example.com/items').then(all => console.log(all.length));

2. Cursor-Based Pagination

Cursor-based pagination relies on an opaque pointer (such as an ID or a token) returned in the response payload or headers to fetch the subsequent page.

const axios = require('axios');

/**
 * Recursively fetches all pages from a cursor-based API endpoint.
 *
 * @param {string} url - The base API endpoint.
 * @param {string|null} cursor - The cursor token for the next page.
 * @param {Array} accumulated - The accumulated results array.
 * @returns {Promise<Array>} - Resolves to the combined dataset.
 */
async function fetchByCursor(url, cursor = null, accumulated = []) {
  try {
    const response = await axios.get(url, {
      params: {
        limit: 100,
        ...(cursor && { cursor })
      }
    });

    const { items, nextCursor } = response.data;
    const combinedData = accumulated.concat(items);

    // Base condition: Stop if there is no next cursor or items array is empty
    if (!nextCursor || items.length === 0) {
      return combinedData;
    }

    // Recursive call with the next cursor token
    return await fetchByCursor(url, nextCursor, combinedData);
  } catch (error) {
    console.error('Cursor pagination failed:', error.message);
    throw error;
  }
}

Key Considerations and Best Practices

1. Define Clear Base Conditions

Always verify that a stopping condition exists to prevent infinite loops:

2. Handle Rate Limits

Rapid consecutive requests can trigger HTTP 429 Too Many Requests responses. Insert a delay between calls if the target API imposes strict rate limits:

const delay = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

async function fetchWithThrottle(url, page = 1, accumulated = []) {
  const response = await axios.get(url, { params: { page } });
  const { data, hasMore } = response.data;
  const combined = accumulated.concat(data);

  if (!hasMore) return combined;

  await delay(200); // 200ms delay between requests
  return fetchWithThrottle(url, page + 1, combined);
}

3. Manage Memory for Large Datasets

Accumulating hundreds of thousands of items in an in-memory array can cause memory exhaustion. For massive datasets, consider processing or streaming each batch to a database or file inside the recursive function rather than concatenating everything into memory before returning.