Dynamic Traffic Allocation with Multi-Armed Bandits

This article explores how multi-armed bandit (MAB) algorithms dynamically route user traffic during the online evaluation of production artificial intelligence recommender systems. Unlike traditional A/B testing, which locks traffic into fixed groups until an experiment concludes, MAB algorithms adjust traffic distribution in real time based on live user interactions. By continuously balancing the exploration of new AI models with the exploitation of proven high-performing ones, these algorithms minimize the opportunity cost of showing inferior recommendations and maximize overall user engagement.

The Flaw in Traditional A/B Testing

In standard A/B testing for AI recommenders, traffic is split evenly and statically—for example, 50% of users see Model A, and 50% see Model B. This fixed allocation remains unchanged for weeks until statistical significance is reached. If Model B is terrible at recommending relevant products, half of your user base endures a poor experience for the entire duration of the test, resulting in lost revenue and engagement. Multi-armed bandits solve this by refusing to wait until the end of an experiment to act on incoming data.

Exploration vs. Exploitation

Multi-armed bandits operate on a continuous loop of two states: exploration and exploitation. In the context of a production recommender system, "arms" represent the different AI models being tested.

  • Exploration involves routing traffic to various models to gather more data about their performance and reduce uncertainty.
  • Exploitation involves routing the majority of traffic to the model currently demonstrating the highest reward (such as click-through rate, conversion, or watch time).

At the start of a test, the MAB algorithm allocates traffic evenly to explore all models. As one recommender model begins generating more clicks or conversions, the algorithm automatically shifts a larger percentage of incoming user requests to that winning model, while gradually starving the underperforming models of traffic.

Core Algorithms Driving the Shift

Production systems rely on specific mathematical strategies to dictate exactly how and when traffic shifts occur. The three most common approaches are:

1. Epsilon-Greedy This is the simplest method. The algorithm defines a small parameter, epsilon (e.g., 10%). For 90% of incoming users, the system exploits the best-performing recommender model. For the remaining 10% of users, the system explores by randomly assigning them to any of the available models. While easy to implement, it wastes some traffic by continuing to explore clearly inferior models at a fixed rate.

2. Upper Confidence Bound (UCB) UCB removes randomness in favor of optimism in the face of uncertainty. It calculates an expected reward for each recommender model alongside a confidence interval based on how many times that model has been tested. Models that have not received much traffic have a wide confidence interval, giving them a high "upper bound" potential. The algorithm routes traffic to whichever model has the highest upper bound. As a model receives more traffic and fails to perform, its confidence interval shrinks, its upper bound drops, and traffic naturally shifts away from it.

3. Thompson Sampling Thompson Sampling is a Bayesian approach and the industry standard for complex production environments. Each AI model is assigned a probability distribution (usually a Beta distribution) representing its likelihood of success based on past rewards and failures. When a user loads a page, the system draws a random sample from each model's distribution and routes the user to the model that generated the highest number. As a model gathers more positive interactions, its distribution shifts higher and narrows, ensuring it gets selected for the vast majority of future traffic.

The Production Feedback Loop

To shift traffic dynamically in a live environment, the system relies on a rapid feedback loop.

  1. Request: A user opens an app, triggering a request for recommendations.
  2. Routing: The MAB layer intercepts the request, evaluates the current state of all models, and selects one to serve the user.
  3. Action: The chosen AI recommender generates a list of items, and the user either interacts with them (a reward of 1) or ignores them (a reward of 0).
  4. Update: The interaction data is streamed back to the MAB controller. Rather than updating on every single click—which can overwhelm production servers—systems usually aggregate rewards in micro-batches (e.g., every few minutes) to recalculate the probabilities and adjust traffic routing weights for the next wave of users.

Adapting to Changing User Behavior

A major advantage of using MABs for AI recommenders is their ability to handle non-stationary environments. User preferences change based on the time of day, breaking news, or seasonal trends. If Model A performs best in the morning, the bandit will shift traffic to it. If Model B suddenly performs better in the evening, the small amount of exploratory traffic it still receives will register a spike in rewards. The MAB algorithm will detect this shift, widen its confidence interval or update its probability distribution, and dynamically pivot the bulk of user traffic over to Model B without requiring human intervention.