Minimum Bayes Risk Decoding in Machine Translation AI

This article explores how Minimum Bayes Risk (MBR) decoding improves machine translation artificial intelligence by selecting the most accurate translation from a pool of generated candidates. Unlike traditional methods that simply look for the single most probable output, MBR decoding uses Monte Carlo sampling to generate multiple possible translations and compares them against one another. By evaluating these candidates using a specific utility metric, the algorithm successfully identifies the translation that minimizes expected loss and maximizes overall quality.

The Flaw in Standard Decoding

To understand Minimum Bayes Risk decoding, it is first necessary to look at standard decoding methods like beam search. Beam search attempts to find the single translation with the highest probability (Maximum A Posteriori, or MAP). However, in neural machine translation, the highest probability sequence is not always the best translation. Because probability mass is often spread across many similar, high-quality translations, the absolute most probable output can sometimes be overly generic, short, or even flawed. MBR decoding solves this by looking for the translation that is most similar to other likely translations, rather than the one with the highest raw probability.

Generating the Pool via Monte Carlo Sampling

Because a machine translation model can generate an almost infinite number of possible translations, it is computationally impossible to compare every single one. MBR decoding overcomes this hurdle by using Monte Carlo sampling.

The AI model generates a finite pool of candidate translations by randomly sampling from its output probability distribution. This pool usually consists of anywhere from a few dozen to a few hundred sentences. Because they are drawn directly from the model's probability distribution, this Monte Carlo pool serves as an accurate, manageable representation of what the model believes are valid translations. These sampled sentences act as both the candidate translations and the "pseudo-references" used to evaluate them.

Calculating Expected Loss and Utility

Once the pool of Monte Carlo samples is generated, MBR decoding must find the candidate that minimizes "expected loss" (or conversely, maximizes expected utility). In this context, "loss" refers to translation errors, which are measured using automated evaluation metrics like BLEU, ChrF, or COMET.

The algorithm performs a pairwise comparison across the entire pool:

  1. It takes Candidate A and compares it against Candidate B, Candidate C, Candidate D, and so on, using the chosen evaluation metric.
  2. It calculates the average score of Candidate A against all other samples in the pool.
  3. It repeats this process for every single candidate in the Monte Carlo pool.

Because the pool was generated using Monte Carlo sampling, a simple average of these metric scores accurately estimates the expected utility of each candidate. High-probability themes and structures will naturally appear multiple times in the sampled pool. Therefore, a candidate that aligns well with the majority of the pool will achieve a high average score, which equates to low expected loss.

Selecting the Final Translation

The final step in MBR decoding is selection. The algorithm ranks all the candidates based on their average utility scores. The candidate translation that achieved the highest average similarity to all other Monte Carlo samples is selected as the final output.

By choosing the consensus translation, MBR decoding effectively filters out bizarre hallucinations and structural errors. Even if the model accidentally assigns a high probability to a flawed translation, that flawed sentence will look very different from the rest of the sampled pool. Consequently, it will receive a low average score and be discarded, ensuring the final selected translation is safe, accurate, and representative of the model's true understanding.