Evolutionary Algorithms for Optimal AI Model Merging

This article explores how evolutionary algorithms automatically discover the best layer-wise recipes for merging disparate open-weight artificial intelligence models. By treating model combination parameters as genetic code, these algorithms iteratively breed, mutate, and evaluate new hybrid models without requiring expensive GPU training. The following sections break down the mechanics of layer-wise merging, the evolutionary process, and how these techniques create superior composite AI systems from existing models.

Understanding Layer-Wise Model Merging

Open-weight AI models, particularly large language models, consist of numerous sequential layers of neural network weights. Model merging combines the capabilities of different fine-tuned models—such as one specialized in coding and another in creative writing—into a single unified model.

Instead of averaging the weights of two models equally across the board, layer-wise merging applies different ratios and mathematical techniques to specific layers. However, because a single model can have dozens of layers, manually guessing the right combination ratio for each individual layer is nearly impossible. This is where evolutionary algorithms step in to automate the discovery process.

Encoding Recipes as Genomes

To use an evolutionary algorithm, the merging recipe must first be translated into a mathematical format called a genome. A genome in this context is simply a list of numbers and instructions that dictate how to combine the source models.

For a layer-wise merge, the genome specifies:

  • Which layers to pull from which source model.
  • The exact weighting ratio (interpolation factor) for each specific layer.
  • The merging method to apply (such as Linear Interpolation, Spherical Linear Interpolation, or TIES-merging).
  • Density thresholds that determine how many parameters are kept or discarded per layer.

Generating the Initial Population

The process begins by creating an initial population of dozens or hundreds of randomized genomes. Each genome represents a unique, untested recipe for merging the selected open-weight models. The system rapidly constructs the merged AI models dictated by these recipes. Because merging only involves basic math on existing model weights rather than training from scratch, generating these candidate models takes seconds or minutes rather than weeks.

Fitness Evaluation

Once the initial population of merged models is created, each candidate must be tested to see how well it performs. This is known as the fitness evaluation.

The models are run through automated benchmark tests tailored to the desired outcome, such as math problem-solving, logic puzzles, or coding challenges. Each model receives a fitness score based on its accuracy and performance on these benchmarks. The recipes that yield clumsy or broken models score poorly, while recipes that successfully blend the strengths of the parent models receive high scores.

Selection, Crossover, and Mutation

After scoring, the evolutionary algorithm mimics natural selection to create the next generation of model recipes:

  1. Selection: The lowest-scoring recipes are deleted. The highest-scoring recipes are selected to act as "parents" for the next generation.
  2. Crossover: The algorithm combines the genomes of two successful parent recipes. For example, a child recipe might inherit the merging ratios for the first 16 layers from Parent A, and the ratios for the final 16 layers from Parent B.
  3. Mutation: To ensure the system discovers truly novel combinations, the algorithm introduces random, slight changes to the child genomes. A mutation might randomly shift the weight of layer 12 by five percent or swap out the merging formula used on layer 24.

Convergence on the Optimal Recipe

This cycle of building merged models, scoring them on benchmarks, and breeding the best recipes repeats over dozens or hundreds of generations. A prominent implementation of this exact method is the open-source tool MergeKit, alongside research frameworks developed by organizations like Sakana AI.

With each generation, the average fitness score of the population rises. Eventually, the algorithm converges on an optimal, non-intuitive layer-wise recipe that human engineers would likely never attempt—such as heavily favoring Model A in the early layers, rapidly oscillating between Model A and Model B in the middle layers, and exclusively using Model B in the final layers. Through this automated trial-and-error process, evolutionary algorithms reliably unlock superior capabilities from existing open-weight AI models.