How AI Upscaling and Frame Generation Boost Game FPS

This article explores the inner workings of real-time neural upscaling and frame generation technologies, such as NVIDIA’s DLSS (Deep Learning Super Sampling), and how they leverage artificial intelligence to drastically improve video game performance. By rendering games at lower resolutions and using deep learning models to reconstruct high-resolution images and generate entirely new frames, these technologies alleviate graphics card bottlenecks. Readers will learn the mechanics of AI-driven spatial upscaling, motion vector analysis, optical flow acceleration, and how these elements combine to deliver high frame rates without sacrificing visual fidelity.

The Core Problem: Native Rendering Bottlenecks

Traditional video game rendering requires a Graphics Processing Unit (GPU) to calculate the color, lighting, and shading for millions of individual pixels on every single frame. At 4K resolution, a GPU must compute over 8.2 million pixels per frame. Doing this 60 to 120 times per second while utilizing heavy workloads like real-time ray tracing puts an immense strain on hardware, resulting in low frame rates. AI-driven rendering solves this by changing the fundamental workflow: the GPU renders far fewer pixels, and artificial intelligence fills in the blanks.

How AI Upscaling Works

Neural upscaling (such as DLSS Super Resolution) increases frame rates by reducing the initial workload of the GPU. Instead of rendering a game at a native 4K resolution, the GPU renders the frame at a lower internal resolution, such as 1080p or 1440p.

Once this low-resolution frame is produced, an AI model takes over to upscale it to the target resolution. This process relies on three main inputs:

  1. The Current Low-Resolution Frame: The raw image just rendered by the GPU.
  2. Previous High-Resolution Frames: Stored data from earlier frames to maintain temporal stability and prevent flickering.
  3. Motion Vectors: Data generated by the game engine that tells the AI the exact direction and speed that objects in the 3D space are moving from one frame to the next.

These inputs are fed into a convolutional autoencoder—a specific type of neural network. Before the technology ever reaches a gamer's PC, this neural network is trained on supercomputers using thousands of pristine, ultra-high-resolution reference images (16K resolution). The AI learns to recognize patterns, edges, and textures.

On the user's GPU, dedicated AI processors (like NVIDIA's Tensor Cores) run this trained model in real time. The AI compares the current low-resolution frame with the previous high-resolution frames, aligns them using the motion vectors, and reconstructs a sharp 4K image. Because the GPU only had to natively render a fraction of the pixels, the frame rate increases significantly.

How Neural Frame Generation Works

While upscaling multiplies pixels, neural frame generation (introduced in DLSS 3) multiplies entire frames. Instead of relying entirely on the CPU and GPU to calculate every frame sequentially, the AI generates completely new intermediate frames and inserts them between traditionally rendered frames.

This process is much more complex than upscaling because the AI cannot rely solely on game engine motion vectors. Game vectors track physical geometry (like a moving car), but they do not track pixel-level phenomena like moving shadows, reflections, or particle effects. Relying only on game vectors to create a new frame would result in severe visual artifacts.

To solve this, frame generation uses a hardware component called an Optical Flow Accelerator. The process works as follows:

  1. Capturing Two Frames: The GPU renders Frame A and Frame B normally.
  2. Optical Flow Analysis: The Optical Flow Accelerator analyzes the pixel-level changes between Frame A and Frame B. This captures the movement of lighting, particles, and reflections that the game engine's motion vectors miss.
  3. AI Synthesis: The neural network takes Frame A, Frame B, the game engine's motion vectors, and the optical flow data. It calculates what the exact midpoint between Frame A and Frame B should look like.
  4. Frame Insertion: The AI outputs this newly hallucinated "Frame 1.5" and displays it to the player before showing Frame B.

By generating every second frame using AI, the technology can effectively double the perceived frame rate. Furthermore, because the AI generates these frames independently on the GPU, it completely bypasses CPU bottlenecks, which often hold back performance in complex, physics-heavy games.

Managing Latency

A major byproduct of frame generation is increased input latency. Because the system must render Frame B before it can generate the intermediate frame that goes between A and B, there is a slight delay between a player pressing a button and seeing the result on screen.

To counter this, AI frame generation technologies are paired with latency-reduction software (such as NVIDIA Reflex). This software synchronizes the CPU and GPU, eliminating the render queue so the player's inputs are sampled at the absolute last millisecond before the frame is processed. This keeps the game feeling responsive while the AI handles the heavy lifting of boosting the visual frame rate.