How to Install NVIDIA Drivers and CUDA on Amazon EC2?

Installing and managing NVIDIA drivers and the CUDA toolkit on Amazon EC2 accelerated computing instances requires matching your driver type to your specific instance family and workload, or bypassing manual setup altogether by selecting an AWS Deep Learning AMI. You can deploy production-ready GPU environments using pre-configured Amazon Machine Images, install public NVIDIA Data Center drivers for compute workloads on P, G4dn, G5, and G6 instances, or configure AWS-provided GRID drivers from Amazon S3 for virtual workstation graphics on G-family instances. Proper ongoing management involves locking kernel and driver versions to prevent unattended breaks, configuring persistence mode, and utilizing containerized runtime toolkits to isolate application CUDA dependencies from host drivers.

Choosing Your Installation Strategy

Before running installation commands, determine whether you need a custom installation from scratch or a pre-built image.

Using AWS Deep Learning AMIs (DLAMI)

The fastest way to deploy an EC2 GPU instance is to launch it with an AWS Deep Learning Base AMI. These official images come pre-installed with NVIDIA drivers, multiple versions of the CUDA toolkit, cuDNN, NCCL, and the NVIDIA Container Toolkit.

On a multi-CUDA DLAMI, you can switch between installed CUDA versions dynamically by updating your symbolic link and environment variables:

sudo rm /usr/local/cuda
sudo ln -s /usr/local/cuda-12.4 /usr/local/cuda

Selecting the Right NVIDIA Driver Type

If you are building on top of a standard base OS (such as Amazon Linux 2023, Ubuntu 22.04/24.04, or Rocky Linux), you must select the driver type that matches your EC2 instance family and workload:

  • NVIDIA Data Center (Tesla) Drivers: Designed for compute, machine learning, and general GPU processing. Supported on P3, P4, P5, G3, G4dn, G5, G5g, G6, and G6e instances. Note that when used on G-family instances, Data Center drivers support compute workloads and only a single virtual display at low resolution.
  • NVIDIA GRID Drivers: Provided directly by AWS via Amazon S3 for G3, G4dn, G5, G6, and G6e instances. These include an active NVIDIA vGPU software license for Quadro Virtual Data Center Workstation features, supporting up to four 4K monitors and hardware-accelerated graphics APIs (DirectX, Vulkan, OpenGL) alongside CUDA compute.
  • AWS Gaming Drivers: Specialized drivers hosted on S3 for single-monitor cloud gaming workloads on G4dn, G5, and G6 instances running Windows Server.

Preparing the Base EC2 Instance

Regardless of your Linux distribution, you must install kernel headers and compiler tools matching your active running kernel, and disable the open-source nouveau driver before installing proprietary NVIDIA modules.

1. Update Kernel and Install Build Dependencies

For Amazon Linux 2023:

sudo dnf update -y
sudo dnf install -y kernel-devel-$(uname -r) kernel-headers-$(uname -r) \
    gcc make dkms acpid libglvnd-glx libglvnd-opengl libglvnd-devel pkgconfig

For Ubuntu (22.04 / 24.04 LTS):

sudo apt-get update && sudo apt-get upgrade -y
sudo apt-get install -y linux-headers-$(uname -r) build-essential dkms pciutils

If the kernel was upgraded during dnf update or apt-get upgrade, reboot the instance (sudo reboot) before proceeding so uname -r matches the newly installed kernel headers.

2. Verify GPU Hardware Presence

Verify that the PCI subsystem detects the attached NVIDIA GPUs:

lspci | grep -i nvidia

3. Disable the Nouveau Driver

Blacklist the nouveau driver so it does not lock the GPU hardware at boot:

cat <<EOF | sudo tee /etc/modprobe.d/blacklist-nouveau.conf
blacklist nouveau
options nouveau modeset=0
EOF

Regenerate the initial ramdisk and reboot:

# On Amazon Linux / RHEL:
sudo dracut --force
sudo reboot

# On Ubuntu / Debian:
sudo update-initramfs -u
sudo reboot

Installing NVIDIA Drivers and CUDA via Package Managers

Using NVIDIA's official CUDA network repositories is the cleanest way to install and manage both the Data Center driver and the CUDA toolkit on compute instances.

Installation on Amazon Linux 2023

Add the NVIDIA CUDA repository and install the driver and toolkit:

sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/amzn2023/x86_64/cuda-amzn2023.repo
sudo dnf clean expire-cache
sudo dnf module install -y nvidia-driver:latest-dkms
sudo dnf install -y cuda-toolkit

Installation on Ubuntu 22.04 / 24.04

Download and install the cuda-keyring package to authenticate the repository, then install the driver and CUDA toolkit:

# Replace ubuntu2204 with ubuntu2404 if running 24.04, or sbsa for Graviton G5g
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get install -y cuda-drivers cuda-toolkit-12-4

Note: Installing cuda-drivers along with a specific cuda-toolkit-X-Y package gives you granular control over the toolkit version rather than installing the meta-package cuda, which automatically pulls the latest release of both.

Installing AWS GRID Drivers from Amazon S3 (G-Family Instances)

If you are running a virtual workstation or graphics workload on a G4dn, G5, or G6 instance, install the pre-licensed GRID driver from the official AWS S3 bucket (ec2-linux-nvidia-drivers). Ensure your instance has an IAM role with AmazonS3ReadOnlyAccess attached.

aws s3 cp --recursive s3://ec2-linux-nvidia-drivers/latest/ .
chmod +x NVIDIA-Linux-x86_64*.run
sudo CC=/usr/bin/gcc ./NVIDIA-Linux-x86_64*.run --dkms -s

After installing the GRID driver, verify that vGPU licensing is active by creating /etc/nvidia/gridd.conf if required by your driver generation, and reboot the instance. You can then install the standalone cuda-toolkit package (without overwriting the driver) or use a runfile installer with the --toolkit flag:

sudo sh cuda_<version>_linux.run --silent --toolkit

Configuring Environment Variables

Once the CUDA toolkit is installed into /usr/local/cuda, configure system-wide environment paths so nvcc and shared libraries are accessible to all users:

cat << 'EOF' | sudo tee /etc/profile.d/cuda.sh
export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
EOF
source /etc/profile.d/cuda.sh

Verifying the Installation

Run nvidia-smi to confirm the kernel module is loaded and communicating with the GPU, and nvcc --version to verify the CUDA compiler toolkit:

nvidia-smi
nvcc --version

In the nvidia-smi output header, you will see a CUDA Version field. This displays the Driver API version (the highest CUDA runtime version supported by the installed driver), whereas nvcc --version reports the CUDA Runtime version installed in /usr/local/cuda. Thanks to NVIDIA's forward/backward compatibility model, your host driver works properly as long as:

\[\text{Version}_{\text{Driver API}} \ge \text{Version}_{\text{CUDA Runtime}}\]

Ongoing Management and Optimization on EC2

Managing GPU fleets in production requires configuring persistence, handling high-speed interconnects on multi-GPU nodes, and preventing accidental version drift.

1. Enable Persistence Mode

By default, the Linux kernel tears down GPU state when no processes are actively using the device, causing initialization latency on subsequent job runs. Enable persistence mode via the nvidia-persistenced daemon:

sudo systemctl enable --now nvidia-persistenced

Alternatively, enable it directly via nvidia-smi:

sudo nvidia-smi -pm 1

2. Disable Autoboost and Lock Clocks for Predictable Performance

On instances like P3, G4dn, and G5, locking the memory and graphics clocks to their maximum supported frequencies eliminates thermal throttling jitter across distributed training workers:

# Query supported maximum clock speeds
nvidia-smi -q -d SUPPORTED_CLOCKS

# Example for NVIDIA A10G (G5 instances):
sudo nvidia-smi --lock-gpu-clocks=1710

3. Manage NVIDIA Fabric Manager on NVSwitch Instances

On multi-GPU instances connected via NVSwitch (such as p4d.24xlarge, p4de.24xlarge, and p5.48xlarge), GPUs cannot communicate over NVLink unless the nvidia-fabricmanager service is installed and running at the exact same version as the kernel driver:

# Install Fabric Manager matching your exact driver version (e.g., 550.90.07)
sudo dnf install -y nvidia-fabric-manager-550.90.07-1
sudo systemctl enable --now nvidia-fabricmanager

4. Prevent Unintended Driver and Kernel Upgrades

Unattended OS updates can upgrade the Linux kernel or partially update NVIDIA packages, causing Failed to initialize NVML: Driver/library version mismatch errors until the instance is rebooted or modules are rebuilt.

Lock packages in production environments:

# Ubuntu / Debian:
sudo apt-mark hold nvidia-driver-* cuda-drivers linux-image-aws linux-headers-aws

# Amazon Linux 2023:
sudo dnf install -y 'dnf-command(versionlock)'
sudo dnf versionlock add nvidia-* cuda-* kernel*

5. Decouple CUDA Versions with NVIDIA Container Toolkit

Instead of managing multiple conflicting CUDA versions on the host OS, install only the NVIDIA driver on the EC2 host and use Docker or containerd with the NVIDIA Container Toolkit. This allows each container to bring its own user-space CUDA toolkit, cuDNN, and framework dependencies while sharing the host's kernel driver.

To configure the NVIDIA Container Toolkit on Ubuntu:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify container GPU pass-through by running an official CUDA container:

sudo docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Troubleshooting Common Issues

  • NVML: Driver/library version mismatch: The user-space NVIDIA library was updated while the old kernel module is still loaded in memory. Terminate processes using the GPU (sudo fuser -v /dev/nvidia*), unload the modules (sudo rmmod nvidia_uvm nvidia_drm nvidia_modeset nvidia), or reboot the instance (sudo reboot).
  • nvidia-smi fails after an OS reboot: The instance booted into a newer kernel that lacks a compiled NVIDIA module. Verify that dkms status shows the nvidia module as installed for $(uname -r). If missing, install linux-headers-$(uname -r) or kernel-devel-$(uname -r) and run sudo dkms autoinstall.
  • Instance hangs or fails status checks after modifying drivers: Ensure AWS Nitro instances have the ena and nvme drivers intact in the initramfs before rebooting after dracut or update-initramfs.