Why OBS Game Capture Fails in Borderless Windowed Mode

When you switch a video game from exclusive fullscreen to borderless windowed mode, your OBS Studio Game Capture source often freezes, displays a black screen, or stops capturing entirely. This issue occurs because of how Windows renders different display modes and how OBS hooks into game engines to capture video. This article explains the underlying technical reasons for this conflict and provides quick, actionable solutions to restore your stream or recording.

Technical Cause: Graphics Hooking and Windows DWM

The primary reason Game Capture stops working is how OBS detects and hooks into the game’s rendering pipeline.

When a game runs in Exclusive Fullscreen mode, it bypasses the Windows Desktop Window Manager (DWM). The game takes direct control of your graphics card’s output. OBS’s “Game Capture” source is specifically designed for this mode; it injects a graphics hook directly into the game’s DirectX, OpenGL, or Vulkan API to capture the frames before they reach your monitor.

When you switch the game to Borderless Windowed mode, the game ceases to have exclusive control of the display. Instead, the Windows DWM takes over, rendering the game as a standard desktop window that just happens to cover the entire screen. Because the game is now behaving like a regular window, OBS’s direct graphics hook loses its target, resulting in a frozen image or a black screen.

How to Fix the Issue

To resolve this issue without reverting your game back to exclusive fullscreen, you can adjust your OBS settings using one of the methods below.

Method 1: Change Game Capture to “Capture Specific Window”

By default, OBS Game Capture is often set to “Capture any fullscreen application.” Once a game is borderless, OBS no longer sees it as a fullscreen application. 1. Double-click your Game Capture source in OBS. 2. Change the Mode dropdown menu from “Capture any fullscreen application” to Capture specific window. 3. In the Window dropdown, select your active game from the list. 4. Click OK.

Method 2: Use Window Capture Instead

If Game Capture still fails to hook into the borderless window, Windows Capture is the most reliable alternative. 1. Add a new source in OBS and select Window Capture. 2. Select your game from the Window dropdown menu. 3. Set the Capture Method to Windows 10 (1903 and up). This modern capture method offers high performance and reduces latency for borderless applications. 4. Click OK.

Method 3: Run OBS as Administrator

Sometimes, Windows prevents OBS from hooking into applications due to user account control restrictions, especially when switching window modes. 1. Close OBS Studio. 2. Right-click the OBS Studio shortcut on your desktop or start menu. 3. Select Run as administrator. 4. Try capturing the borderless game again.