Reinforcement Learning in Pumped-Storage Hydro Scheduling

Pumped-storage hydroelectricity (PSH) operators increasingly rely on reinforcement learning (RL) to automate and optimize the scheduling of water pumping and power generation. This article provides an overview of how RL algorithms interact with volatile energy spot markets, the specific data inputs required to make real-time operational decisions, and the way artificial intelligence balances financial rewards against the physical constraints of hydroelectric reservoirs.

The Challenge of Volatile Spot Markets

Energy spot markets experience extreme price fluctuations driven by unpredictable renewable energy output, sudden shifts in consumer demand, and grid congestion. PSH plants act as massive batteries: they pump water from a lower reservoir to an upper reservoir when electricity prices are low, and release that water through turbines to generate electricity when prices are high.

Traditional scheduling relies on linear programming and historical price averages. However, these static methods struggle to adapt to sudden market anomalies, such as negative pricing events caused by excess solar or wind generation. Reinforcement learning solves this by dynamically adapting to real-time market chaos without needing a pre-programmed model of how the market behaves.

How Reinforcement Learning Works in PSH

Reinforcement learning is a type of machine learning where an autonomous "agent" learns to make decisions by performing actions in an environment and receiving feedback in the form of rewards or penalties. In the context of PSH, the system is structured into three core components:

1. The State (Inputs)

The state represents the current reality the AI observes at any given timestamp. Operators feed the RL agent a continuous stream of data, including:

  • Current and historical spot market electricity prices.
  • Water levels in both the upper and lower reservoirs.
  • Weather forecasts (which influence both regional power demand and natural water inflows).
  • Grid frequency and ancillary service demands.
  • Current operational status of the pumps and turbines.

2. The Action (Decisions)

Based on the current state, the RL agent must choose an action for the next time interval (often divided into 5-minute, 15-minute, or hourly blocks). The primary actions are:

  • Pump: Consume grid electricity to move water to the upper reservoir.
  • Generate: Release water through turbines to sell electricity to the grid.
  • Idle: Do nothing and hold current water levels to wait for better market conditions.

3. The Reward (Feedback)

The reward function tells the AI how good its decision was. The primary reward is net financial profit: revenue earned from generating power minus the cost of electricity used for pumping.

However, operators also program penalties into the reward function. The AI is penalized if it attempts to pump water when the upper reservoir is full, tries to generate power when the reservoir is empty, or switches between pumping and generating too frequently, which causes mechanical wear and tear on the turbines.

Training the AI in Simulated Environments

Operators cannot let an untrained AI experiment directly on a live power plant and real energy markets, as early trial-and-error mistakes would result in massive financial losses and equipment damage.

Instead, operators train RL algorithms—such as Deep Q-Networks (DQN) or Proximal Policy Optimization (PPO)—inside a digital twin of the hydro plant. They run the AI through years of historical spot market data and millions of synthetic market scenarios. Through this process, the agent learns complex strategies. For example, it learns to resist the urge to sell water during a minor morning price spike if weather data suggests a massive wind-power drop-off will cause an even larger price spike that evening.

Real-Time Execution and Bidding

Once trained, the RL model is deployed into the plant's Energy Management System (EMS). As the spot market updates, the AI processes the new state data in milliseconds and outputs the optimal operational schedule.

In many jurisdictions, the RL agent does not just flip switches at the plant; it actively formulates bidding strategies. It calculates the exact price thresholds at which the plant should offer to buy or sell power in day-ahead and real-time intraday markets. By continuously recalculating the future value of stored water against immediate market prices, reinforcement learning maximizes arbitrage revenue while stabilizing the broader electrical grid.