Byte Blackouts: How large data center loads are surfacing new issues

In mid-July 2024, data center demand equaling nearly a third of all households in Virginia suddenly disappeared from the grid. These data centers, concentrated over just a few miles in the Dominion zone of PJM, were so flexible in shifting load that they nearly triggered a byte blackout.

While PJM isn’t alone in facing load growth from data centers, other regions have seen battery capacity expansion that can match the flexibility of new large loads to turn on and off with little notice. Conversely, PJM’s battery participation has declined from a decade ago.

In this post, we’ll take a look at batteries in PJM, and the data from a specific event in the summer of 2024 where, in order to keep themselves online, data centers risked grid reliability for everyone else.

A Byte Blackout

In a recent incident report, NERC documented how data centers shifting from the grid to uninterruptible power supply (UPS) systems created critical conditions for an unnamed entity in the eastern interconnection. Reuters then reported more details following an investigation: it was data center alley, in Dominion, and the fault was on the Ox-Possum 230kV line near Fairfax.

During a thunderstorm on the evening of July 10th, 2024, a lightning strike arrestor on a 230kV transmission line failed, which led to six voltage drops as controls attempted to account for this failure. These rapid voltage drops led to nearly 1,500 MW of data centers automatically switching to internal generation and suddenly going off grid, an extreme outlier in terms of typical 5-minute load swings in Dominion. 

Unlike some new drivers of load growth like transportation electrification, data centers require stable, non-cyclical power in order to run servers and cycle waste heat. In addition to the need for constant power, voltage sensitivity is a consideration for operators as fluctuations from 60Hz may cause damage to sensitive, and expensive equipment. 

As we’ve documented, Northern Virginia is home to the largest collection of data centers in the world. This region, known as “data center alley” has led to considerable increases in Dominion’s load as well as their future load forecasts. Due to the high concentration of load and limited generation nearby, congestion in Northern Virginia is a relatively common occurrence, particularly on high load days. 

On the day of the event, Beaumeade substation, located next to a data center, had an on peak average congestion price of over $200 leading up to 19:00.      

Thunderstorms in the summer often reduce load as cooler temperatures and cloud coverage help to reduce air conditioning load, especially over dense, power hungry areas like cities, so some congestion drop off would be expected as storms rolled into the urbanized DMV. However, the magnitude of this drop in congestion is far from typical, particularly as Beaumeade MCCs were ~$159/MWh for the interval preceding the load drop.

System wide energy costs more than halved as load dropped, falling from ~$134/MWh to $56.70/MWh in the following five minute interval, while congestion costs at the Beaumeade substation dropped by ~97%, highlighting the ability of nodal electricity markets to pinpoint system stress by assigning public values to the point of both injection and withdrawal. 

As we mentioned earlier, thunderstorms during the summer are bearish for cooling load, particularly over the afternoon and evening. The same system that pushed through Northern Virginia had impacts across the Potomac in densely populated Washington, DC. On both July 9th and 10th, warm summer weather drove demand peaks over 5 GW in the late afternoon. Load fell more gradually after the peak on the night of the 9th, with a 10% reduction in load over a four hour period as temperatures slowly moderated after sunset. Thunderstorms the following night allowed for temperatures to fall rapidly, resulting in a 16% drop in load over the same time period, even as load peaked slightly stronger day over day.

Limited tools for new operational headaches

The sudden drop in load impacted more than just prices. While loss of generation on the grid can drop frequency below its ideal level, loss of load sends frequency in the opposite direction. The NERC report noted that PJM-wide frequency went as high as 60.047 Hz, well above the +/- 0.036 Hz band that NERC targets.

In order to return the local transmission network to the appropriate configuration, Dominion removed capacitor banks from service in the local transmission network. Despite quick utility action, the data centers remained off the grid for hours.

While UPS backups can be activated automatically, reconnecting to the grid requires manual intervention. This event raises important questions for both operators and planners.

As a recent Duke University report noted, data centers could be flexible grid assets, but that flexibility coupled with high volume of large load customers also opens up the possibility for demand swings steeper than operators are used to handling, with new recovery mechanics. PJM’s balancing operations manual illustrates the limited options operators have to manage high frequency. 

Grid operators have time-tested tools at their disposal, such as synchronized reserves, demand response, and other ancillary services to bring generation online or manually lower demand in order to restore balance.

However, as a general rule of thumb, it's easier to ramp something up than to arrest a rapidly rotating mass of metal.

While batteries respond extraordinarily quickly in both directions, most markets are years away from ancillary services saturation by new technologies. Other active loads in the market can be reduced when frequency is too low, but the flexibility to instantly increase consumption by large amounts at PJM’s directive does not currently exist.

Unlike the trip of a large facility, operators have few options to respond, relying instead on the physical processes of the local transmission service provider. This dearth of tools was reflected in a lack of ancillary service deployments in the minutes surrounding this event. PJM breaks frequency regulation into two categories: Regulation A (RegA), a slower-responding program reliant primarily on traditional thermal generation, and Regulation D (RegD), a fast-responding service ideal for BESS. BESS are particularly well-suited for RegD as they can rapidly charge or discharge to stabilize grid frequency as needed. Following the load drop on July 10th, RegD mileage increased by approximately 10 MW, but total procured MW actually declined. With insufficient storage resources available, PJM had to rely more heavily on the slower RegA, which increased by 110 MW during that hour.

In one sense this demand was overly flexible, and unlike operator-directed demand response, the assets dropped, and stayed, offline, for customer prerogatives.

While Northern Virginia is a legacy hotbed of data center development, other regions are growing fast — with Dallas anticipating more than 10 GW of new large loads according to ERCOT’s 2024 RTP. As the new vanguard of varied growth in large loads, ERCOT has been ramping up stakeholder processes to understand and standardize these new entities. The March 28th meeting of the ISO’s Large Flexible Load Task Force identified this exact issue, backup generation failover, as “When a good thing becomes a bad thing.”

PJM’s Battery Burnout 

Outside of ERCOT, the growth in flexible load has not been matched by a corresponding rise in symmetrically flexible generation. PJM, in particular, lags behind other ISOs, with notably limited participation of battery storage in its market.

According to February’s EIA Form 860, PJM has just ~ 400 MW of installed battery capacity, compared to nearly 8 and 12 GWs for ERCOT and CAISO respectively. But even this paltry capacity number does not reflect reality on the ground. In PJM’s fuel mix data, batteries remain a vanishingly small slice, with almost no generation reported from 2018 until 2023, when it rose to a regular ~20 MW in November. This uptick appears to be driven by a single 20MW, 4-hour system, Dominion’s Dry Bridge Storage, as PJM’s reported storage generation has yet to exceed 20 MW since its commissioning.

At one point, PJM was a national leader in battery storage, but more than 100 MW of batteries that came online between 2012 and 2017 have been deactivated. These deactivations arose from a combination of issues related to an industry in its infancy, including EPC problems and cycling wear and tear, as well as a rapid deterioration in project economics following PJM rule changes at the beginning of 2017 as detailed in this Greentech Media piece from that summer. 

PJM’s lack of battery storage is not likely to change in the immediate future. 

The top-line numbers reflect broader market conditions—ERCOT and CAISO continue to lead by a wide margin, while PJM lags significantly behind, with less than 100 MW of new projects expected to come online by the end of the year.

These MW numbers are not precise due to the inconsistent treatment of MW vs MWh in public queue data, but this uncertainty is insufficient to alter the disparity between regions.  While PJM, and particularly Dominion, are actively preparing for growing data center load, this increase in flexible load is not being met in tandem with flexible battery storage. 

Additionally, as interconnection backlogs are leading to longer wait times and the push for more behind the meter generation, events in the inverse of this, where back-up generation trips and load may shift to the grid, pose reliability problems in the other direction.

Conclusion

The rise of compute-driven power consumption and the need for additional data center capacity has juiced the already rapid demand growth in northern Virginia and is reshaping the landscape of PJM. As these new loads continue to come online, PJM has limited options to manage the new operational headaches driven by data centers. Without additional battery storage capacity on the immediate horizon and limited tools at operators’ disposal, the stalwarts of frequency regulation and related ancillary services will play a growing role in maintaining reliability until increasing flexible load is matched by flexible generation. 

In the near future we'll also be publishing a look at the ERCOT experience, where flexible loads from batteries to Bitcoin mining actively participate in the market.

Nationally, discontinuities in planning processes between load and generation will exacerbate these issues, particularly in regions that struggle with interconnection, a struggle that PJM knows all too well. While years of interconnection reform are now in place, and a new partnership with Google promises an AI solution to queue woes, a second, and hopefully longer-lasting, PJM battery era remains only on the horizon.