Before AI Can Learn, the Grid Must Be Configured to Remember: Part 1

While generative AI appeared seemingly overnight, its foundation took decades to build. The friendly chat interfaces of ChatGPT and Claude were the first part the public could see. However, these frontier models were built on a vast corpus that had been accumulating for over three decades: catalogs, transactions, publications, conversations, and cat videos compounded as businesses and society moved online. While none of this digitalization was done to train future AI, the AI revolution today would not exist without this essential training data.

The grid inherits no such gift. The grid’s behavior is not accidentally captured, but must be intentionally measured with sensors. Complicating this process, the behavior of this critical infrastructure is analog, continuous in both time and value. Sampling captures isolated moments of a signal that never pauses, and digitizing rounds each captured moment to the nearest step a device can represent. The faster the sensor samples, the more potential information is captured. If the industry wants AI that can reason about physical infrastructure, what I call Physical AI, it cannot assemble a corpus after the fact. It must build an observatory on purpose and start before it knows every question the data should answer. Note that this is fundamentally different from the AI craze sweeping utility boardrooms today, where the goal is applying those new AI tools to back-office processes and enterprise data.

Faster Problems, Weaker Models

Two trends make deliberate data collection unavoidable.

First, the grid’s problems are getting faster. AI data centers can draw power as rhythmic oscillations, with swings of tens of megawatts at millisecond timescales. Inverter-based resources (IBRs) run layered control loops that can interact across hundreds of miles and produce oscillations no individual device intended. The half-hour before the 2025 Iberian blackout exhibited exactly such oscillations. A conductor can slap, arc, and ignite dry vegetation between two SCADA scans, ultimately starting a utility-defining forest fire. These dynamics play out in milliseconds while operational data paths see in seconds. An architecture that senses slowly can see only slow problems.

What cannot be modeled must be measured

Second, modeling is losing ground. Traditional grid assets, often giant hunks of spinning metal, obey physics understood, validated, and verified over decades. Semiconductor-driven equipment now floods both sides of the grid, with IBRs and batteries on the generation side and EVs and inverter-based loads like data centers on the consumption side, and none of it obeys equations that engineers can look up. Their defining behavior is encoded in proprietary software, and a firmware update can change it overnight. What cannot be modeled must be measured.

Configured to Forget

Many utilities do not realize the vast number of sensors already deployed. Market forces have pushed asset manufacturers to embed sensing into literally millions of devices already deployed across today’s grid. Relays, fault recorders, power quality meters, phasor measurement units, and smart meters observe the grid continuously at up to kilohertz sample rates. However, between every sensor and every engineer runs a chain of decisions about what gets sampled; what survives processing; what leaves the device; what the historian keeps; and what anyone can retrieve. For continuous sensors, the measurement chain downsamples data aggressively. A kilohertz observation arrives at the archive as one value every few seconds, and well over 99.99 percent of native samples never survive.

For event driven sensors, even more data is lost forever by the measurement chain. A fault recorder or protective relay watches the waveform through a rolling buffer, saving a few seconds of data only when a preset trigger fires. Every trigger is, therefore, a hypothesis set years in advance about what will matter, while everything else is overwritten. If incipient arcing preceded a conductor failure but never crossed the “right” threshold, the only direct evidence of what the equipment did before the fire was discarded by default, a choice litigation may force the utility to explain.

Many utilities have never traced this measurement chain from end to end. A physical behavior becomes usable evidence only after surviving six stages.

1. A transducer converts the primary quantity into an electrical signal.
2. Conversion turns the signal into digital samples.
3. Processing reduces the samples into reported quantities.
4. Transport carries the reports off the device.
5. Archival policy and technology decide what persists and at what precision.
6. Access determines whether anyone can retrieve the result and at what speed.

The first three stages are usually set by the sensing hardware. The last two stages are where incumbent data infrastructure quietly fails. Legacy historians apply lossy compression that distorts the very signal they were built to keep, and an archive that takes hours to answer a query is, for iterative work, effectively useless: the record exists, but the answer never will. The usable bandwidth of these measurements is set by the narrowest stage. Because the stages often belong to different departments, vendors, and even organizations through procurement decisions made years apart, few utilities can state, for any archived signal, how much of the original behavior survived the journey.

Even when data survives the chain, it arrives scattered across systems that were never designed to work together. A relay trips on a 115 kV feeder at 2:47 AM, and the protection engineer begins assembling what the grid recorded. SCADA confirms the breaker position. The relay’s event recordings carry timestamps from a different time source, so she aligns them by hand. The historian’s tag names do not match SCADA’s point names, so she reaches for a mapping spreadsheet someone no longer with the utility built three years ago. The synchrophasor data sits on a network managed by a different group, and the fault recorder at the older station has no remote communications at all, requiring a truck roll. Two and a half hours in, five systems deep, she has been functioning as human middleware. The expert panel investigating the Iberian blackout reported the same condition at continental scale: the critical data it needed was missing, incomplete, or inconsistent.

AI Changes What Measurement is For

To this point, the case for keeping data has been the traditional one: evidence for investigations, defense in litigation, fewer hours lost as human middleware. AI completes the argument. Economists call it a general purpose technology, an innovation like electricity whose impact spreads across every sector rather than staying confined to the industry that produced it. Stripped of hype, AI offers improved prediction, using information you have to estimate information you do not. Better prediction creates a world in which data should be kept for use cases not yet understood, because historical observations now acquire future value across applications unknown when they were captured.

Organizations typically absorb a general purpose technology in three stages. First they upgrade: an existing task built on prediction, such as load forecasting, becomes faster and more accurate using the data required by the prior process. Then they recast: the question itself changes, and post-fault analysis becomes fault prediction from precursors buried in continuous waveforms. Finally they transform: maintenance schedules and capital priorities reorganize around predicted failure risk rather than the calendar. The upgrade stage runs on data collected for yesterday’s methods, but recasting and transformation often depend on far more data than was initially required. Every measurement collected today for one function may be reused by AI to answer future questions no one has yet thought to ask. While compute can be purchased and talent hired after a question arises, yesterday’s measurements cannot be bought back. A year of history takes a year to accrue. The data clock can be started sooner; it cannot be made to run faster.

Yesterday’s measurements cannot be bought back. A year of history takes a year to accrue. The data clock can be started sooner; it cannot be made to run faster.

The standard utility objection to such data retention is cost, and it has collapsed despite the AI boom’s “chipflation.” A year of continuous kilohertz waveform data from a substation costs less to retain than a single truck roll. The arithmetic is straightforward: six waveform channels sampled at a kilohertz produce roughly 1 terabyte per year compressed, which costs less than $300 to keep in instantly accessible cloud storage and a tenth of that in archival tiers, while a single truck roll runs more than that before the first bolt is turned. At the meter, the numbers are embarrassing; a decade of smart meter data (4 channels, 15 minute intervals) for a residential customer in the most expensive cloud storage tier costs a fraction of a cent before compression. Amazon would never delete a household’s purchase history, because that record is understood to be an asset. Utilities discard the electrical equivalent as a matter of policy or lack the tools to access and work with this data.

The deeper obstacle is how the industry justifies measurement in the first place. The standard sequence is familiar: identify a problem; establish the ROI of solving it; win the approval of regulators, accountants, and executives; and then spend capital on the sensors the solution requires. Under this multi-year procedure, every measurement must be justified by a named application in advance, and data earns retention only when a known use case can defend the expense. The sequence made sense when storage was scarce, but it carries an insidious assumption that an observation has no value beyond the problem that initiated the acquisition. Physical AI and cheap prediction have now made that assumption false.

Capture-First

Thus, the default mentality must be reversed. Retain observations at the highest available native fidelity and preserve the context that gives them meaning. Further, it is irreversible reduction and not retention that must carry the burden of proof. Let’s call this “capture-first.”

Note that this is not “store everything and sort it out later.” A measured voltage of zero has significantly different implications when the power line was or was not de-energized. Measurements without the context of the physical system from which they were taken (timing, configuration, and lineage) accumulate bytes but not evidence. This is why going from a legacy historian with limited scalability and storage to a data lake is an upgrade but not a solution. What the solution actually requires is an engineering story of its own, and the subject of a follow-up piece. It spans continuous capture, contextualization, tiered retention, retrieval at working speed, and security appropriate for such data. Triggers survive too, but only to mark events and not determine whether the data exists.

The same history explains why no unifying platform has emerged on its own. Every measurement silo was procured to solve one problem, so no silo was ever asked to work with the others, and no application’s value ever depended on unification. Physical AI is the first that does. Learning across a fleet requires all of the fleet’s evidence in one place, contextualized and retrievable at working speed, and that requirement finally gives the industry a reason to build a complete memory of the grid rather than a fragmented one.

The good news for the industry is that the capital has already been spent and the hard part is already done; sensors of all types and sample rates are embedded in virtually every smart asset deployed on the grid, a process that started at the turn of the century, and a single vendor has embedded advanced measurement into more than half a million smart assets on the North American transmission system alone. The vast majority of that sensing is either turned off or stranded by the communications and retention policies downstream of it, so the immediate opportunity is harvesting what sunk capital already measures. Most everything else is a software problem, and those can be solved rapidly, assuming legacy accounting rules don’t stand in the way.

One question reveals where any utility stands against this principle: how long would it take to retrieve a month of synchrophasor data at full resolution, time-aligned with SCADA, relay events, and weather, for one transmission asset? If the answer is measured in weeks, or in meetings, the organization holds records rather than memory.

If the answer is measured in weeks, or in meetings, the organization holds records rather than memory.

Mainstream generative AI inherited a memory the digital world accumulated by accident. Physical AI will inherit only what utilities choose to remember. The best time to start was yesterday. The second-best time is now.