Intelligence per Watt: Measuring Local AI Efficiency

At our latest Snorkel AI Reading Group, Jon Saad-Falcon and Avanika Narayan (Stanford University, Hazy Research) presented Intelligence per Watt (IPW), a metric that divides task accuracy by power draw to compare models and hardware. The study runs 20+ local language models on 1 million real-world chat and reasoning queries, measuring accuracy, energy, latency, and power. It asks how much inference demand can move from data centers to the laptops and workstations people already own. They also previewed Joules to GDP, a companion study that measures the dollar value and expert time AI systems produce per joule of energy, across 220 tasks in 44 occupations.

Transcript

Lightly edited for readability.

Why inference demand is exploding

The core idea of our intelligence per watt project was to measure the intelligence efficiency of local devices as well as cloud intelligence, and to give people a sense of how much we can lift and shift workloads from the cloud to the edge.

Why do we care? It’s no surprise to anyone that demand for inference is absolutely exploding. Google showed a plot at its latest I/O conference with demand growing more than 10x year over year. On top of that, the infrastructure projected to support this demand is fundamentally more than the energy we have available to power these devices.

If you double click into what that demand looks like, about 77% of it is for very practical, information-seeking tasks, the kind that can be done with smaller or local models. Local inference, meaning things that run on your laptop on models under 20 billion parameters, offers a nice surface area to lift and shift these tasks toward.

The intelligence-per-watt metric

As local language models and local accelerators improved, we wanted to understand the role local inference can play in moving demand away from data centers and toward local hardware. To help us understand that, we proposed a metric called intelligence per watt.

It quantifies how many of today’s language models can handle the chat, reasoning, agentic, and coding tasks people care about, against how efficiently they solve them. In other words, how much useful compute, in the form of accurate inference per watt, can run on these local accelerators with these local language models.

Study setup: models, accelerators, and workloads

We wanted to cast a wide net and see how well local AI fares across models, hardware, workloads, and evaluation metrics. We tested 20+ state-of-the-art local language models across the Qwen family, the GPT-OSS family, Gemma, and IBM Granite, both dense and MoE, ranging from 1 billion to 200 billion parameters. We ran them on 10 local accelerators across 2023 to 2025, spanning Nvidia, AMD, SambaNova, and Apple, so we could see the frontier of intelligence on each.

We ran a million-plus queries: ChatGPT-type chat queries, general reasoning tasks, and standardized benchmarks for professional work such as SuperGPQA and MMLU-Pro. We analyzed more than accuracy. We also measured the energy, latency, and compute required to serve these models on local accelerators.

Results: local models are capable and improving quickly

The first thing we found is that local LMs are incredibly capable and improving rather quickly. Over just two years, from 2023 to 2025, we saw a 3x improvement in accuracy.

On intelligence efficiency, meaning how much intelligence they deliver per watt, we saw about a 2.3x improvement each year, or 5.3x total over two years. That gain is driven primarily by better language models, as opposed to better hardware.

Intelligence per joule: hardware takes the lead

When we break it down by intelligence per joule, intelligence delivered per unit of energy rather than per unit of power, the gains are even more dramatic. We found an 18x improvement over just 16 months, and this was driven primarily by better hardware.

Accelerators now deliver more memory and more compute, and new quantization techniques have moved from FP16 down to mixed-precision formats such as FP4. Memory techniques pushed by Apple and Nvidia also let you pack larger models onto denser hardware. Think of the DGX Spark series, the Mac minis, and the M5 Pro. These have enabled local deployments that simply weren’t possible before.

Routing between local and cloud models

In deployment, the simplest way to capture these gains is to route more intelligently between local models and cloud models. Even a routing system that is only 80% accurate can reduce energy consumption by about 80%, compute consumption by upwards of 77%, and cost by upwards of 73%.

You see these gains across all three resources that matter to inference deployments and inference providers.

Where local accelerators still lag

Local accelerators aren’t perfect. They often serve a single user, so they run at lower batch sizes than cloud deployments, where amortization across many users and larger racks lets you deploy larger models. As a result, local accelerators still lag enterprise chips on both IPW and IPJ.

In a head-to-head comparison between an Apple M4 Max and a B200, the M4 Max delivers about 2x to 3x less intelligence per watt and per joule. Bespoke inference accelerators such as the SambaNova SN40L open an even larger gap, particularly on intelligence per joule, because they are optimized for that setting. Even so, there is a big opportunity as we optimize for local models and local architectures.

Collaborators and what comes next

This work wouldn’t have been possible without collaborators at NVIDIA, Google, Apple, AMD, OpenRouter, and SambaNova, whose resources and advice helped us build out the study over the past year. Over the past several months we’ve been expanding the analysis, pushing toward an era in which we care about capabilities alongside the efficiency of these systems.

The most obvious next step is better inference serving engines, both for local models and for hybrid systems that use local and cloud compute together. There is a huge amount of work to do on model architectures, on kernels that deploy more energy-efficient inference, and on software and hardware co-design. We also want more granular analyses across tasks, hardware, and deployment settings, to learn which components in accelerators and models drive the greatest efficiency gains.

Most recently, we studied how intelligence per watt connects to GDP: the full pipeline from joules of energy spent in language model systems to the economic gains from deploying them. We released it today, and we’re excited to see how people think about it and how we can expand it.

Q&A

Q: How are you defining intelligence?

We look at the capabilities of the language models on a specific set of tasks people care about. Intelligence can look different for coding than for agentic or scientific tasks, so we picked benchmarks covering chat, reasoning, agentic, and coding work, which were relevant when we started and still are.

A listener noted that this is still mostly accuracy, alongside cost and latency, and welcomed the push to look beyond accuracy. Accuracy came first because for a long time the capabilities weren’t there, so it made no sense to talk about efficiency. Now the capabilities are there across coding, agentic, chat, and reasoning tasks, and we want to deploy at scale. For the first time, dollar cost, total energy spent, and average power during deployment all begin to make sense. We wanted to see how the Pareto frontier shifts when you consider those efficiency metrics.

Q: Will inference move off the cloud, given the efficiency advantages of large batch sizes in data centers?

Efficiency is going to grow, period, because demand is immense and deployments are varied. More realistically, the world becomes local by default but hybrid by design. The vast majority of queries get handled locally, on a laptop, a workstation, or an on-prem deployment closer to the office or home. The remaining 5% or less, the frontier queries in coding, mathematics, and science, get pushed to data centers. There will be immense demand for both, but increasingly more inference shifts to local deployments.

Q: Have hardware gains versus model gains shifted since an earlier version of the paper?

We’ve updated the graph, which originally covered 2023 and 2024, through Q4 of 2026. The trend of rising intelligence per joule is consistent for both local and cloud models. For local settings, we now see about a 30x increase when you combine improvements from language models and accelerators. Most of that is driven by the accelerators themselves, including newer generations of the DGX Spark, the Apple M6, and AMD’s Strix Halo, which run larger and larger models increasingly fast. Asked whether hardware gains were ahead of model gains, the answer was yes.

Q: If an enterprise already pays for data center GPUs, should it route simple queries to a smaller model in the same data center?

It depends on how much traffic they have. With a discrete amount of traffic and GPUs already paid for, deploying there may make sense. But many companies have so much traffic that any additional inference capacity would be used. They are solving a resource allocation problem: queries that don’t need bleeding-edge resources, such as Nvidia B200s or expensive rack-scale computing, move to cheaper compute, while the most intensive queries that truly need frontier capabilities go to cloud deployments. We’ve talked a lot with folks at Google Cloud, who have different TPUs for both cloud and local deployments and need to make this distribution decision in real time.

Q: Can power be measured beyond the chip, and is that different for local versus cloud?

For local deployments, measuring power is fairly straightforward: you read the power draw from the wall the device is plugged into. For the data center deployments we compared against, we kept to node-level setups, such as 8x H200s or 4x B200s or fewer. Rack-scale deployments add auxiliary infrastructure around networking and cooling, which makes the equations much harder. You have to assume how much power is drawn or add empirical measurements, so for this study we stayed with the more vanilla deployments that represent the majority of workloads today.

Q: What is the Joules to GDP work, and how does it fit the roadmap?

The original study connected energy to downstream accuracy and intelligence. Joules to GDP takes it one step further. People are using AI to generate economic value across professions, and AI is taking more and more energy within the economy, in the US and worldwide. We wanted to connect the energy going into AI systems to the economic value coming out. We curated tasks across finance, engineering, healthcare administration, consulting, and more, and connect the energy spent to the dollar value generated. If I run an H200 on engineering tasks, how much value do I generate? What about accounting tasks, or an M5 Max, an AMD Ryzen, or an AMD Instinct? Broadly, we want to know which combinations of models, hardware, and tasks generate the most economic value per joule.

Q: Do the efficiency gains hold up for long-running tasks with accumulating context?

Yes, and especially as tasks get longer. A local deployment handles queries for a single user, so you don’t have to keep resources hot and spread them across users. Data centers emphasize handling a task quickly and moving on. Long tasks that wait on tool calls, search results, and code execution become costly there, because you have to store an increasingly large KV cache and keep it hot, which limits how many users you can serve. Locally you serve one person and can use the resource at will. The new arXiv version includes experiments for this exact setting in the appendix.

Q: What retrospective will you write in two years?

Two or three years from now, it will be increasingly obvious that local AI is the first choice for most consumers and users, in professional and everyday settings alike. If capability trends hold and local LMs keep trailing frontier deployments at a reasonable clip, intelligence will keep compressing. The frontier capability of this January was Claude Opus 4.6, and local open-source deployments now deliver those capabilities only nine or ten months later. If that continues, local intelligence will eventually cover 90 to 95% of the tasks people care about, and everybody can have frontier intelligence in a box on their desk. Data center inference will still exist, but it will focus on the truly frontier tasks in mathematics, science, and coding.

FAQ

Q: What is intelligence per watt (IPW)?

IPW is task accuracy per unit of power. It lets researchers compare how much useful intelligence different model and hardware pairs deliver for the power they draw. The paper uses it to ask whether local models on devices like laptops can take over some demand from cloud data centers.

Q: How is intelligence per joule different from intelligence per watt?

Intelligence per joule measures intelligence delivered per unit of energy rather than per unit of power. In the talk, the speakers reported that these gains were even larger than the per-watt gains, 18x over 16 months, and driven primarily by better hardware.

Q: How many models and queries did the study cover?

The study evaluated 20+ local language models on 1 million real-world, single-turn chat and reasoning queries. For each, the authors measured accuracy, energy, latency, and power. The hardware spanned Nvidia, AMD, SambaNova, and Apple accelerators.

Q: How fast is local AI improving?

The paper reports that IPW improved 5.3x from 2023 to 2025. It also finds that the share of queries local models can serve rose from 23.2% to 71.3%. In the talk, the speakers attributed the per-watt gain mainly to better language models rather than better hardware.

Q: Can local AI replace cloud AI?

Not entirely. The speakers expect the world to become local by default but hybrid by design, with the most demanding frontier queries in coding, mathematics, and science still going to data centers. Local accelerators also trail cloud chips on efficiency: the paper reports at least 1.4x lower IPW than cloud accelerators running the same models.

Q: What is Joules to GDP?

Joules to GDP extends the intelligence-per-watt idea from accuracy to economic value. It defines Dollar Value per Joule (DPJ) and Time Saved per Joule (TPJ), then applies them to 220 GDPval tasks across 44 occupations, five AI systems, and four accelerators. DPJ and TPJ vary by factors of 8.94 and 3.75 across occupation categories, so the return on energy depends on the task and the full system, not the model alone. The work was a poster at the NeurIPS 2026 ML for Systems workshop, and Jon and Avanika are co-authors. Read the poster page on OpenReview.

Speaker bio

Jon Saad-Falcon and Avanika Narayan are PhD students in Computer Science at Stanford University and members of Hazy Research. Their work includes the Intelligence per Watt paper and the Joules to GDP study.

Links

• Paper: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI (arXiv)
• Project site: intelligence-per-watt.ai
• Snorkel research page: Intelligence per Watt
• Related post: Intelligence per watt: A new metric for AI’s future
• Reading Group: snorkel.ai/reading-group
• Code: HazyResearch/intelligence-per-watt on GitHub
• Video: Intelligence per Watt Reading Group recording on YouTube
• Joules to GDP: Joules to GDP poster (OpenReview)
• Also from Hazy Research: Understanding Olmix: A Framework for Data Mixing Throughout Language Model Development