2024 has been a big year for on-device AI in consumer electronics. Both Microsoft and Apple took swings with their respective operating systems, with Microsoft debuting its “Copilot+ PC” branding for AI-capable laptops and Apple releasing Apple Intelligence.
These early examples offered mixed results. Some features, like real-time translations and on-device speech-to-text, can be useful. Others, like Microsoft’s Windows Recall, have yet to prove themselves.
All of this hype for AI has important implications for the new year. 2025 looks set to become the year when mainstream developers make their attempts to add on-device AI to their Windows apps, and that means you’re going to want to pay even closer attention to the AI performance of modern Windows laptops before you buy a new one.
I spoke with two experts in AI research and testing to probe their brains for insights on how Windows on-device AI will grow in 2025.
Big gains are coming for NPUs
If you’re curious about Windows laptops’ AI performance, you’ll likely end up comparing the “TOPS” promised by each laptop model. TOPS (“Trillions of Operations Per Second”) is a measurement of an NPU’s ability to perform matrix multiplications for on-device AI tasks. (Learn more about what an NPU is and why it matters for AI.)
2024 saw big gains in the TOPS performance available from Windows laptops. To qualify for Microsoft’s “Copilot PC+” branding, a Windows laptop must have at least 40 TOPS of NPU performance. For reference, Qualcomm’s first Copilot+ PCs quoted about 45 TOPS — that’s a four-fold uplift over Intel’s “Meteor Lake” Core Ultra 7 165H, which had only quoted 11 TOPS of NPU performance.
Microsoft / Samsung
“I think Qualcomm really woke everyone up,” said Karl Freund, founder and principal analyst at Cambrian AI Research. Freund has noted that AMD and Intel have been quick to respond with their own chips, which delivered a similar uplift.
By the end of 2024, shoppers looking for a premium Windows laptop — like a Microsoft Surface, Asus ProArt, or Dell XPS — can expect a roughly three- or four-fold increase in NPU performance compared to similarly premium laptops that were available at the end of 2023. That’s a huge bump up. But will that trend continue into 2025?
Ryan Shrout, president of performance testing lab Signal65, thinks it could. “It wouldn’t surprise me if we see double again, and triple again wouldn’t surprise me.” However, he expects those eventual gains to be weighted more towards the end of next year. “My guess is it will be late 2025, and probably into 2026, when we see the most significant NPU improvements.”
TOPS may not stay on top for long
A potential two- to three-fold improvement for on-device AI performance is significant. However, Freund and Shrout warned it’s best not to give too much credence to the TOPS performances that chip makers quote.
“TOPS really stands for ‘Terribly Overused Performance Stat,’” said Freund. “It doesn’t have a lot of value.”
Shrout agreed, comparing TOPS to the TFLOPS figures that AMD and Nvidia often quote when marketing GPUs. These numbers, which point to a GPU’s maximum possible computation speed, offer surprisingly little insight into actual real-world performance.
Real-world AI performance is currently a bit of a wild card, in part because Windows has yet to coalesce around a single API for tapping an NPU’s AI capabilities. That’s a problem for owners of Copilot+ laptops that lack a Qualcomm chip inside.
Mark Hachman / IDG
Though AMD and Intel have chips that qualify for Copilot+ branding, Qualcomm has enjoyed a favored status so far. Qualcomm machines were the first to receive support for Windows Recall and several popular apps, like Blender and Affinity Photo, which were recently announced to only work on Qualcomm Snapdragon X hardware.
That should change through 2025, however, as Microsoft rallies support for its low-level machine learning API (DirectML) and the Windows Copilot Runtime, which includes several task-specific AI APIs (some of which have yet to be released). For now, it’s clear that Copilot+ PCs leave a lot to be desired and have lots of room for growth coming up.
“I think Microsoft will have this solved in 2025,” said Shrout. “Once application developers attach to DirectML, like they did with DirectX, it will be a solved problem. And I don’t think it will be a problem for long.” Shrout compared it to the early days of 3D on the PC, which initially saw competing APIs but eventually consolidated around the leaders, with Microsoft DirectX becoming the most popular option.
Proving the case for Windows AI
Better NPUs and a unified API that makes it easier for Windows application developers to actually use an NPU’s full performance are both important steps forward, but they don’t necessarily guarantee that on-device AI will become commonplace.
That’s because developers still have the option to turn towards companies like OpenAI and Anthropic, who make their AI models and services available to any device with internet access. And their AI models are still more capable than on-device AI models, able to do more and generate those results far more quickly.
However, those AI models hosted in the cloud have a major downside that will become more relevant in 2025 — price.
“The fact we can have small language models run on an NPU continuously in the background to monitor what’s happening, that’s something the cloud can’t do, or at least would be much more expensive from an infrastructure standpoint,” said Shrout.
OpenAI’s recent release of ChatGPT Pro, a new premium tier for power users, seems to drive this point home. ChatGPT Pro provides unlimited access to the company’s new o1 model and priority access to the Sora video generator, but it’s priced at $200 per month. The per-token price paid by app developers to make o1 available to users is similarly steep.
Users and developers who turn to a Windows laptop’s on-device NPU, on the other hand, can essentially use it whenever they want for free. That’s arguably going to be the final brick laid in the road towards on-device AI. Developers and users will have both the tools and incentives to rely on a Windows laptop’s NPU whenever possible to cut costs.
It remains to be seen how quickly the shift towards on-device AI will happen, and to what extent it will proliferate through Windows’ software ecosystem, but it’s likely that 2025 will be a huge turning point.
“I think Qualcomm had it right five years ago when they said AI would move on-device. At first, I was skeptical. But now I’ve become a believer,” said Freund.
Further reading: Free AI tools that run locally on your PC
2024 has been a big year for on-device AI in consumer electronics. Both Microsoft and Apple took swings with their respective operating systems, with Microsoft debuting its “Copilot+ PC” branding for AI-capable laptops and Apple releasing Apple Intelligence.
These early examples offered mixed results. Some features, like real-time translations and on-device speech-to-text, can be useful. Others, like Microsoft’s Windows Recall, have yet to prove themselves.
All of this hype for AI has important implications for the new year. 2025 looks set to become the year when mainstream developers make their attempts to add on-device AI to their Windows apps, and that means you’re going to want to pay even closer attention to the AI performance of modern Windows laptops before you buy a new one.
I spoke with two experts in AI research and testing to probe their brains for insights on how Windows on-device AI will grow in 2025.
Big gains are coming for NPUs
If you’re curious about Windows laptops’ AI performance, you’ll likely end up comparing the “TOPS” promised by each laptop model. TOPS (“Trillions of Operations Per Second”) is a measurement of an NPU’s ability to perform matrix multiplications for on-device AI tasks. (Learn more about what an NPU is and why it matters for AI.)
2024 saw big gains in the TOPS performance available from Windows laptops. To qualify for Microsoft’s “Copilot PC+” branding, a Windows laptop must have at least 40 TOPS of NPU performance. For reference, Qualcomm’s first Copilot+ PCs quoted about 45 TOPS — that’s a four-fold uplift over Intel’s “Meteor Lake” Core Ultra 7 165H, which had only quoted 11 TOPS of NPU performance.
Windows Recall on a Copilot+ laptop by Samsung.Microsoft / Samsung
“I think Qualcomm really woke everyone up,” said Karl Freund, founder and principal analyst at Cambrian AI Research. Freund has noted that AMD and Intel have been quick to respond with their own chips, which delivered a similar uplift.
By the end of 2024, shoppers looking for a premium Windows laptop — like a Microsoft Surface, Asus ProArt, or Dell XPS — can expect a roughly three- or four-fold increase in NPU performance compared to similarly premium laptops that were available at the end of 2023. That’s a huge bump up. But will that trend continue into 2025?
Ryan Shrout, president of performance testing lab Signal65, thinks it could. “It wouldn’t surprise me if we see double again, and triple again wouldn’t surprise me.” However, he expects those eventual gains to be weighted more towards the end of next year. “My guess is it will be late 2025, and probably into 2026, when we see the most significant NPU improvements.”
TOPS may not stay on top for long
A potential two- to three-fold improvement for on-device AI performance is significant. However, Freund and Shrout warned it’s best not to give too much credence to the TOPS performances that chip makers quote.
“TOPS really stands for ‘Terribly Overused Performance Stat,’” said Freund. “It doesn’t have a lot of value.”
Shrout agreed, comparing TOPS to the TFLOPS figures that AMD and Nvidia often quote when marketing GPUs. These numbers, which point to a GPU’s maximum possible computation speed, offer surprisingly little insight into actual real-world performance.
Real-world AI performance is currently a bit of a wild card, in part because Windows has yet to coalesce around a single API for tapping an NPU’s AI capabilities. That’s a problem for owners of Copilot+ laptops that lack a Qualcomm chip inside.
Da Vinci Resolve running on a Qualcomm Snapdragon X Elite laptop.Mark Hachman / IDG
Though AMD and Intel have chips that qualify for Copilot+ branding, Qualcomm has enjoyed a favored status so far. Qualcomm machines were the first to receive support for Windows Recall and several popular apps, like Blender and Affinity Photo, which were recently announced to only work on Qualcomm Snapdragon X hardware.
That should change through 2025, however, as Microsoft rallies support for its low-level machine learning API (DirectML) and the Windows Copilot Runtime, which includes several task-specific AI APIs (some of which have yet to be released). For now, it’s clear that Copilot+ PCs leave a lot to be desired and have lots of room for growth coming up.
“I think Microsoft will have this solved in 2025,” said Shrout. “Once application developers attach to DirectML, like they did with DirectX, it will be a solved problem. And I don’t think it will be a problem for long.” Shrout compared it to the early days of 3D on the PC, which initially saw competing APIs but eventually consolidated around the leaders, with Microsoft DirectX becoming the most popular option.
Proving the case for Windows AI
Better NPUs and a unified API that makes it easier for Windows application developers to actually use an NPU’s full performance are both important steps forward, but they don’t necessarily guarantee that on-device AI will become commonplace.
That’s because developers still have the option to turn towards companies like OpenAI and Anthropic, who make their AI models and services available to any device with internet access. And their AI models are still more capable than on-device AI models, able to do more and generate those results far more quickly.
However, those AI models hosted in the cloud have a major downside that will become more relevant in 2025 — price.
“The fact we can have small language models run on an NPU continuously in the background to monitor what’s happening, that’s something the cloud can’t do, or at least would be much more expensive from an infrastructure standpoint,” said Shrout.
CKA / Shutterstock
OpenAI’s recent release of ChatGPT Pro, a new premium tier for power users, seems to drive this point home. ChatGPT Pro provides unlimited access to the company’s new o1 model and priority access to the Sora video generator, but it’s priced at $200 per month. The per-token price paid by app developers to make o1 available to users is similarly steep.
Users and developers who turn to a Windows laptop’s on-device NPU, on the other hand, can essentially use it whenever they want for free. That’s arguably going to be the final brick laid in the road towards on-device AI. Developers and users will have both the tools and incentives to rely on a Windows laptop’s NPU whenever possible to cut costs.
It remains to be seen how quickly the shift towards on-device AI will happen, and to what extent it will proliferate through Windows’ software ecosystem, but it’s likely that 2025 will be a huge turning point.
“I think Qualcomm had it right five years ago when they said AI would move on-device. At first, I was skeptical. But now I’ve become a believer,” said Freund.
Further reading: Free AI tools that run locally on your PC Read More