Agentic AI Is Increasing CPU Demand Alongside GPUs

Agentic AI changes the compute mix not by replacing large language models (LLMs), but by putting a decision-making layer around them. Models still depend on GPUs or other accelerators for neural-network calculations; agents add the branching logic, orchestration and repeated tool calls that make CPUs do more than host the accelerators. Industry estimates cited by semiconductor executives put the CPU-to-GPU ratio for some agentic workloads as high as 1:1, compared with roughly 1:4 for typical LLM systems.

That shift is the clearest infrastructure consequence among several AI approaches now gaining attention. Reasoning models and mixture-of-experts (MoE) architectures alter how models spend compute, while reservoir computing targets a narrower set of time-based tasks. Most still rely on GPUs for their core calculations. Agents stand apart because they create more work around those calculations—and often send a single user request through a chain of model calls and actions.

From model output to action

Traditional inference is comparatively direct: a system passes input to a model, then returns the output. GPUs and dedicated neural processing units (NPUs) suit the dense mathematical operations involved. An agent uses those same models, but treats their responses as inputs to another process. It evaluates what came back, selects a next step and may call a tool or another model before deciding what to do next.

That process relies on conditional logic—if a result meets one condition, take one path; otherwise, take another. CPUs are designed to handle this kind of branching more effectively than GPUs, which excel at parallel neural-network operations. The agent therefore doesn’t displace accelerator work. It adds a CPU-heavy layer beside it.

Ed Doe, CEO and co-founder of Delos Data, said research suggests agentic inference could eventually exceed other inference workloads combined. That remains a projection, not a settled measure of demand, but it captures why hardware planners are revisiting the balance between CPUs and accelerators.

An agent is an orchestration layer, not a new kind of neural model. It needs training, but its role is to direct models and other tools toward a goal. A user might ask an agent to arrange a trip within a budget, specifying lodging, transportation and activities. The agent can interpret the request, call relevant application programming interfaces (APIs), inspect the results and continue until it has completed the sequence.

Rob Knoth, product management director in Cadence’s digital and signoff group, compared the model to delegating a task to a team: give people the outcome to achieve rather than micromanaging each step. The analogy points to the intended benefit—less step-by-step supervision—but also hints at the engineering challenge. The system must keep track of progress, decide when to act and recover when a tool returns an unexpected result.

Arm’s Satadal Bhattacharjee said agents have gained momentum since the previous year and described them as systems that can act on an outline by calling APIs. Their ability to sequence actions can also be shaped with programming, including JSON scripts, he said. This combination of model output and explicit workflow logic makes agents more than a chat interface, but it also expands the software surface that teams must test and maintain.

Why CPUs are back in the discussion

In conventional LLM servers, CPUs often prepare data, manage communication and coordinate accelerator devices. That host role can leave one CPU supporting several GPUs, producing the approximate 1:4 ratio cited for typical LLM configurations. With agents, CPU cores spend more time executing the workflow itself: evaluating outputs, branching, managing API calls and tracking the sequence of tasks.

Ron Lowman, strategic marketing manager for IP at Synopsys, said the earlier design question was how many AI accelerators could be connected to each CPU. Agentic workloads raise the CPU-side demand, changing that calculation. Bhattacharjee likewise described a rise in CPU demand as teams seek more compute for running agents efficiently. Yong Meng Sua, CTO at QCI, said interest in agents over the previous 18 months had contributed to CPUs becoming expensive.

Those comments describe market pressure, not a universal configuration. The actual ratio will depend on the system and the agent workload. A workflow with frequent decisions and tool interactions could put more load on CPUs than a mostly linear request, while a model-heavy task can remain dominated by accelerator calculations. The key change is that CPU capacity can no longer be treated as a light supporting resource by default.

Doe’s description of the CPU’s return is apt: it’s doing a different job in the data center than it did when it mostly prepared work for GPUs. More CPUs may raise system cost, and the agent’s many intermediate calls can increase total compute per request. A request that looks like one interaction to a user may trigger a series of model inferences and actions behind the scenes. Capacity planning needs to account for that chain, not just the final answer.

Reasoning adds more accelerator work

Agents are only part of the change in model behavior. Early LLMs were trained to predict a likely next token in a sequence. That objective can produce fluent text without ensuring that the answer is factually sound. Retrieval-augmented generation (RAG) addresses some of this weakness by giving a model access to information stored in embeddings, so its response can draw on retrieved material rather than relying only on learned patterns.

Reasoning models take another approach: they perform additional inference before producing a response. Depending on the system, they may also expose some account of how they reached an answer. Bhattacharjee described reasoning as combining subject knowledge, conversation context and information about the person asking, in order to produce a response that covers more than the literal wording of a prompt.

That added work is still neural-network inference. Generating extra tokens and performing the associated calculations are generally better suited to GPUs or other AI accelerators than CPUs, Bhattacharjee said. Feedback on faulty reasoning steps can also be used during training to improve later performance. OpenAI’s O-series models and DeepSeek-R1 were identified as commercially deployed examples.

Reasoning and agentic behavior can therefore compound rather than compete for resources. An agent may call a reasoning model, examine its response, then choose another action. The model’s additional inference raises accelerator work, while the agent’s decisions and tool orchestration add CPU work. How much of each is required depends on the task and system design.

MoE models route work to specialists

MoE architectures aim to increase a model’s range without activating every parameter for every token. Instead of one dense feed-forward network (FFN) at a layer, an MoE can use multiple FFNs, known as experts, along with a learned router. Each expert develops different strengths during joint training; developers don’t usually assign each one a fixed subject by hand.

At inference time, a gating function examines a token and directs it to one or a small number of experts. The router learns from training which experts tend to handle different tokens effectively. The model can thus contain many experts while activating only a subset for each token. The router itself is generally part of the model and runs on the accelerator, Bhattacharjee said.

Synopsys product management director Priyank Shukla contrasted this arrangement with scaling a single dense model by continually adding parameters. In an MoE, expert networks can specialize in areas such as legal material or mathematics, while an earlier stage parses the prompt and establishes its context. The router then directs relevant tokens to appropriate experts. Those examples describe the architecture’s possible specialization; they aren’t a claim that every MoE uses those exact divisions.

Context is carried forward in a key-value cache (KV cache), which grows as inference proceeds and informs later processing. Shukla pointed to transmitting that cache from the initial model to the relevant experts as part of the system’s workload. That movement matters: sparse activation can reduce the computation needed for each token, but it doesn’t make data movement free.

MoEs can have more total parameters than a conventional dense model because they contain several experts. Their advantage is that not all those parameters need to be fetched and computed for every token. Bhattacharjee said this sparse activation can raise total model capacity without requiring every parameter to participate in each inference operation.

The efficiency picture is less straightforward than parameter counts suggest. Routing, memory access and communication among devices add overhead, and the power outcome depends on the complete system. An MoE may perform less computation per token than a dense model with a comparable total parameter count, but deployment efficiency will also depend on where experts and cache data reside and how quickly the system can move them.

Small models make a different tradeoff

Small language models (SLMs) and MoEs both seek to avoid running an enormous general-purpose model for every job, but they do so differently. An SLM is a self-contained model trained for a narrower domain. It can process a full string of tokens while using fewer resources than a large general model, but its scope is limited by its specialization.

An MoE, by contrast, keeps a broader set of capabilities within one model and selects experts for portions of the input. Individual experts handle certain tokens, rather than independently processing an entire prompt. The architecture may have a higher total parameter count than a dense model, yet only selected experts are active at a time. That makes MoE a route to broader coverage with sparse computation, while an SLM favors a smaller, focused model that can run on its own.

Fig. 1: A specialized LLM can be condensed into an SLM. An MoE instead provides multiple experts for key parts of the model and uses a router to choose which experts handle the next token. Source: Bryon Moyer/Semiconductor Engineering

Reservoir computing targets time-based inputs

Reservoir computing is a more specialized approach for sequential data. Most of the model is fixed and random; training adjusts only the final layer, which maps the reservoir’s output to the desired answer. That makes training simpler than adjusting all the weights in a typical neural model, though it also means the technique has a narrower remit.

Bhattacharjee described it as a fit for dynamic temporal inputs where low power and simple training matter. Because most of the structure doesn’t need trained weights, parts of it can be hard-coded into silicon. Only the output layer’s weights need to be fetched during operation, which can reduce energy use.

Potential applications include speech and audio recognition, sensor-data processing, motor control in robots and time-series prediction. QCI is preparing a photonic reservoir-computing system for launch. Prajnesh Kumar, the company’s quantum technology lead, said photonics is a promising implementation because of its bandwidth.

Kumar also described wavelength-division multiplexing (WDM), which can carry separate tasks on different wavelengths. If the tasks use the same kernel, multiple operations can run in parallel without interfering, he said. The attraction is specialized processing that combines speed with low power; the practical value will depend on the applications and system implementation.

A mixed compute future

No single architecture is taking over every AI workload. CNNs remain useful for image-oriented tasks, while LLMs continue to support a broad set of applications. MoEs are gaining attention as a way to expand model capacity through specialist networks; reasoning models add inference steps; reservoir computing serves particular temporal workloads. Each approach shifts a different part of the compute problem.

For infrastructure teams, the immediate distinction is that most of these model approaches still put their heaviest calculations on GPUs or other accelerators. Agentic systems retain that demand and add more CPU work to coordinate decisions and actions. A useful deployment plan will need to consider both sides: accelerator capacity for model inference, and sufficient CPU resources for the branching, tool use and sequencing around it.

The balance will vary with the workload, and claims about future ratios should be treated as estimates rather than fixed design rules. What’s already clear is that counting GPUs alone gives an incomplete picture of agent capacity. As agents take on longer task sequences, CPU availability, memory movement and the cost of repeated inference will all deserve closer attention.

Reference
1. Michael te Vrugt, “An introduction to reservoir computing,” book chapter to appear in: Artificial Intelligence and Intelligent Matter, Springer, Cham, arXiv:2412.13212v1

Related Article
Agentic AI Is Changing Data Center Architectures
Standalone GPUs are being replaced by heterogeneous SoCs and chiplets that combine CPUs, GPUs, and NPUs to eliminate memory bottlenecks, reduce latency, and boost efficiency.

The post Agentic AI Alters The CPU-GPU Balance appeared first on Semiconductor Engineering.