Agentic AI Raises CPU Demand as Model Architectures Diversify

Agents change the shape of AI compute

AI systems are no longer just models generating outputs from prompts. Agentic systems use models as part of a longer process: they interpret a request, choose a next step, call tools or services, inspect the result and decide what to do next. That orchestration adds work suited to CPUs, pushing data center designs toward a more balanced mix of CPUs and accelerators.

For conventional large language model (LLM) inference, CPUs often prepare inputs, manage communication and coordinate attached GPUs. That host role can let one CPU serve several GPUs. The source puts a typical CPU-to-GPU ratio for LLM systems at about 1:4; some industry estimates suggest agentic workloads could move it toward 1:1. Those figures describe a possible shift, not a universal system configuration.

The change matters because agents generally issue multiple model queries while carrying out a task. A user may ask an agent to arrange a trip within a budget, for example. The agent could consult services for lodging, transport and activities, then use each response to determine what to do next. Its work includes model inference, but also branching logic, sequencing and calls to application programming interfaces (APIs).

That mix puts more pressure on CPUs. GPUs remain central to the neural network calculations behind many models, but they are less suited to the conditional, step-by-step control flow that agents need. The result is not a replacement of GPUs by CPUs. It is a larger CPU role alongside continuing demand for accelerators.

“Agentic inference” could eventually outweigh other inference workloads, said Ed Doe, CEO and co-founder of Delos Data, describing research in the field. The claim remains a forecast. What’s already clear from the workload design is that one request to an agent can trigger a chain of queries and actions, making it more compute-intensive than a single model response.

From image and sequence models to LLMs

The shift builds on several generations of AI architecture. Convolutional neural networks (CNNs) became useful for analyzing image data, while recurrent neural networks (RNNs) handled sequences such as sound and language. RNNs can process incoming audio frames as they arrive, which can help with real-time response, and they generally need fewer resources than LLMs. Their limitation is that they don’t retain context in the same way.

LLMs use tokens to represent text, and the same basic approach can represent other kinds of input. Their ability to work with long context helps them interpret and generate language with a wider view of what came before. But early LLMs were trained to predict a likely next token. A plausible continuation could be grammatically sound yet factually wrong.

Retrieval-augmented generation (RAG) addresses part of that problem by giving a model access to information retrieved from external sources. The model can then draw on facts relevant to a prompt instead of relying only on patterns absorbed during training. RAG can improve grounding, though it doesn’t by itself guarantee that a response is correct.

These architectures still matter. CNNs continue to serve image tasks, and RNNs have advantages for some streaming inputs. LLMs now underpin many general-purpose systems, while newer approaches—including reasoning models and mixture-of-experts models—alter how models handle inference and capacity. Agents sit above those models as an orchestration layer.

Agents add decisions and actions

An agent isn’t a new kind of neural network that replaces an LLM. It coordinates one or more models and decides what to do with their outputs. A model may interpret a request or suggest an action; the agent can then evaluate the result, call a tool and continue through a sequence of steps.

That distinction has practical consequences. A conventional chatbot can return an answer and stop. An agent is intended to carry out a task with less step-by-step direction from a person. Rob Knoth, product management director in Cadence’s digital and signoff group, compared the idea to delegating a goal to a team: a manager specifies the desired work and checks progress rather than directing every action.

Agent systems also need a way to sequence work. Arm’s Satadal Bhattacharjee said that scripts, including JSON, can be used to tune and arrange an agent’s actions. The precise implementation depends on the system, but the general job is to track state, choose among possible steps and route requests to models or APIs.

That control logic is a natural fit for CPUs. Conditional branches—if a service returns one result, take one action; if it returns another, take a different action—are routine CPU work. GPUs excel at performing many similar numerical operations in parallel, which is why they remain well suited to neural network inference. Agentic systems need both kinds of processing.

“The CPU is back,” Doe said, describing a different role for the processor in the data center. CPU demand has risen as companies seek more compute for agents, Bhattacharjee said. Ron Lowman, strategic marketing manager for IP at Synopsys, likewise described a change from maximizing the number of accelerators attached to each CPU toward accounting for the additional CPU work agents require. Yong Meng Sua, CTO at QCI, also linked the recent rise in CPU prices to interest in agentic AI.

For teams building or buying systems, the implication is a potential change in server balance and cost. A configuration designed around GPU-heavy model inference may need more CPU capacity to handle agent control flow. The actual ratio will vary with the workload, how often agents call tools and how much processing happens between model queries. The source doesn’t give a universal sizing rule.

Reasoning models spend more time on inference

Reasoning models tackle a different limitation: producing an answer without enough intermediate consideration. They use additional inference before responding, generating more tokens and performing more neural network calculations. Some can also provide an account of their reasoning, though the source doesn’t establish that such accounts offer a deterministic record of every internal computation.

Bhattacharjee described reasoning as combining subject knowledge, prior conversational context and information about the person asking. The intended result is a more considered response to a broader problem, rather than a quick answer to the literal wording of a question. Feedback during training can also help improve future performance when a reasoning step proves incorrect.

These extra calculations are still accelerator work. “Thinking involves generating additional tokens and neural-network computation,” Bhattacharjee said, describing why reasoning typically runs on GPUs or other AI accelerators. Reasoning models therefore add to accelerator demand even as agent orchestration increases CPU demand.

OpenAI’s o-series models and DeepSeek-R1 were cited as commercially deployed examples. Their presence shows that reasoning models have moved beyond research demonstrations, but the source provides no comparative benchmarks for their speed, accuracy or operating cost. For practitioners, the key tradeoff is the extra inference needed to produce a more considered response.

Reasoning also doesn’t resolve the broader difficulty of explaining how a trained model reached an answer. Model weights are numerical values refined through training; different starting weights can produce different models that still perform correctly. That makes it difficult to trace a basic model’s calculations in a deterministic way. For systems where people need to establish why an output is trustworthy, additional reasoning steps may help show work, but they don’t make the model’s internal process fully transparent.

MoE models route tokens to specialists

Another response to the growing size of LLMs is the mixture-of-experts (MoE) architecture. Instead of activating every parameter for every token, an MoE model contains multiple expert networks and routes each token to a small selection. The architecture can increase the model’s total capacity while limiting how much of that capacity runs for a given token.

In a typical design, some dense feed-forward network layers are replaced by several separate feed-forward networks, or experts, plus a learned router. The router examines a token and chooses which expert or experts should process it. It could direct a token to every expert, but systems typically select only one or two.

The router learns its choices during training by observing which experts perform well on different tokens. Experts are generally trained jointly rather than assigned manually to fixed subjects. At inference time, the router’s decision determines which parts of the model are active.

Bhattacharjee described a two-part process: an initial phase parses the prompt and builds context, followed by routing to relevant experts. A healthcare question, for example, might be sent to an expert that has learned to handle that topic. Synopsys product management director Priyank Shukla described the broader aim as splitting work across expert models, such as ones suited to legal or mathematical tasks, rather than making one dense model cover everything in the same way.

The model must also preserve context as inference proceeds. A key-value cache (KV cache) stores information from earlier processing so later steps can use it. Shukla noted that this cache must be transmitted from the initial model to the relevant experts. The routing or gating function itself generally runs on the accelerator alongside the model, Bhattacharjee said.

That communication matters. Sparse activation can cut the computation required per token compared with a dense model that has a similar total parameter count. But an MoE model may contain more parameters overall, and moving data between memory, experts and devices adds overhead. Power use and performance therefore depend on system design, including routing, memory access and communication—not just the number of active parameters.

Small models and MoE solve different problems

A small language model (SLM) offers another way to reduce the resources needed for a task. It is a self-contained model trained for a narrower area and can process a full string of tokens. An MoE system instead divides processing among experts that handle selected tokens within that string.

That difference affects capacity and deployment. SLMs are smaller and need fewer resources to run, but their specialization limits the range of material they can handle. MoE models can cover a wider set of topics through their experts. Their total parameter count can exceed that of a standard dense model because certain layers contain several expert networks, yet only a subset of those experts is typically activated for each token.

In practical terms, an SLM can suit a bounded task where a compact, independent model is enough. An MoE model is aimed at broader coverage while avoiding the cost of running every parameter for every token. Neither design is automatically cheaper or more efficient in every deployment: the workload, hardware and data movement all matter.

Fig. 1: An SLM condenses a specialized model, while an MoE model uses a router to select experts for incoming tokens. Source: Bryon Moyer/Semiconductor Engineering

Reservoir computing favors temporal tasks

Reservoir computing takes a different route. It is designed for time-dependent inputs and can be easier to train than a typical neural network because most of the model remains random. Training adjusts the final layer, which maps the reservoir’s output to the desired result.

That structure can make the approach attractive where fast training and low power matter. Since most of the reservoir doesn’t need to change, its contents can be hard-coded into silicon. Only the trained weights in the final layer need to be fetched, reducing the data movement involved in running the model.

Potential uses include speech and audio recognition, sensor processing, robot motor control and time-series prediction. The technique is less prominent than LLMs, but its narrower remit can be an advantage when an application depends on sequences rather than broad language understanding.

Quantum Computing Inc. (QCI) is preparing a photonic reservoir computer for launch, according to the source. QCI’s quantum technology lead, Prajnesh Kumar, said photonics is a strong fit in part because of its bandwidth. He described wavelength-division multiplexing (WDM) as a way to run separate tasks on different wavelengths. If those tasks use the same kernel, multiple operations could run in parallel without interfering with one another.

Kumar pointed to trajectory planning and analysis of brain signals as possible application areas for the company’s approach. These examples remain application possibilities, not evidence of broad deployment. The next question is whether the photonic implementation can translate its bandwidth advantage into useful performance and energy savings on real tasks.

More architectures, different bottlenecks

The emerging picture is a collection of approaches with different strengths. CNNs remain suited to image analysis; RNNs can process streaming sequences with relatively low resource needs; LLMs offer broad language capabilities; reasoning models spend more inference on complex answers; MoE architectures route work among experts; and reservoir computing targets temporal data with a simpler training process.

Most of these approaches continue to rely on GPUs or other accelerators for neural network calculations. Agents stand out because their orchestration adds substantial CPU work while still depending on models that use accelerators. That means data center planners may have to provision for more than accelerator count: CPU capacity, memory movement and communication between components can shape the cost and performance of a complete system.

For buyers and engineering teams, the architecture should follow the workload. A narrow task may fit an SLM or reservoir computer; broad coverage may favor an MoE model; a multistep task may require an agent that coordinates models and services. The next releases will show how well these designs manage their respective bottlenecks—and whether the projected CPU-to-GPU shift holds across real deployments.

Reference
1. Michael te Vrugt, “An introduction to reservoir computing,” chapter forthcoming in Artificial Intelligence and Intelligent Matter, Springer, Cham, arXiv:2412.13212v1

Related Article
Agentic AI Is Changing Data Center Architectures
Data center designs are moving toward heterogeneous SoCs and chiplets that combine CPUs, GPUs and NPUs to address memory bottlenecks, latency and efficiency.

The post Agentic AI Alters The CPU-GPU Balance appeared first on Semiconductor Engineering.