Agentic inference reshapes AI infrastructure at RAISE Summit

Agentic inference is reshaping the center of gravity in AI infrastructure. What began as a race to scale training has shifted into a phase defined by expanding context windows, memory‑augmented reasoning and the need to keep graphics processing units continuously fed with data.

As enterprises push deeper into agentic systems, storage has moved into the critical path of AI performance. Agentic inference is exposing new bottlenecks, accelerating the adoption of design patterns and pushing organizations to rethink how intelligence is staged, retrieved and delivered to GPUs, according to Greg Matson (pictured), senior vice president and head of marketing and products at Solidigm, a trademark of SK Hynix NAND Product Solutions Corp.

“It started a couple of years ago with training, where the need for high-capacity, high-performance storage very adjacent to the GPUs was all of a sudden center stage,” Matson told theCUBE. “But now, as we go from last year to this year, inference phase into agentic inference, it’s exploding even more. Storage is actually a whole new storage tier that’s being created to extend the memory for the system.”

Matson spoke with theCUBE at RAISE Summit, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. Conversations with cross-industry leaders showed how agentic inference is expanding infrastructure design beyond raw compute to encompass specialized architectures, memory and storage, capital deployment and sovereign control of enterprise data. (* Disclosure below.)

Here’s theCUBE’s complete video interview with Greg Matson:

Here are three insights you may have missed from theCUBE’s coverage of RAISE Summit:

Insight #1: Agentic inference drives specialization across the AI stack.

Advanced Micro Devices Inc. is responding to varied AI workloads by optimizing across CPUs, GPUs, adaptive computing and networking rather than focusing on individual chips. Its ROCm software stack is designed to provide a consistent layer across data center clusters, edge deployments and AI-enabled PCs, according to Mark Papermaster, chief technology officer and executive vice president of AMD.

“The workloads are so complex because people are looking at what they do end to end,” he said during the event. “They’re looking at whole processes, not just one bespoke task. That means you need different computing engines, and they need to work together at scale. We’re talking across massive clusters of racks.”

Tensordyne Inc. is addressing power constraints by changing the math inside the silicon. Its Napier inference chip uses the proprietary Pareto logarithmic number system, replacing multiplications with additions to reduce reliance on large, power-intensive multiplier circuits. A 72-chip Napier pod draws 30 kilowatts, compared with 150 kilowatts for a comparable Nvidia Corp. system, according to Gilles Backhus, co-founder of Tensordyne.

“Our logarithmic math — it’s completely under the hood,” he told theCUBE. “From a user point of view, from a [software development kit] point of view, you don’t even notice it. It just looks like normal floating-point math. It’s just that the engine under the hood is more efficient.”

Purpose-built accelerators are also moving into production alongside GPUs. A Parasail deployment pairs d-Matrix Corsair accelerators with Nvidia Hopper and Blackwell GPUs to serve the different requirements of compute-heavy prefill and latency-sensitive token generation, according to d-Matrix Corp. co-founders Sudeep Bhoja, chief technology officer, and Sid Sheth, president and chief executive officer. It represents an early commercial-scale example of heterogeneous inference in production.

“Low latency is the name of the game today,” Bhoja said during the event. “Agents are running for a long time; users don’t want to wait. So, there’s a demand on trying to get the latency down, and that means disaggregated inference.”

Here’s the complete video interview with Sudeep Bhoja and Sid Sheth:

Insight #2: Storage becomes an active extension of AI memory.

As agentic inference expands from individual prompts to longer-running sessions, the volume of context data can exceed GPU memory capacity. Hyperscalers are replacing legacy infrastructure with high-capacity solid-state storage positioned near accelerators, according to Matson.

“While the GPU is the most expensive part of your infrastructure, you want that thing to be humming 100% of the time generating tokens,” he told theCUBE. “And if it’s down, waiting for data, then you’re wasting your money on GPUs.”

Solidigm is testing storage as part of complete AI systems rather than as an isolated component. Its AI Central Lab runs actual workloads across accelerator hardware and partner software. At the same time, high-density solid-state drive configurations show how greater capacity in less rack space can reduce storage power demands, according to Avi Shetty, vice president of AI ecosystem, solutions and market enablement at Solidigm.

“Nobody cares about random read, random write on a data sheet right now,” Shetty said. “What people care about is how does it operate in an AI data center and an AI workload.”

Here’s theCUBE’s complete video interview with Avi Shetty:

Insight #3: Capital and sovereignty become part of the AI infrastructure stack.

Agentic inference projects require more than power and GPUs; they can stall when developers can’t assemble financing quickly enough. Argentum AI Inc.’s demand-first model secures customers before committing capital, using contracted revenue to support construction while remaining neutral regarding silicon and original equipment manufacturers, according to Andrew Sobko, founder and chief executive officer of Argentum AI Inc.

“We formed the view that the biggest bottleneck in the focus on the speed of deployment is a capital stack,” he told theCUBE. “How do you get the projects financed as fast as possible? That sort of became one of our core products, where we call it bringing power, compute and capital.”

Data sovereignty, meanwhile, is moving from compliance into infrastructure architecture as agentic inference draws on proprietary enterprise context. The concept spans territorial, operational, stack, legal and unit economics concerns, according to Amit Eyal Govrin, chief executive officer of Agentcy Labs Inc., and Philip Rathle, chief technology officer of Neo4j Inc.

“Sovereignty is exerting agency and control over your AI,” Govrin said during the event. “You have to be free and clear of state, economic and threat actors overtaking any level of control over your stack. You’re not paying rent to somebody else.”

Knowledge graphs add another form of control by allowing some decisions to run deterministically rather than relying entirely on probabilistic models. That can give enterprises consistent business rules, explainability and governance alongside the flexibility of large language models, Rathle noted.

“Having the capacity to do AI with a full brain, both hemispheres, is ultra important,” he told theCUBE. “LLMs are spontaneous, creative — they make mistakes, you don’t know why. Having the graph as the left brain to the LLM right brain is really at the core of where graphs fit in.”

Here’s theCUBE’s complete video interview with Amit Eyal Govrin and Philip Rathle:

To watch more of theCUBE’s coverage of RAISE Summit, here’s our complete video playlist:

https://www.youtube.com/watch?v=videoseries

(* Disclosure: TheCUBE is a paid media partner for the RAISE Summit event. Neither Solidigm, the headline sponsor of theCUBE’s event coverage, nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE