Inference chip startup SambaNova valued at $11B in $1B funding round

Chip startup SambaNova Inc. today announced that it has raised $1 billion in funding at a $11 billion valuation.

General Atlantic led the Series F round with contributions from more than a dozen others. Intel Capital, Vista Equity Partners and JPMorgan Chase & Co. were among the participants. The investment follows a $350 million round in February. 

SambaNova debuted its flagship product, an inference chip called the SN50, in conjunction with the February raise. The company says the accelerator can provide more than three times as much throughput as Nvidia Corp.’s B200 graphics card. The SN50’s top speed, in turn, is described as being five times faster.

Artificial intelligence models generate prompt responses through an iterative process. A set of artificial neurons analyzes the user’s prompt, generates a preliminary response and saves the response to memory. A second set of artificial neurons then retrieves the preliminary response, refines it and saves the results back to RAM. The workflow is repeated numerous times until the AI arrives at an answer.

The repeated movement of data to and from memory accounts for much of the time required to generate prompt responses. According to SambaNova, the SN50’s performance is the result of an architecture that speeds up on-chip traffic. 

The SN50 comprises dozens of modules called tiles that each combine processing circuits with high-speed SRAM memory. The close proximity of the memory to the processing circuits reduces the amount of time it takes data to travel between them, which speeds up inference. When a tile completes the calculations assigned to it, it sends the results to the next tile for further processing.

The SRAM memory in the SN50’s tiles is one of three RAM varieties used by the chip.

According to SambaNova, the SN50 includes an HBM memory pool that can store an AI model’s active model weights and KV cache. Those are the components that perform the bulk of the processing involved in answering prompts. The SN50 also features a DRAM module that enables it to store multiple inactive neural networks. If an application needs to swap the AI model it uses to process prompts, the chip can load a neural network from DRAM to HBM in a few milliseconds.

SambaNova ships the SN50 as part of a 16-chip appliance called the SambaRack SN50. The system uses about 20 watts of power, which means that it produces a relatively limited amount of heat. That enables the system to use standard air cooling equipment instead of the more complicated liquid cooling systems that many high-end graphics card servers require.

JPMorgan, one of the contributors to SambaNova’s new funding round, today announced plans to adopt the SN50. It also intends to use the company’s previous-generation SN40 chip. The bank will integrate the processors into its on-premises inference infrastructure.

SambaNova will use the proceeds from the round to enhance its chip lineup, rack design and software. The company also plans to accelerate its go-to-market efforts.

Image: SambaNova