Gimlet Labs Inc., a startup that helps developers speed up their inference workloads, has raised $300 million in funding at a $3 billion valuation.
Andreessen Horowitz led the Series B round. Gimlet stated in ablog posttoday that the firm was joined by Arm Holdings Inc., Samsung Ventures, Microsoft Corp.’s M12 fund and more than a dozen others. The company’s total outside funding now stands at $392 million.
A large language model comprises multiple software modules that have vastly different hardware requirements. For example, some modules make heavier use of the underlying graphics cards’ memory than others. Gimlet has developed a platform that harnesses that phenomenon to speed up inference.
The company’s software can automatically break up an LLM into modules. From there, it deploys each module on the chip architecture that best aligns with its hardware requirements. For example, Gimlet might send a memory-intensive component to an accelerator with a large amount of onboard RAM.
There are multiple ways to split up an LLM. The most common approach, PD disaggregation, runs the prefill and decode phases of the inference workflow on separate chips. The prefill and decode phases are processing steps in which an LLM interprets prompts and generates a response, respectively.
Gimlet also supports more granular disaggregation approaches.
Some developers don’t stop at splitting the inference workflow into prefill and decode phases. They take it a step further by diving the decode phase into smaller workflows that can each be assigned to a different chip. In other projects, developers use a lightweight “drafter” model to generate initial prompt responses and a frontier LLM to refine those drafts. Each model runs on a separate chip architecture.
Gimlet’s platform reduces the amount of work required to implement such disaggregation workflows. Additionally, it optimizes each LLM module for the chip architecture on which it’s deployed. The software optimizes model code using a combination of AI agents and a custom compiler.
According to Gimlet, its agents find the best way of adapting LLM code to a chip by exploring multiple design approaches. After identifying a promising approach, the AI runs tests to ensure that it’s correct. The company’s compiler, in turn, applies a mix of generic and chip-specific optimizations to customer models.
Gimlet sells its software in a serverless edition and as a managed service that enterprises can deploy on their own infrastructure. The company says it has received billions of dollars’ worth of customer orders. In March, itstatedthat its customer base includes one of the world’s largest cloud providers and a top three AI lab.
The funding will help the company grow the infrastructure capacity of its platform’s serverless edition. Gimlet plans to add several hundred megawatts of computing power. Additionally, the company intends to expand into the custom hardware market. Gimlet is developing an inference-optimized server that doesn’t feature a motherboard and can be used outside data centers.
Photo: Unsplash
A message from John Furrier, co-founder of SiliconANGLE:
Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network, where technology leaders connect, share intelligence and create opportunities.
- 15M+ viewers of theCUBE videos, powering conversations across AI, cloud, cybersecurity and more
- 11.4k+ theCUBE alumni — Connect with more than 11,400 tech and business leaders shaping the future through a unique trusted-based network
Are you an AWS customer? Support SiliconANGLE financially by buying your AWS services from ourMarketplace portal page and links:https://siliconangle.com/aws-marketplace/
About SiliconANGLE Media
SiliconANGLE Media is a recognized leader in digital media innovation, uniting breakthrough technology, strategic insights and real-time audience engagement. As the parent company of SiliconANGLE, theCUBE Network, theCUBE Research, CUBE365, theCUBE AI and theCUBE SuperStudios — with flagship locations in Silicon Valley and the New York Stock Exchange — SiliconANGLE Media operates at the intersection of media, technology and AI.
Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a dynamic ecosystem of industry-leading digital media brands that reach 15+ million elite tech professionals. Our new proprietary theCUBE AI Video Cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.
