In one of the most exciting podcasts I’ve had the chance to shoot, I sit down with Sid Sheth, the founder of d-Matrix, to explore how a startup can successfully compete against $100B+ incumbents in the semiconductor inference space. Based in the Bay Area, d-Matrix is a leading provider of Digital In-Memory Computing (DIMC) solutions for transformer and generative AI inference acceleration.

The key to mass adoption of AI lies in cost-effective and energy-efficient compute solutions that can scale. Today, most hardware accelerator platforms are optimized for training large language models (LLMs). However, for AI to thrive across diverse use cases, we need large-scale infrastructure designed specifically for inference. Current silicon solutions from incumbents like NVIDIA are suboptimal for this purpose—they repurpose training-focused general-purpose GPUs, add high-bandwidth memory, and market them for inference customers. Yet, inference (expected to become a larger market than training) requires accelerators specifically optimized for individual use cases, such as data centers, edge personal computing, computer vision and others.

In this candid fireside chat, we delve into Sid’s background and how his Gujarati upbringing has instilled a relentless focus on customers, revenue, and profitability. We also discuss the hardware and software decisions that d-Matrix has made to leverage DIMC. While GPUs excel at batch processing for training, inference involves sequential outputs from LLMs, necessitating an architecture optimized for fast matrix multiplication with memory located closer to compute to minimize latency. Sid explains how d-Matrix has optimized for inference without locking itself into a specific architecture and shares insights into how an underdog startup can outpace trillion-dollar giants in the race for inference dominance.

At Z47, we are highly optimistic about opportunities across compute layers, including semiconductors, data centers, and embedded systems. If you’re building in this space and would like to connect, feel free to reach out at semiconductors@z47.com