ZML launches chip-agnostic AI server to cut inference costs

ZML has released LLMD, a high-performance inference server that runs open-source large language models across Nvidia, AMD, Google TPU, Apple Metal and Intel Arc hardware. The Paris startup says the technical preview is built to cut the cost of serving models by reducing dependence on any one chip family.

LLMD includes OpenAI-compatible API endpoints, continuous batching, paged attention and sharding, and ZML says it supports model families including Gemma, Llama, Ministral, Mistral and Qwen. The company has framed the software as a way to break hardware silos and let enterprises and cloud operators mix chips for speed, cost and energy-efficiency gains.

The release lands in a market where inference, the work of processing prompts after a model is trained, has become a major infrastructure fight. If ZML’s performance claims hold up, operators could choose among different chip families instead of being tied to a single vendor, a shift that would matter most to startups watching burn rates, cloud customers under pressure to trim usage bills and enterprises running private models at scale. ZML has also said the approach could benefit newer chipmakers, including Axelera, Fractile, Kalray, OLIX, Q.ANT, SiPearl, SpiNNcloud and VSORA, even as the company says it keeps a good relationship with Nvidia.

ZML says LLMD is not intended for production use yet. The company’s GitHub profile describes its stack as a high-performance AI inference system built with Zig, OpenXLA, MLIR and Bazel, while its Docker image page calls LLMD a technical preview. ZML/v2, announced in March 2026, was described as a complete rewrite focused on developer experience, performance and composability.

Steeve Morin, ZML’s founder and chief executive, previously spent seven years as vice president of engineering at Zenly before Snap acquired the company. ZML says it was founded in 2023 and now has a team of 20 people. Morin has said the company’s goal is to make different chips available for AI use cases at their maximum speed, and sometimes faster, as the economics of inference become a central battleground in AI infrastructure.