OctoML

OctoML built infrastructure for optimizing and running machine learning models across different hardware. Its products evolved from Octomizer, which optimized and packaged models, to OctoAI, a managed cloud compute service, and OctoStack, a platform for running generative AI in customers’ own environments. By April 2024, the company operated as OctoAI; NVIDIA acquired it later that year.[4][7][8][9]

The company’s earlier work centered on Apache TVM, the open-source compiler framework underlying Octomizer. That research provides the technical context for its first model-optimization service.[3][4]

In a 2020 founder interview, Luis Ceze traced TVM’s genesis to an observation he and Carlos Guestrin made: both the range of machine learning models and the hardware available to run them were growing. He credited PhD students Tianqi Chen, Thierry Moreau and Jared Roesch as key contributors behind the project, alongside the wider open-source community.[3]

TVM turns model computations into code suited to a target processor. In his August 2023 explanation, OctoML co-founder Tianqi Chen described importing computational graphs or loop-level programs, combining operators through fusion, planning memory use, and generating low-level code for GPU and non-GPU backends. Python APIs allow developers to customize intermediate transformations. Chen credited compiler collaborators and inspiration from Halide rather than claiming sole invention of machine learning compilation.[1]

Ceze recalled that the December 2018 TVM conference drew well over 200 people and helped reveal interest in the technology. Chen, Moreau and Roesch were approaching graduation, while Jason Knight—then at Intel, where he supported TVM work—joined the founding discussions. Ceze dated company formation to late July 2019 and said Knight demonstrated an initial product at the December 2019 conference.[3]

OctoML contributed to TVM while the project remained under Apache governance. Knight described company work on documentation, packaging and coordination of larger structural decisions. Ceze distinguished the open-source framework, which users could adopt directly, from a commercial offering intended to make optimization more automated and easier to use.[3]

OctoML announced early access to Octomizer on December 3, 2020. Through a SaaS interface or API, users supplied serialized models and selected hardware targets; the service optimized and packaged the models for those targets. Knight’s earlier description included comparing optimized benchmarks across hardware and receiving outputs usable from Python, a C shared library and API, or a Docker image with a gRPC wrapper.[4][3]

The December announcement reported that an unnamed computer-vision team reduced model latency from 95 milliseconds to 10 milliseconds using OctoML, without specifying the model, hardware or measurement protocol. Optimization and packaging still left a deployment step: in March 2021, Octomizer users deployed the generated artifacts themselves, while deployment support remained on the roadmap.[4][5]

OctoML announced a $3.9 million seed round in October 2019, led by Madrona Venture Group with Amplify Partners participating. On April 3, 2020, it announced the close of a $15 million Series A led by Amplify. Ceze attributed the decision to raise early to hiring opportunities and customer-engagement needs, with planned spending on machine learning systems, platform engineering and business development.[2][3]

On March 17, 2021, OctoML announced that it had raised a $28 million Series B led by Addition, with Madrona and Amplify participating, bringing reported total funding to $47 million. Ceze called the round pre-emptive and cited close to 1,000 early-access waitlist sign-ups, alongside more paying customers than anticipated. The waitlist was not a customer count. At that time, Microsoft was both a partner and customer; Qualcomm and AMD were also partners.[5]

By November 1, 2021, OctoML had raised a further $85 million in Series C funding.[6]

On June 14, 2023, OctoML introduced OctoAI as fully managed cloud infrastructure for running, tuning and scaling open-source, off-the-shelf and custom models. Launch templates included Stable Diffusion 2.1, Dolly v2, Llama 65B, Whisper and Vicuna. Automated hardware selection offered price-performance choices. This extended the product’s role from delivering optimized artifacts to managing the infrastructure that ran them.[7][5]

Chen explained the service’s engineering challenge as broader than making a model run once: requests could grow or change, hardware availability could shift, and models needed to be located close to their execution environments. He described product improvement through work with users and model-specific optimization, with engineers’ changes then reflected in the service.[1]

By its April 2, 2024 OctoStack announcement, the company identified itself as OctoAI, formerly OctoML. OctoStack provided inference, model customization and asset management within customers’ cloud virtual private clouds or on-premises environments. Advertised hardware targets included NVIDIA and AMD GPUs and AWS Inferentia.[8]

The company described Apate.ai, a service combating telephone scams through generative conversational AI, using OctoStack to run customized multilingual and regional-dialect models across geographies. Apate AI founder and CEO Dali Kaafar emphasized the importance of choosing environments for models processing call data, citing flexibility, scale and security.[8]

NVIDIA acquired OctoAI in 2024, as reported by GeekWire on September 25.[9]