Philip Kiely

Philip Kiely is a software developer at Baseten and the author of Inference Engineering, a guide to running generative AI models in production. He joined Baseten in January 2022. His contributions include technical documentation, conference teaching, and explanations of the engineering that separates generating a token from delivering a fast, reliable model API.[3][1]

Kiely built Inference Engineering from his documentation, blogs, and talks, interviews with Baseten engineering colleagues, and conversations with customers and builders. The book connects model architecture and GPU hardware with serving software, optimization techniques, and production infrastructure, with a particular focus on NVIDIA GPUs and data-center inference.[3]

His stated teaching goals were to give readers “a working vocabulary for the space as a whole” and intuition for how inference techniques work. He deliberately included Medusa, an older speculative-decoding method, to explain the field’s evolution rather than present it as the newest approach. In his August 2026 conversation with Ali Taha on Latent Space, he described his AI Engineer talk as the book’s first public addendum, reflecting how quickly speculative decoding had advanced.[1]

Kiely says he tried AI assistance for research and proofreading but found its useful contributions limited to tasks such as alphabetizing an appendix and correcting figure-number inconsistencies. To meet his quality standard, he wrote the book himself and worked with human designers; he specifically credited Luke from Baseten’s design team for illustrations. For the audiobook, he worked with Rime to create a clone of his voice and built a multistage audio-generation pipeline.[3][1]

His teaching also includes a TensorRT-LLM workshop at the 2024 AI Engineer World’s Fair. He subsequently helped connect Latent Space’s hosts with Baseten co-founder Amir Haghighat and model-performance engineer Yineng Zhang for their January 2025 discussion of mission-critical inference.[2]

Kiely distinguishes basic model support from a production-ready API. In his explanation, getting a model to emit tokens on an open-source inference engine is only the starting point. Production serving also requires calibrating quantization to preserve model intelligence, training speculative decoders, setting up and testing infrastructure, and continuing to fix bugs after launch. He recalled a GLM serving issue in which certain prompts and temperatures caused repeated-token output, illustrating how real traffic exposes failures beyond pre-release testing.[1]

For long prompts, Kiely starts with whether part of the input has already been processed. He explained how cache-aware routing sends requests toward replicas with available prefill workers and reusable cached input, avoiding repeated computation. On certain models, separate GPU workers handle prefill—processing the input and creating the KV cache—and decode, which generates subsequent tokens. He also described dedicated endpoints as a way to choose batch sizes, parallelism, and model precision while isolating a workload from other customers’ traffic.[1]

Kiely’s quality standard is fidelity to the original model: inference optimization should make serving faster while preserving intended behavior. He distinguishes cache reuse and verified speculative decoding from quantization, which can lose information and therefore requires careful format selection, layer selection, and calibration. His performance estimates carry similar qualifications: large gains depend on hardware, cache hits, batch size, and parallelism. When Taha asked to hold hardware and GPU count constant, Kiely estimated two- to fourfold gains from inference optimizations; this was an interview estimate, not a controlled benchmark.[1]

He makes a parallel distinction between output format and correctness. A state machine can constrain generation to a specified tool-call format, but the model can still select the wrong tool or call none. Kiely explains an LLM’s role as suggesting actions: an action occurs when another system interprets and executes the formatted suggestion.[1]

Kiely sees training and inference becoming more closely connected. He described a prospective improvement loop in which product traces inform post-training, new models are deployed and A/B tested, and the results guide further changes. He also identified continual learning as a research interest, asking how inference systems would need to change if they learned from everything they processed.[1]

Before joining Baseten, Kiely had already published a book about technical writing. That earlier work supplies context for his later documentation and educational publishing.[5][3]

Kiely began writing technical tutorials in spring 2019. He recalls that requests from friends and coworkers about how to publish similar work inspired Writing for Software Developers. He published it on May 12, 2020, six days before his virtual graduation ceremony from Grinnell College, where he received a B.A. with honors in computer science. The book teaches idea development, writing, editing, and publishing, alongside interviews with eleven writers.[5]

In his first-person launch account, Kiely reported selling more than 520 copies and grossing over $19,000 in four days, after starting with 20 Twitter followers and an eight-person newsletter. He said several interviewees promoted the launch. He also described four rounds of his own edits, chapter-level beta reading by about a dozen friends, and developmental copyediting and proofreading by his mother, a professional editor. Griffin Mareske narrated this earlier book’s audiobook.[6][5]