Ishan Dhanani

Ishan Dhanani is a senior Deep Learning Algorithms engineer at NVIDIA whose work focuses on scalable, distributed inference systems. His collaborative work on NVIDIA Dynamo addresses how self-hosted models serve coding agents: keeping reusable context cached, routing requests to suitable workers, and preserving the reasoning and tool-call structure that agent software expects.[3][4][5]

In their April 2026 technical article, Dhanani and Matej Kosec identify a central opportunity: “The biggest optimization surface in agentic inference is the gap between what the harness knows and what the infrastructure can see.” A harness—the software driving an agent—knows which agents are waiting for tools, which will resume soon, and which context remains useful. An inference server receiving anonymous token requests cannot make the same distinctions.[4]

They describe Dynamo work across three layers: frontend APIs, routing, and KV cache management, which retains computed model state for reuse. A common internal representation supports Chat Completions, Responses, and Messages APIs. The nvext.agent_hints extension lets a harness communicate request priority, estimated output length, and intent to warm a prefix before a request is ready. Cache-aware routing weighs existing prefix overlap against a worker’s decode load, reducing the need to recompute context on another worker.[4]

Their design distinguishes repeatedly useful system prompts and conversation history from short-lived subagent state. It also separates implemented controls from further work: as described in their April 2026 article, updated May 14, cache-retention directives apply locally to a worker, while carrying those policies across shared storage, prefetching context before an agent resumes, and associating eviction with agent lifecycles remain development directions.[4]

Dhanani coauthored a May 2026 follow-up with Matej Kosec, Benjamin Klieger, and Alec Flowers. One concrete problem was Claude Code’s session-specific billing header at the beginning of a prompt: changing the first tokens prevented otherwise identical instructions and tool definitions from matching cached prefixes. The team added --strip-anthropic-preamble to remove the header before tokenization. In their NVIDIA B200 experiment with a 52,000-token prompt, they reported time to first token of 168 ms for a stable prefix, 912 ms with the varying header, and 169 ms after stripping it—a result specific to that workload.[5]

The coauthors also describe correcting how Dynamo reconstructs interleaved reasoning and tool calls. Combining separate reasoning spans into one block could preserve the tokens while losing their relationship to the tools they explained. Their account emphasizes model-specific replay policies and explicit ownership of reasoning parsing. For responsiveness, Dynamo can emit a parsed tool_call_dispatch event when a complete call is ready, allowing the harness to execute the tool before the response stream ends.[5]

Their compatibility argument extends beyond valid HTTP responses. In the Codex setup they tested, model-catalog metadata changed system instructions, reasoning support, and how tool output was truncated before returning to the model. A 10,000-byte fallback budget preserved less ASCII-heavy coding output than a 10,000-token budget. They argue that serving compatibility must therefore include the harness’s request-shaping behavior as well as the API schema.[5]

Before this agent-serving work, Dhanani worked on access to GPU compute. In an October 2023 Semafor report, Reed Albergotti described him as a Columbia University computer-science graduate student seeking resources to fine-tune Llama 2. After difficulty obtaining GPUs through conventional providers and the university, he rented an NVIDIA A100 through Akash for a reported $1.10 per hour, completing roughly seven hours of work.[8]

Semafor reported that the experience prompted Dhanani and two friends to launch Agora Labs to simplify booking GPU time through blockchain markets. His stated concern was access: “The OpenAIs and Anthropics can’t be the only ones that have the power to train and host models like ChatGPT.”[8]

In his July 16, 2024 account, Dhanani named Anish Maddipoti and Thomas Montfort as his Agora collaborators. He described dropping out and moving to San Francisco in mid-March to work on Brev, followed by working inside NVIDIA after its acquisition of Brev.dev.[7]

By May 2025, Dhanani was one of seven coauthors explaining Dynamo v0.2’s infrastructure changes. Their autoscaling argument was that request count alone poorly predicts inference load: long inputs stress prefill GPUs, while long outputs stress decode GPUs. The described planner monitors pending prefill requests and decode KV-block utilization against configurable thresholds to rebalance resources. This earlier work supplies the distributed-serving context for his later agent-inference articles.[6][4]

Dhanani also hosted the AI infrastructure track at the 2025 AI Engineer World’s Fair, as credited in Latent.Space’s recap. In his contemporaneous announcement, he described a program spanning edge devices and data-center-scale inference and announced the first NVIDIA Dynamo user meetup, featuring the vLLM and SGLang teams discussing prefill/decode disaggregation and smart routing.[1][2]