Inference Efficiency Starts with the Application

From manual classification to production inference

Nyla Worker’s introduction to machine learning began with tedious astronomical image classification. While manually selecting regions in images, she discovered a 1996 paper called Source Extractor describing a neural network trained on synthetic data to distinguish stars from galaxies. The appeal was practical: a model could automate work she was doing by hand. She wanted to understand both how to train it and how to deploy it.

At eBay, deployment became her focus. Working in the hardware team, Worker optimized software so the AI research team’s models would run efficiently. Inference—running a trained model—became the engineering priority once the model had a useful job. This Latent Space conversation, published September 3, 2024, describes her Convai-era work. Its recording date is unspecified, but it was recorded before she joined Google AI to work on efficiency.

Charlie’s introduction proposes a theoretical, nearly 3,000-fold improvement in computer-vision throughput over six years at constant quality, combining gains from hardware, quantization, pruning and distillation. That is a claim about potentially stackable improvements, not a single demonstrated benchmark. The deployment examples show why latency, hardware configuration and task quality must accompany any throughput number.

Four images inside seven milliseconds

eBay’s image search extracted an embedding—a numerical representation—from a layer of [ResNet-50](https://en.wikipedia.org/wiki/Residual_neural_network) and compared it with other image embeddings using a distance calculation. The model had a seven-millisecond inference budget. Worker estimated that the complete service would respond in under roughly 50–100 milliseconds.

The original deployment processed one image at a time. In 2018, Worker used TensorRT on a V100 to process four images within the same seven milliseconds. Relevant optimizations included kernel fusion and reduced numerical precision: moving from FP32 to FP16. INT8 was still early in that deployment context. The gain was more work from the hardware while preserving the product’s response time.

Four images every seven milliseconds is approximately 571 images per second. Worker contrasts that with a reported MLPerf result of 88,000 samples per second from the preceding year, while explicitly qualifying the comparison: eBay optimized under a latency ceiling; the benchmark optimized throughput. The newer system’s hardware configuration and detailed conditions are unspecified, so the figures do not establish a like-for-like speedup.

When Alessio asks whether teams should optimize for present or future hardware, Worker starts with the workload. She describes a ResNet-50 application that ran on a V100 in 2018 as later suitable for an inexpensive Jetson device, while a huge batch workload could still justify a large accelerator. She is skeptical of forecasting beyond two years or planning around chips three years away. At eBay, making these choices also required bridging a learning gap among hardware, platform and AI research teams.

Batching, quantization and the quality constraint

Performance can improve on the same chip as software evolves. Batching illustrates the opportunity: processing several images together raises throughput, up to memory limits, provided the batch still meets the latency ceiling. Increasing eBay’s batch size to four reportedly produced substantial savings. The discussion also compares dynamic batching with continuous or in-flight batching for large language models (LLMs).

Quantization reduces the numerical precision used by a model. In the NVIDIA edge-device work described, INT8 was paired with [quantization-aware training](https://developer.nvidia.com/blog/improving-int8-accuracy-using-quantization-aware-training-and-tao-toolkit/) so the model could learn to perform under those constraints. Hardware formats and framework support contributed gains alongside changes to the network itself.

Pruning offered another route. The speaker describes removing near-zero activations from a trained network, then training and deploying it again. This reportedly helped computer-vision customers, but did not translate efficiently to LLMs at the time of the discussion.

Each optimization still had to preserve useful task performance. Quantizing a ResNet-18 for license-plate detection required deciding how much accuracy or recall could be sacrificed for efficiency. A manufacturing line could put greater weight on latency because items keep moving past the inspection point. Throughput gains count only while response time and task quality remain acceptable.

Whether vision or language tolerates quantization better remains an open question here. One participant suggests that language may depend more on fine numerical distinctions. The response offers a competing intuition: each parameter in a small model may carry more responsibility, making model size more consequential than modality. A poorly quantized, single-task ResNet-18 could deteriorate sharply. Neither intuition establishes a general rule.

The discussion of 1.58-bit ternary models is similarly exploratory. A participant mentions apparent open-source reproductions while remaining uncertain about adoption and prospects. Another suggests that very large models might depend more on the direction of many parameter contributions than on their exact magnitudes—an explicitly speculative, gate-like analogy.

The data wall: new tape, missing poses

The NVIDIA customer-work account moves from a solutions-architect program supporting hundreds of customers, primarily in computer vision, to an edge-AI team working with companies including Siemens. Across industrial deployments, available models and large general image datasets repeatedly failed to cover the actual task: scratches on cars, products on shelves, or packaging that had changed since the last training run.

An Amazon project addressed one specific change: new package tape. The team replaced tape synthetically in a 3D environment to support rapid retraining of package-recognition networks. Robotics exposed a different problem. To label an object’s pose, teams attached QR tags, but those tags changed its appearance and could not provide coverage of every position. Generated examples offered a way around those collection limits.

This was the data wall: existing examples no longer supported progress on the required task. Synthetic data could fill the gap, but choosing useful variations—domain randomization within the generated 3D environment—required its own expertise. The speaker predicts a related constraint for LLMs while leaving synthetic data’s eventual role unresolved.

A game engine supplied both controllable events and labels. In the described product, objects had adjustable weight and falling motion, producing physics-informed examples for pose estimation. Their positions across frames supplied ground truth that was difficult to obtain reliably from human-labeled video. That experience did not establish how Sora worked: the speaker declined to assess its private implementation or endorse rumors about game-engine training data.

Creating the 3D assets remained rougher. Text-to-texture, text-to-material and image-to-3D systems could assist production, but flawed geometry still needed retopology, or repair of the asset’s geometric structure. One personal experiment turned a generated depiction of Andrej Karpathy into a Slender Man-like figure. The longer-term ambition was a navigable generated world populated by conversational characters capable of planning tasks—a flexibility that fixed video could not provide.

How much model does the task need?

Identifying a car’s color does not require all the mathematics, physics and computer science a broadly pretrained model may contain. In the computer-vision experience described, teams tried EfficientNet and vision transformers, then repeatedly returned to ResNet-50 for constrained applications. Data requirements, training difficulty and the established model’s adequacy mattered more than novelty.

Computational efficiency also influences architectural adoption. A participant points to attention’s greater ability to train in parallel than the preceding long short-term memory networks, enabling more data processing alongside the research advances. The proposed next step follows the same practical logic: identify valuable training data, train and quantize more efficiently, or distill useful capabilities into a smaller model. Matching GPT-4-class performance on a constrained task is presented as a possibility, not a demonstrated result.

Databricks Assistant illustrates matching model cost to task complexity. It is described as using different models for different operations: some need classification, others full assistant behavior. Expensive language-model inference on every operation could erode the margins of a software service.

Distillation transfers something learned by one model into another. One participant distinguishes three targets: - Knowledge: information or capabilities. - Preferences: training signals supplied by an LLM acting as a judge. - Reasoning: step-by-step reasoning examples, with Orca mentioned as an example.

What gets transferred matters for evaluation. Distillation can target benchmark content or the preferences rewarded by a leaderboard such as LMSYS; success there does not establish broadly transferable capability. Another participant recalls researchers selecting a successful checkpoint despite being unable to reproduce the result. The practical response in customer work was to build a representative, repeatable test environment that measured what the application actually needed.

Better data meets the general-purpose problem

FineWeb, a Hugging Face dataset, enters the discussion alongside LLM-based filtering: using models to select better training material rather than generate it. A participant cites preliminary LLM.C results in which a GPT-2 architecture trained on 10 billion higher-quality tokens reportedly matched or slightly improved the evaluation scores and perplexity obtained with 100 billion tokens. The tasks and other training conditions are unspecified; this is a preliminary comparison, not a general tenfold savings rule.

That fits the reported vision experience: synthetic data matched to deployment could reduce the number of samples needed. But a general-purpose model complicates the analogy. Conventional machine learning can start with a defined problem and tailor its data and architecture accordingly. A foundation model must cover an open-ended range of tasks. Knowing what to optimize for is itself part of the difficulty.

One proposed answer is a complete feedback loop. Coding environments let a model rerun work and evaluate the outcome; robotics needs capable sensors to support comparable feedback. A related interpretation treats human-written text as knowledge already selected and organized through human effort: textbooks package information that next-word prediction can learn from.

These are intuitions about progress toward artificial general intelligence (AGI), whose definition remains contested. One participant sets a particularly demanding threshold—self-improving intelligence beyond anything humans could produce—and doubts that current methods can reach it. The assessment depends on what intelligence means and how it is measured.

Embodiment makes latency visible

Convai returns the discussion to an interactive product: conversational 3D non-player characters (NPCs). The described stack combines a language model, retrieval-augmented generation, automatic speech recognition and text-to-speech. Embodiment adds avatar integrations such as MetaHuman, facial and gesture animation, perception and a fine-tuned action model. A character should be able to follow an instruction to move somewhere and retrieve an axe.

Conversation can become a game mechanic. One indie game asks players to persuade NPCs to evacuate a region or kill them; a social variation involves persuading one character to convince the others. Outside games, embodied brand agents are pitched as support interfaces whose appearance, personality, voice and knowledge represent a company. Hippocratic’s digital-nurse demonstrations provide a separate example of digital-human interfaces; no specific Convai brand deployment is disclosed.

Perception makes the exchange more responsive. A reported nonpublic demo let a character notice a user’s facial reaction, initiate a conversation about it and change its animation state. At Computex, a participant recalls a character display whose eyes followed visitors: convincing at a distance, visibly artificial up close, and still troubled by delay. Appropriate reactions to tone and gestures require expressions and actions to arrive promptly. Response latency becomes part of the character’s believability.

The recording inserts an NVIDIA GTC demo of Jin and Nova conversing and acting in a ramen shop. Charlie cautions that the excerpt will sound scripted; the demo narrator says conversations vary between runs. When the interview resumes, the requirement is concrete: the character must pick up the ramen while conversing, with a natural reaction time. Fluent dialogue alone does not complete the interaction.

Characters that act, teach and disagree

Convai NPCs are reported as live in Second Life, helping users onboard and meet other people. They explain how to pick up a surfboard, surf, fly and dance, with personalities suited to the environment. The broader immersion problem is familiar: a character that remains cheerful after nearby violence, or offers routine assistance after its shop has been looted, ignores the world it supposedly inhabits.

Character-to-character interaction adds different knowledge for different roles. In the NVIDIA example, Jin was configured around ramen knowledge, while Nova had information about recently shipped GPUs. Their exchanges mixed both subjects. A proposed enterprise extension pairs a sales-representative NPC with a deliberately disagreeable counterpart to explore objections and responses. This could generate practice scenarios; its ability to predict real customers is not demonstrated.

Staff training is described as an existing enterprise use case, illustrated by doctors practicing with friendly or belligerent patient characters. Humanoid corporate training was still early. In entertainment, a participant expects generated dialogue and procedural worlds to enable new experiences, with independent developers doing much of the experimentation because established studios are reluctant to adopt untested mechanics.

Longer-lived games and approved personas

The closing discussion widens to generated entertainment. One speaker recalls making an AI-generated podcast during the week ChatGPT launched, aiming to generate everything from title and image to voice and editing. By their own assessment, it was poor, but it suggested what a complete generation workflow might become.

Another proposed direction lets an intellectual-property owner reuse characters across games, television, anime and short videos. Fan-fiction demand suggests an appetite for more stories, while leaving open whether established franchises would crowd out new ones. Nyla sees an opportunity to extend a game’s life through AI modding, new characters and world expansion after years of development. She also expects intellectual-property disputes and litigation to complicate that future.

Her personal motivation is more specific: she wanted to converse and take a virtual walk with an Einstein character, echoing walks with her physics professors. She attempted a simulation by uploading material, while acknowledging that she could not know whether Einstein would have approved it.

For living people, Nyla proposes person-approved AI representations, using a sanctioned Jensen Huang persona as a hypothetical example. Approval could make users more comfortable, she argues, though it would not establish fidelity to the person’s knowledge or judgment. She closes by inviting people interested in AI characters, 3D characters and synthetic data to contact her.