From GPU Scarcity to AI That Works on Your Data — Part 3 of 3

When local inference costs more than the network saves

Running a model on your phone or laptop becomes a different proposition if you need to replace the device first. In this final part of Latent Space Chats, an early hardware discussion examines edge inference: running a model on a local device. One participant takes a strongly skeptical position: existing devices constrain model size, while spending on more capable local hardware could serve more users in the cloud.

The argument separates two costs: - Hardware spending per user: The participant estimates that equivalent cloud spending could serve ten times as many users, without specifying a model, hardware configuration or utilization assumptions. - Execution time versus network time: They argue that network delay is small relative to model execution time, so faster cloud hardware can outweigh the advantage of avoiding a network request. They apply this reasoning to large language models (LLMs) and diffusion models, while acknowledging that small models can run locally. The claim concerns the time and cost of delivering a result; eliminating a network request alone does not settle that comparison.

A chip must beat the competition at arrival

Chip startups face another constraint: incumbents keep improving during the development cycle. The host uses a hypothetical five-year development period to illustrate why a design that looks faster or cheaper at conception must compete with five more years of incumbent progress. The respondent argues that an individual feature offers little protection if NVIDIA can implement a version within a year or a couple of generations. A startup needs an architectural difference whose value survives four or five years.

Customer feedback shapes hardware design. The respondent argues that major AI developers give established vendors detailed requirements and guidance about forthcoming model changes, while startups receive less of that information. They cite MatX as a potentially better-positioned startup because its team includes people who built or worked on large models at Google. In the respondent’s assessment, NVIDIA’s principal competitive concerns are Google and Amazon’s chips, and to a lesser extent AMD, rather than smaller startups such as MatX, Etched, Groq or Positron.

The discussion then moves from designing chips to buying them. Asked whether the graphics processing unit (GPU) shortage will become a glut, the respondent predicts an overordering cycle: long lead times encourage customers to order more than they need; suppliers expand capacity; excess orders become visible when deliveries catch up. Automotive microcontrollers supply the analogy—the participant recalls buyers ordering two years’ requirements to secure one year’s supply.

Demand for AI can persist while chip purchases pause. The respondent expects large buyers such as OpenAI and Meta to keep buying, while startups without sufficient economic returns may stop. They do not expect the boom to end in the year of the conversation. As a historical analogy, they recall Google, Microsoft and Amazon pausing memory purchases around 2018, followed by a sharp price decline within about a month. A modest demand slowdown and accumulated inventory allowed buyers to wait; the suggestion that they coordinated is speculative.

Learning video from Blockbuster tapes

The conversation then turns to Milind’s early work on neural networks and video understanding. For his bachelor’s thesis, he implemented neural networks on India’s homegrown supercomputers, using message passing and distributed computation for weights and forward passes. For his PhD, acquiring training data meant renting Arnold Schwarzenegger and Sylvester Stallone movies from Blockbuster.

Action movies provided events such as explosions that could be modeled through multimodal cues—audio and visual information used together. Milind digitized tapes, cleaned the data, built a labeling tool and recruited a friend’s spouse to label it. The research goal was to combine those cues into an understanding of what was happening in a video. Larger datasets, richer computing architectures and GPUs subsequently changed what could be attempted; he identifies the 2012 ImageNet breakthrough as a turning point.

His thesis went beyond detecting isolated objects or events. Context lets detections reinforce one another: seeing sky above land can increase confidence in both classifications. Limited labels and a small corpus constrained this contextual modeling. Milind sees multimodal LLMs trained on large amounts of internet data as tools he would have wanted for that work, though he offers no direct experimental comparison with his original system.

Milind considers reasoning an unfinished prerequisite for intelligent machines and expresses interest in Yann LeCun’s work on models of the world. He argues that video understanding is necessary for such a model. Language already supplies a compact, human-developed representation; a robot navigating a room or a vehicle interpreting traffic must instead contend with pixels, physical variability and distorted signals. In his view, video requires handling complexity that language has already abstracted away.

From understanding scenes to improving traffic

Milind describes his career’s constant as building AI solutions and platforms for practical problems. At IBM Research, his early work focused on video understanding for media companies and broadcasters; he says his team helped establish TRECVID, a video-understanding benchmark. Later, his attention shifted toward using sensor data to observe and optimize physical environments.

He recalls using telecommunications data to infer how people moved through Istanbul and inform bus-route and metro planning, then applying a similar platform to public transport in Dubuque. The work had two decision horizons: - Real-time control: Adjust traffic signals to improve passage along major roads. - Longer-term planning: Design public-transport routes to serve more people, reduce travel time and improve fuel efficiency. These were the stated objectives, without quantitative outcome measurements.

The career account continues through customer-service applications of embeddings based on Mikolov’s Google Research work, alongside streaming network-data analysis for early threat warnings. At NVIDIA, the focus returns to video with NVIDIA Metropolis, an accelerated-computing platform that abstracts model construction and the processing, management and storage of high-volume video. Suggested applications include traffic monitoring, checkout-free retail, automatic building access and parking guidance. The respondent claims developers could build a traffic-monitoring system in days rather than a year, but gives no benchmark conditions for that comparison.

A digital twin—a simulated representation of a physical environment—can help supply rare training examples. For a traffic intersection, the proposed workflow is: 1. Use observed traffic-flow statistics to simulate normal activity. 2. Generate anomalies, such as a vehicle driving against traffic. 3. Add those simulated examples to the training dataset. A dangerous event can occur often enough to matter operationally yet too infrequently to provide a large training corpus. Simulation supplies additional examples; no measured detection improvement is reported.

Build the application to discover the platform

The respondent advocates staying focused on the problems a technology can solve over decades, recalling Jensen’s advice to concentrate on playing the game rather than watching the score. Transformer workloads and ChatGPT’s November 2022 announcement brought GPU computing widespread attention, but the timing and trigger of that response were difficult to predict.

Attention cycles are only part of the explanation for Metropolis’s trajectory. Mapping the physical world into digital representations lengthens development cycles. Observing a physical environment and making its signals usable adds complexity that applications beginning with digital inputs do not face.

A practical platform-design lesson from NVIDIA is to build a concrete application, then remove its application-specific pieces. Use the computing stack as a developer would, construct the solution, and strip it back until the reusable components remain. The respondent pairs this method with first-principles reasoning, honesty about failures and comparison with the best attainable result. Implementation reveals what belongs in the platform.

The respondent describes NVIDIA’s expectation of growing AI compute demand as an early conviction reinforced by deeper networks and expanding research workloads. Supplying that compute at low power consumption was a longstanding priority. But turning compute into useful experiences involves distinct responsibilities: - Platform efficiency: Integrate hardware, drivers, CUDA and the software stack to improve energy and compute efficiency. - Customer value: Turn that foundation into useful experiences for customers, developers or employees.

Applied research close to enterprise data

That division of responsibility explains the interviewee’s move from NVIDIA to Capital One: a desire to work close to enterprise data and customer interactions. Their position is that customer problems cannot be solved in a data-agnostic way from the platform side alone. They describe joining as senior vice president for technology in AI Foundations, an organization intended to turn AI platforms into solutions for customer, developer and associate experiences. At the time of the conversation, they put Capital One’s scale at 14,000 associate engineers and more than 100 million customers.

Applied Research was designed to bring research and engineering together. Capital One created that job family, recruited from NVIDIA, Amazon and Apple, and established a pilot location in San Jose. The interviewee describes academic partnerships and participation in research conferences, with aspirations to publish and collaborate with academia. An applied-research internship program was also being built, with its first cohort planned for the coming summer.

The stated research interests span several directions: - Models suited to banking: What a world model would mean in the domain, alongside interest in smaller models and mixtures of experts. - Trust and temporal data: Responsible and trustworthy AI, and ways to use data over time. - Learning across domains: Lessons from healthcare and biological applications for financial AI. These are research interests, without disclosed implementations or performance results.

The host asks specifically about commercially usable open weights, distinguishing access to model weights from a fully open-source system. The interviewee favors a mix of open, partially open and proprietary models. Access enables customization, evaluation on internal datasets and comparison with proprietary alternatives. They argue that dependence on only a few proprietary models could slow both progress and adoption.

Understanding why a model works matters alongside measuring its performance. The interviewee wants to examine architectural choices and the relevance of pretraining data sources, while benchmarking open or partially open models against proprietary alternatives. Access to weights is not the same as disclosure of pretraining data.

On compute, the interviewee describes prudent GPU use and credits NVIDIA’s optimizations and libraries with making transformer experimentation feasible. They expect capacity bottlenecks to ease over time, without providing capacity or utilization figures. Research allocation likewise has no fixed formula: urgent opportunities receive attention, while alternative architectures get some patience. The boundary is a plausible path to application; deeply theoretical, open-ended projects with ten-year horizons are lower priorities.

The closing test: does it work on your own data?

Reflecting on research cycles, the interviewee recalls a first NeurIPS with several hundred attendees, compared with roughly 18,000 at the December meeting discussed in the conversation. They describe visual research flourishing between AlexNet and the 2017 transformer paper, with architectures such as ResNet, followed by greater emphasis on sequence-to-sequence and text work. Their interpretation is that effective pairings of modality and architecture attract research effort. A future surge around healthcare or gene sequences is a possibility, rather than a reported breakthrough.

Returning to financial services, the interviewee identifies accessible data, cloud infrastructure, a foundation in machine learning and room for technical creativity as enabling conditions. They emphasize that lower customer tolerance for error changes the research problem: finance and healthcare can demand more accuracy than shopping or viewing recommendations. Meeting those requirements may mean modifying open-source systems or extending published methods. The team’s stated priorities include domain-specific datasets, time-series and tabular data, and data imbalance.

The closing hiring discussion makes that applied-research standard concrete. At the time, openings spanned applied research, data science, machine learning engineering and AI engineering, from new master’s and PhD graduates to senior individual contributors equivalent to principal and distinguished research scientists.

The desired capabilities connect product understanding with empirical discipline: - Translate product requirements into research problems: Understand the needed behavior well enough to reformulate a problem or change an architecture, then persist through the difficult final stage of making it work. - Demonstrate improvement on internal evaluations: External benchmark performance is useful, but results on the organization’s own data matter more. Designing meaningful evaluation frameworks and methods is itself research work. - Revise ideas when evidence disagrees: Handle ambiguity, move from theory to practical systems, and rethink or abandon a favored approach when empirical results fail to support it.