From Transformer Experiments to Unreliable GPU Clusters — Part 1 of 3

Improve one task, or improve the model?

An extra two percentage points on question answering can look like progress. But does the change help anything else? Early in this conversation with Yi Tay, that distinction becomes the research problem: pursue an application-specific trick, or change an architecture and test it across roughly ten tasks. The percentage is illustrative; the consequential choice is whether an improvement generalizes across tasks.

Introduced as Reka’s chief scientist and previously an architecture co-lead on PaLM 2 at Google Brain, Tay joined Google at the end of 2019. His early work focused on model architecture and efficient transformers. Long Range Arena and the 2020 Synthesizer paper belong to that period, including experiments asking whether models could dispense with attention.

Tay recalls a research environment centered on machine translation benchmarks and fine-tuning models such as T5 or BERT. He associates the turn toward few-shot in-context learning with GPT-3 and subsequent work. Beneath that shift, he sees continuity: repeatedly taking transformers apart built intuition that remained useful as compute budgets grew and the target became a general-purpose foundation model.

The shift also reorganized research. Application categories such as question answering gave way to shared workstreams in data and architecture, all aimed at improving a universal model. In Tay’s account, ChatGPT made an existing shift unavoidable: large labs had already been moving toward general-purpose models, including T5, but the wider community could no longer ignore that direction.

From holiday experiments to PaLM 2 leadership

A concentration of work around 2022—UL2, PaLM, emergent abilities, and generative retrieval—prompts a question about career strategy. The guest describes following interesting, promising problems as the field moved, with conversations and collaborators shaping the direction. The trajectory looked more deliberate in retrospect than it felt at the time.

The guest distinguishes three kinds of research at Google during that period: - Organization-wide efforts: contributing to PaLM 1, then co-leading a modeling workstream on PaLM 2. - Personally initiated projects: UL2 and the Differentiable Search Index (DSI), begun with available compute during the December 2021 break. - Collaborator-led projects: work such as Flan, where the guest stresses being a co-contributor and acknowledges receiving disproportionate credit through greater public visibility.

UL2 was a 20-billion-parameter encoder-decoder model with a new training objective and the same C4 pretraining dataset as T5. The guest confirms accidentally leaving it running for a month. Its release followed a more deliberate institutional path: presenting UL2 as a larger, updated T5 gave reviewers an established precedent for approval.

As UL2 worked well and gained internal visibility, work to incorporate it into PaLM 2 led toward co-leadership of the modeling workstream. Research initiated from below met appointments made from above: visible contributors were selected to lead. The guest emphasizes the scope of that responsibility—one workstream, in a project with many co-leads.

When the interviewer suggests that timing made the opportunity easy to obtain, the guest pushes back. Fewer people were pursuing the work, but it was still difficult. A career history is not a reproducible recipe: starting conditions differ, luck matters, and the causal path from one project to a later opportunity is hard to isolate.

Emergence: measurement and interpretation

The emergent-abilities paper brings another distinction about credit. The guest credits Jason with most of the original idea and thesis, describing a supporting role in shaping the paper and involving stakeholders. That collaboration also began a friendship and a sequence of further papers.

Both speakers defend emergence, though they do not develop a technical definition here. The interviewer makes the measurement argument: making some benchmark curves linear does not, in that interpretation, eliminate emergence as a broader phenomenon. The disagreement with the Mirage paper is framed partly as a question of magnitude; the exchange does not examine its underlying metrics.

The sharper objection is to using recognition to settle a scientific argument. The interviewer praises the Mirage paper’s execution but argues that its title encouraged overly broad conclusions. The guest objects to people invoking its NeurIPS best-paper award as if that resolved the emergence question, while declining to attribute bad intentions to its authors.

Research taste, public accountability, and visibility

The guest describes learning research taste—the judgment of what makes a problem interesting and a result worth attention—from Quoc Le through repeated informal conversations and brainstorming. The influence accumulated gradually; Quoc figures in the account as an active researcher and friend as much as a manager.

Jason Wei offered a lesson in public accountability. Watching him post provocative opinions and withstand criticism encouraged the guest to move beyond paper announcements. The principle was to stand behind defensible opinions under a recognizable identity, while retracting them when criticism proved persuasive.

Visibility also creates pressure to say less. The interviewer argues that treating every OpenAI researcher’s personal post as a clue to company plans discourages candid discussion. The guest recalls screenshots circulating of Jason expressing excitement about experiments during speculation about GPT-2 chatbots—an ordinary remark being read as a product signal.

The professional lesson is to keep explaining the work after publication. The guest points to Jason’s repeated blog posts and talks about emergent abilities, including a keynote roughly two years later. That sustained effort contrasts with the guest’s earlier habit of releasing papers without much thought about how they would be received.

For researchers starting without an audience, the speakers suggest complementary approaches: - Earn a prospective coauthor’s respect. Do work they find valuable; internships or direct messages can start a relationship. A collaborator with an established research reputation and even a modest audience can help introduce the work. - Take on work a mentor lacks time to do. The interviewer recommends demonstrating reliability through useful contributions, giving a busy mentor reason to support the collaborator’s growth.

Own the training run; keep learning the tools

A story about Hyungwon makes engineering habits concrete: a 13-inch MacBook, one terminal, and rapid keyboard-driven screen switching. He argued that switching views could beat moving one’s head between monitors. The storyteller admired the systematic thinking but found the workflow too painful to adopt; the anecdote offers no measured speed comparison.

Asked what distinguishes a strong AI researcher, the guest emphasizes ownership of training jobs. If a run fails at 4 a.m. on Saturday, restarting it promptly can preserve time that would otherwise disappear before Monday. The guest connects that responsiveness to faster progress and compares exceptional research intensity to Olympic competition, while explicitly calling compulsive overnight debugging unhealthy and unsuitable as a sustained practice.

On technical prerequisites, the answer is flexible. Asked about CUDA kernels, the guest says they do not personally know them, then immediately adds: be willing to learn when the work requires it. Frameworks change, so no single stack or workflow defines an end-to-end researcher. Continuously updating skills is the more durable requirement.

Reka: rented capacity meets operational reality

Reka’s founding relationships grew from collaborations between Google Brain and DeepMind. Co-founder Danny initiated the venture, which was taking shape by late 2022. After prolonged consideration and wrapping up PaLM 2 work, the guest left Google in 2023 to experience co-founding a company. The choice was staying at Google or becoming a founder, with a continuing identity as a scientist and researcher.

Compute delivery challenged the plan. The guest recalls leaving around late March or early April, while most of Reka’s compute arrived only in December. H100 delays left the team working with 500 rented A100s for an extended period. Provider commitments and the difficulty of obtaining substantial capacity in one place limited alternatives.

Delivered hardware did not equal usable training capacity. Poor reliability invalidated the initial calculations connecting hardware capacity to floating-point operations and achievable models. The guest describes having to work ten times harder to keep training moving—an expression of the operational shock, rather than a measured multiplier.

The interviewer proposes returning to Google’s TPU hardware and the familiar software stack. The guest worries that the external environment would differ from Google’s internal one, particularly in codebases and availability; the interviewer disputes the supply impression. Those concerns came from documentation and impressions, since the guest had never run a TPU job outside Google. The immediate decision rested on an existing GPU and PyTorch setup already training models, making migration costly.

One failed node can stop the whole job

Reka described providers as passing through chaotic and stable phases, borrowing language from The Three Body Problem. Newly provisioned nodes could be unreliable. Conditions improved as faulty nodes were taken out of service, returned, and repaired, but incremental deliveries could introduce fresh failures into the cluster.

The financial mismatch is between the hardware that fails and the job that stops. One failed node can halt an entire distributed training run. In a hypothetical 500-GPU job interrupted for four hours, the provider may think in terms of refunding one node while the customer sees lost time across the full job. Both speakers regard making providers liable for every resulting loss as unrealistic.

The guest identifies two ways providers could help: - Share some failure costs. Willingness to absorb part of the loss would signal confidence in infrastructure reliability, without promising reimbursement for an entire run. - Support hot swapping. The guest suggests replacing failed nodes as an operational response, but reports that most providers the team tried lacked the capability.

Refunds still cannot return elapsed time. Frequent failures require monitoring rosters, interrupt engineers’ sleep, and create constant anxiety. The earlier discussion of individual intensity acquires an infrastructure dimension: unreliable systems demand repeated human intervention.

Checkpointing saves training state for recovery, but adds overhead. Reka adjusted checkpoint frequency according to observed job stability and filesystem performance. The guest roughly recalls a checkpoint taking around 30 minutes for a 20-billion-parameter model under slow file input/output, explicitly expressing uncertainty about the figure.

Recovery engineering involves several costs: - Training slowdown: assess average utilization of floating-point computation; the guest suggests that a 1–2% speed penalty can justify more frequent checkpoints. - Restart delays: automatic restart recovers some time after failure, but does not eliminate losses. - Filesystem and storage pressure: the interviewer argues that storage is cheap relative to compute; the guest counters that frequent checkpoints for very large models can overwhelm available resources.

A final infrastructure disagreement concerns orchestration across multiple clusters. Drawing on experience with Temporal, Kubernetes, and Airflow, the interviewer argues that suitable tooling should already exist. The guest says Reka uses Kubernetes for several purposes but had not had time to build a complete solution for experimentation. Available tools still left integration work for the team.

Confidence in reaching the goal through iteration

Part 1 closes with the question of model quality: could the team know before training that it would reach frontier performance? The guest describes confidence in making progress with fewer iterations, grounded in prior training experience, rather than certainty about a capability level by a particular date.

Smaller experiments reduced training risk. Erratic loss in an exploratory run could warn that the setup was heading toward a poor result. Compute constraints limited evaluation, but the guest says the team ran enough experiments to gain confidence in its infrastructure and overall setup. The first attempt was still uneven, and the field continued moving during the work.

The closing example is hypothetical: observe an initial human-evaluation score, then believe the team can reach the desired level in roughly five more tries. The confidence lies in knowing how to navigate the next iterations, informed by experience and smaller tests.