The End of Finetuning: Reusing Capabilities Without Erasing Them

Start with a model that already sees

An image classifier has already learned to recognize edges, gradients and corners. Why should every practitioner spend data and compute learning those features again? Transfer learning reuses that work: take a trained model and fine-tune it—continue training it—for a new task. Jeremy Howard presents this as fast.ai’s central strategy for making deep learning accessible.

Howard dates fast.ai’s initial thinking to around 2015 and practical work to 2016. He recalls expertise concentrated in a few labs, unpublished training tricks, scarce software and datasets, and difficult compute setup. Theano and its Lasagne wrapper offered a starting point. He and his wife were unsure whether they could make deep learning accessible, but considered restricting such a consequential technology to a small group both wasteful and dangerous.

The Practical Deep Learning courses put transfer learning in lesson one. Howard contrasts that choice with the recorded PhD classes he encountered, where it received little attention. The curriculum focused on vision, tabular data, recommendations and text: areas where learners could put the technique to useful work.

Sentence completion as a route to transferable knowledge

Howard found the natural language processing practice around him too focused on academic demonstrations and elaborate preprocessing. His alternative drew on an interest in cognitive science: the Chinese room thought experiment, where a person follows symbolic rules to answer Chinese questions without speaking Chinese. He found it compelling that the overall system could nevertheless understand the language. The practical question interested him more than the philosophical verdict: could learning patterns from enough text produce intelligent-seeming responses?

Wikipedia made that question approachable. A language model trained to complete its sentences might need representations of people, objects, time and causal relationships. Howard’s example is a sentence naming the US president who signed a law in a particular year: completing it requires connecting an officeholder with a date. His goal was to develop latent capabilities—abilities learned through sentence completion that later training could draw out for another task.

ULMFiT organized this approach into three stages: 1. Train a language model on a large text corpus. 2. Adapt it to a more curated corpus. 3. Further adapt it to a specific task. Howard reports reaching a new state-of-the-art IMDB result within hours of his first attempt, though he gives no score or evaluation configuration.

He had delayed the experiment because colleagues confidently rejected transfer learning for language. He acknowledges earlier language-model fine-tuning by Dai and Quoc Le at Google, distinguishing their domain-specific corpus from his general-purpose approach. He also recalls Alec Radford later saying that ULMFiT encouraged him to try general-corpus pretraining at greater scale, with better results.

Howard argues that enthusiasm for GPT’s zero-shot and few-shot results diverted attention from fine-tuning for years. Frustration with promoting transfer learning contributed to his stepping away from language research. He sees later instruction and behavioral tuning as a return to staged adaptation: general training, then instruction data, then a further behavioral objective. His comparison concerns the sequence of training stages, rather than a precise equivalence between their implementations.

Research that reaches practitioners

Howard deliberately prioritizes methods that work with modest compute budgets. Most practitioners lack large computing installations, yet he worries that attention to those installations draws talent toward already wealthy organizations. Earlier business success gave him room to choose that priority: he describes selling FastMail and Optimal Decisions after ten years. The businesses addressed synchronized consumer email and profit-optimized insurance pricing, respectively.

The ensuing reflection separates commercial achievement from social value. Synchronized email seemed useful but likely to emerge anyway; helping insurers extract more money prompted discomfort. Financial independence allowed a reconsideration of what deserved further effort. Experience at Enlitic suggested an educational strategy: instead of trying to solve medicine within one organization, equip passionate domain experts to improve their own fields.

Fast.ai combines research, software and courses. Its projects include the fastai library, nbdev, FastCore and GHAPI. The organizing principle is to turn research into something people can learn and use. ULMFiT followed that path: the paper grew from course material after Sebastian Ruder proposed publishing it and offered to write it.

GHAPI applies the same philosophy to everyday development. It derives its interface from GitHub’s official OpenAPI specification, exposing methods with autocomplete in notebooks and on the command line. Brief documentation and links to fuller documentation accompany the operations. An EC2 utility similarly completes instance names. Small conveniences make recurring work easier to explore and more enjoyable.

The DAWNBench competition put this efficiency agenda under pressure. Concerned that a major Google effort would make fast training appear dependent on specialized hardware and large organizations, a team including Rachel and students assembled an entry with ten days remaining. They applied progressive resizing, described as training mainly on smaller images, and used non-square images. The speaker reports a win as a demonstration that small teams could compete; the precise category, hardware, training time and accuracy threshold are unspecified.

The loss curve that refused to be a bug

A Kaggle competition supplied a new puzzle: answer multiple-choice questions about Wikipedia within nine hours, using 14 GB of RAM, two CPUs and a small, old GPU. While experimenting with Jono Whittaker, one participant noticed that training loss declined gradually, then dropped abruptly between epochs, or passes through the training dataset. This investigation is discussed alongside “Can LLMs learn from a single example?”.

The first suspicion was a training-loop bug: perhaps validation was accidentally updating the model, or a moving average was mishandled. Switching from Hugging Face Trainer to fast.ai Learner did not remove the drops. Other practitioners reported the pattern with different datasets and trainers. That weakened the case against one implementation, but calling the behavior normal did not explain why an epoch boundary should change learning.

Other practitioners proposed single-pass memorization: the model had learned the examples well enough that seeing them again gave it an immediate advantage. Initially skeptical, the researchers designed experiments with predictions stated in advance about when the effect should appear. They report that all supported the hypothesis, while explicitly crediting others with observing the pattern and proposing the explanation. Without model sizes, dataset sizes or experimental manipulations, the account leaves its range of applicability open.

The proposed mechanism returns to ULMFiT: pretraining may make new text easy to compress because the model recognizes it as a variation on familiar patterns. Further training can then increase confidence enough to worsen validation loss even while accuracy improves. The recommendation is to track validation accuracy in that situation, rather than infer progress from loss alone. Rapid memorization might enable useful behavior, but it could also weaken subsequent learning signals or disrupt training dynamics; no settled fine-tuning recipe follows from it yet.

Adaptation must preserve what the model already knows

Catastrophic forgetting is the accompanying risk: a model gains a specialization while losing previous capabilities. The discussion offers CodeLlama, described as a 500-billion-token adaptation of LLaMA 2 on code and prose about code, as an example. The speaker attributes broad non-code losses to retaining too little general-purpose data. This remains an assessment without comparative benchmark results; the stated non-code proportion also has an unclear denominator.

That concern changes the requirements for staged training. ULMFiT’s original destination was a narrow task such as sentiment classification. A general assistant must preserve many kinds of competence. Applying the same sequence unchanged can therefore be inappropriate: what must survive adaptation matters as much as the new behavior being taught.

The proposed alternative is continued pretraining with a changing data mixture. Include instructions, exercises, code and general document completion from the beginning. Gradually raise quality and increase task relevance while retaining substantial amounts of every desired data type. Filtering out poor examples is compatible with this approach; dropping whole categories removes support for capabilities the assistant still needs.

On this view, some tradeoffs between coding and general ability—and some capability losses associated with alignment—may come from training choices rather than unavoidable limits. A better mixture is a proposed remedy, not a demonstrated universal solution. The broader criticism is that publication incentives favor recognizable incremental improvements over questioning the prevailing recipe.

Join research communities through useful work

Howard says apparently quiet AI Discord communities may have substantial activity in private channels. He created one after years of public discussion, gathering consistently helpful contributors and reducing speculative conversations about artificial general intelligence that were unaccompanied by attempts to train models.

His advice is practical: contact an administrator, moderator or developer, share relevant GitHub work or writing, and explain what you want to discuss. He recommends fast.ai as an accessible entry point, while noting that Eleuther’s long-running discussions assume considerable shared context. Above all, complete a small useful task and follow up. Beginners who do that stand out and give others a reason to support them.

The medium shapes those interactions. Howard finds that persistent forum threads encourage thoughtful contributions, while chat encourages expectations of immediate answers. His forum had become less active as Discord grew more popular, leaving a tension between where people gather and the discussion an interface encourages.

Compiler ideas need usable tools

Chris Lattner supplied Howard with a memorable example of follow-through. After listening to roughly half an hour of problems Howard saw in AI, Lattner returned months later with responses and proposed projects. One was Swift for TensorFlow, with automatic differentiation built into the compiler. Howard recalls learning that compiler optimization of the backward computation could reduce the context saved during the forward pass.

Those possibilities changed Howard’s understanding of what could be improved, but he insisted that adoption would require usability. Lattner suggested a fast.ai-like interface and offered guidance while Howard learned Swift. The collaboration developed through two months of numerical programming experiments, performance reports and supporting libraries, eventually producing joint lessons and a fast.ai-like library.

The discussion treats Google’s decision to discontinue Swift for TensorFlow as understandable given the difficulty of asking users to learn another language. Mojo renews the prospect of combining compiler capabilities with a language designed for numerical computing. It is presented as building on Lattner’s compiler and runtime work, with the hope of offering JAX-like benefits. In this October 2023 conversation, that hope remains explicitly speculative: creating a new language is difficult.

Combine retrieval with adaptation

Consider asking a collection of research papers for every way to make inference more efficient. Similarity search might find a review that explicitly lists methods while missing individual techniques scattered across separate papers. This is the criticism of retrieval-augmented generation (RAG), which retrieves text to help answer a question, when it relies on arbitrary document chunks and embedding similarity. Finding passages similar to the question does not necessarily assemble a complete answer.

The proposed direction combines retrieval with fine-tuning that incorporates information into model weights. When another participant questions whether fine-tuning can introduce new information, the response returns to continued pretraining: training introduced knowledge originally, so further training should be able to introduce more. This is a conceptual case for investigating the combination, rather than a measured comparison with retrieval alone.

A SQL-generation demonstration makes adaptation concrete: a pretrained base model reportedly learned to generate SQL after fifteen minutes of training on one GPU. The speaker interprets this as drawing out a latent capability; model specifications, GPU details and accuracy are unspecified. BTLM 3B and coding models in the 1–2B range receive similar attention as promising examples of small models handling complex tasks, without a benchmark ranking.

Phi-1 and Phi-1.5 sharpen the distinction between demonstrating an ability and providing an application foundation. Phi-1 is described as good at short Python snippets. Phi-1.5 earns praise for reasoning, math and coding learned from a nearly entirely synthetic dataset, alongside criticism of its limited world knowledge. Another participant identifies the practical difficulty: low resource requirements are appealing, but uneven task performance makes it hard to know where to trust a small model. Fine-tuning for repetitive enterprise work is proposed as an opportunity, without a demonstrated software-replacement result.

Working systems have outpaced understanding

AI coding assistants open another accessibility frontier. One speaker describes people who had barely programmed a year earlier now training language models with Codex and Copilot. They need teaching that fills the gaps around what they can already build. Removing fast.ai’s prerequisite of a year of full-time programming is raised as a possible course redesign, rather than an announced change.

The wider assessment is that useful recipes have outpaced understanding of training, adaptation, retrieval, model sizing and prompting. A secondhand anecdote about roughly 6,000 lines of Python built around GPT-4 for chess raises a related question: how much apparent capability comes from the surrounding software? The reported engine results lack code and an evaluation protocol, so they do not establish GPT-4’s own chess rating.

The comparison is with early deep-learning vision: useful systems arrived before clear explanations of their internal representations and training dynamics. Rapid commercialization is treated as economically rational, while leaving technical debt and opportunities to reduce runtime and memory use.

FlashAttention prompts disagreement about the source of such opportunities. One participant recounts Tri Dao attributing an overlooked optimization to a lack of systems expertise. Another argues that the opportunity was known and the difficult achievement was implementing it well. Triton helps with GPU optimization, but still requires detailed architectural knowledge and CUDA-like programming. The hope for Mojo or an equivalent tool is that more practitioners could turn optimization ideas into working implementations.

Understand learning, preserve capabilities, widen participation

In the closing lightning round, Howard says his forecasts were slightly too bullish. He recalls predicting in a 2014 TED Talk that computers might surpass average humans at most tasks within five years. At the time of this conversation, he considers that perhaps nearly true for nonphysical tasks, roughly twice as late as expected. Yet his first experience of GPT-4 was visceral: anticipating a capability felt different from using it.

His central unsolved question is how language models learn. He wants explanatory tools that connect model internals and training behavior to better learning methods. He recalls work visualizing how architectural changes affected a convolutional network’s loss surface, and the discussion cites a PyTorch matrix-multiplication visualizer as another promising tool for understanding computation.

The final technical agenda turns that ambition into concrete research questions: - What data does a model need, at which stages, and how often? - Which mixtures preserve capabilities during adaptation? - Do some foundational capabilities support many others, making their loss especially costly? - Could changing the loss function reduce forgetting? A better training recipe must account for both new learning and retained competence.

The closing appeal returns to people with diverse experiences and abilities. Drawing parallels with access to writing, printing and education, the speaker favors enabling more people to contribute over concentrating powerful tools among a trusted few. The hope is that ordinary practitioners can produce valuable work, improve safety and help future generations flourish. Another participant connects that goal to supporting open-source developers and communities. Broad participation is the purpose behind the effort to make training cheaper, software usable and deep learning teachable.