Fast.ai

Fast.ai is a deep-learning research, software and teaching lab created by Jeremy Howard and Rachel Thomas to help domain experts use AI themselves. Its work includes the free Practical Deep Learning for Coders courses, the fastai library built on PyTorch, and the notebook-development environment nbdev. Research feeds into both software and teaching: ULMFiT addressed transfer learning for text classification, while its DAWNBench work explored faster and cheaper image-model training.[2][5][7][3][4]

That combination of teaching and tools dates to the lab's 2016 launch, following Howard's work on deep learning for medicine at Enlitic.[2]

Howard announced fast.ai on October 7, 2016, approximately six months after resigning as Enlitic's CEO. He and Thomas created the lab around the view that domain experts understood their fields' important problems, had access to relevant data, and knew the constraints on implementing solutions. The announced program combined practitioner education, automated model-training tools, software for understanding training and results, and example applications in areas where deep learning was poorly utilized.[2]

In his October 2023 retrospective, Howard described transfer learning as the initial practical strategy: reuse a model that has already learned useful features, then adapt it to a new task, reducing the data and compute needed to begin. He recalled that transfer learning appeared in the first lesson. The teaching covered vision, tabular data, recommendation systems and text—four kinds of problems the founders considered broadly useful.[1]

Jeremy Howard and Sebastian Ruder's Universal Language Model Fine-tuning for Text Classification, first submitted on January 18, 2018, introduced ULMFiT. Howard later explained its three stages as training a language model on a large general corpus, adapting it on a more curated corpus, and fine-tuning it for a specific task. He recalled that Ruder approached him after seeing the course and offered to write the paper.[3][1]

The ULMFiT paper reported improvements over the previous state of the art on six text-classification tasks, with error reductions of 18–24% on the majority of datasets. It also reported that using only 100 labeled examples matched training from scratch on 100 times more data. The authors released pretrained models and code.[3]

Fast.ai's 2018 DAWNBench project turned classroom discussion into a collaborative training experiment. Howard credited Leslie Smith's super-convergence research and reported that student Sylvain Gugger implemented it within 24 hours of its discussion in class. Andrew Shaw automated experiments on AWS spot instances and integrated NVIDIA's half-precision training ideas; Brett Koonce explored hyperparameters. Ben Johnson developed an independent student entry that came a close second in both CIFAR-10 training categories.[4]

Howard reported that fast.ai won the CIFAR-10 training-speed and training-cost categories, reaching 94% accuracy in under three minutes with a modified ResNet-34 on multiple GPUs. Its cost-winning Paperspace run cost $0.26. For ImageNet, he reported the fastest training on publicly available infrastructure, on GPUs, and on a single machine. His account distinguished actual expenditure from DAWNBench's official cost calculation, which assumed on-demand pricing.[4]

The ImageNet work used progressive resizing: start training with small images, then increase their size as training proceeds. The reported setup began with 128×128 images and used 288×288 images in the final epochs. Small images made early training faster; larger images supplied detail later. Howard also reported improved generalization from exposing the model to different image sizes.[4]

On October 2, 2018, the organization released fastai v1, a free open-source library built on the PyTorch v1 preview. Fast.ai had previously done most of its research and teaching with Keras using TensorFlow. The new library supplied a consistent interface across vision, text, tabular data, time series and collaborative filtering, while extending PyTorch's API and keeping its underlying operations available for customization.[5]

An early application was Hamel Husain and Ho-Hsiang Wu's experimental semantic code-search tool at GitHub, which found code snippets from questions written in plain English. Fast.ai's release account reported that they switched from TensorFlow Hub to fastai for access to AWD-LSTM architectures and cyclical learning-rate techniques. The release also credited Gugger with much of v1's development, alongside contributions from alumni and other collaborators.[5]

Howard and Gugger's February 2020 layered-API paper described fastai v2 while it was still in prerelease. Its architecture combined high-level components for practitioners with composable lower-level components for researchers. Semantic tensor types and dispatch, a data-block API, decomposed optimizer components, and two-way callbacks supplied the building blocks. The callbacks could inspect or change data, models and optimizers during training, allowing users to customize behavior without rebuilding the entire framework.[6][7]

Fast.ai released the rewritten v2 on August 21, 2020, alongside fastcore, fastgpu, a new Practical Deep Learning for Coders course, and Howard and Gugger's O'Reilly book, Deep Learning for Coders with fastai and PyTorch. Practical Data Ethics had been released earlier that week. Howard credited Gugger's close collaboration over the preceding two years and Thomas's teaching of the data-ethics lesson and development of much of the book's ethics material.[7]

The 2020 course began with a complete working model, then progressively explained and rebuilt its components. It combined machine learning and deep learning with deployment, including web interfaces for model-powered applications. Its stated prerequisites were high-school mathematics and a year of coding experience, preferably in Python; students could complete it for free using hosted GPU notebooks.[7]

By the 2020 release, fast.ai used nbdev, a literate programming environment built on Jupyter Notebooks, for all its projects. Code and documentation shared an interactive format: documentation pages could be opened as notebooks for experimentation and customization.[7]

In August 2022, nbdev2 addressed a concrete collaboration problem: ordinary Git conflict markers could invalidate a notebook's JSON, while execution counts and metadata created unnecessary differences. Its merge driver compared cell source rather than raw lines, displaying conflicting versions as separate cells that could be resolved in Jupyter. A save hook removed unnecessary metadata and nondeterministic output.[9]

The solution followed several earlier approaches. Howard and Stas Bekman had developed Git smudge and clean filters; Gugger created the v1 tool nbdev_fix_merge. Howard rewrote v2 using Python's SequenceMatcher, and Wasim Lorgat proposed moving cleanup into a Jupyter save hook and conflict handling into a Git merge driver. Howard reported that some fast.ai repositories had many hundreds of contributors, making notebook collaboration important to the lab's workflow.[9]

In October 2023, Howard reconsidered separate fine-tuning stages for general-purpose language models. His preferred approach was “There's only continued pre-training.” He advocated retaining substantial amounts of all relevant data types—such as instructions, code and general documents—while gradually improving quality and task specificity. This was his proposed response to catastrophic forgetting, rather than an established universal replacement for fine-tuning.[1]

Howard also described experiments with Jono Whittaker prompted by abrupt loss drops at epoch boundaries during language-model fine-tuning. He said their experiments supported the hypothesis that models were memorizing the training dataset after a single pass. He explicitly credited others with observing the phenomenon and proposing that explanation; the consequences for training remained an open question in his account.[1]

Howard described fast.ai as self-funded in August 2020, saying its costs were paid “out of our own pockets” and that it took no grants or donations. Earlier announcements separately acknowledged AWS credits for the 2018 DAWNBench project and AWS sponsorship of Gugger as its first Researcher in Residence.[7][4][5]

The October 2018 announcement reported that hundreds of thousands of people had taken Practical Deep Learning for Coders. Latent Space's October 2023 introduction reported more than 25,000 GitHub stars for fastai. In that interview, Howard said the lab had operated with at most two people and was then just him—a description of the lab's operating team, alongside a much broader community of students and software contributors.[5][1][9]