Answer.AI’s small-team bet: reuse models, remove friction, build better tools

Expect more from an already trained model

The earlier episode’s “end of fine-tuning” framing had obscured Jeremy Howard’s argument. As Alessio acknowledges the editorial exaggeration, Howard clarifies the practical problem: developers underestimate how much they can change an already trained model. His preferred starting point is to adapt existing weights; training from scratch needs a compelling reason, such as the absence of a model trained on broadly similar data.

Howard proposes training as a continuum. Initial pretraining, instruction tuning and later task or preference training need not be sharply separated. He recommends retaining more of the original dataset in later stages and letting those stages run longer, pointing to Llama 3 as an example of that direction.

Alessio offers Snowflake Arctic as a concrete example: its three training phases begin with roughly 75% web text, then reduce web text and increase code. Howard would extend that progression into a function evaluated every batch. Just as a learning-rate schedule can vary continuously instead of jumping between preset values, the training mixture can change throughout the run.

This preference for control explains his lukewarm response to schedule-free optimizers. He likes experimenting with additional training parameters, even scheduling Adam’s epsilon. But optional control should not require configuration: fastai aimed to supply useful defaults without demanding schedules or even a learning rate, while leaving those controls available.

The original conversation was recorded more than a month before release, initially to coincide with FastHTML’s launch. The projects and plans that follow reflect that stage of the team’s work.

Put the mission into the company’s rules

Howard sees corporate governance as another unresolved alignment problem. His critique of OpenAI centers on conflicting incentives: a nonprofit board nominally controlled a commercial organization whose employees expected equity-related gains. In his analysis, that structure made it difficult to enforce safety decisions that reduced profits.

Answer.AI was deliberately established as a Delaware C corporation with a commitment to maximizing long-term value. For its second funding round, an angel round, investors had to use a shared investment vehicle and agree to vote according to long-term value principles. The rationale was practical: verbal support for a mission can disappear when a particular decision sacrifices short-term returns. The commitment therefore belongs in legal documents.

The company also became a Public Benefit Corporation (PBC), incorporating a stated public benefit—maximizing societal benefit through AI—into its corporate documents. The guest presents this as protection against a mission-breaking acquisition. His hypothetical buyer offers a 50% premium to repurpose the technology for marketing cigarettes to children; in his legal interpretation, PBC status provides grounds to reject the offer despite its financial appeal.

Self-direction needs shared judgment

Recruiting starts with distinctive work, followed by a conversation about why its creator chose that problem or approach. Among comparably successful candidates, the guest favors people who succeeded through constraints and unconventional routes. He associates that experience with creativity, tenacity, openness and willingness to take risks.

Complementary expertise creates its own friction. Nearly everyone on the team had reportedly expressed impostor syndrome in the preceding weeks: developers were intimidated by researchers’ mathematics, while researchers admired developers’ ability to ship useful tools. The response was to make that asymmetry explicit. Being weaker at a colleague’s specialty is compatible with contributing valuable strengths; learning across specialties is a reason to join a research and development lab.

Answer.AI’s experiment removes managers and managerial hierarchy. The guest includes himself in that restriction: he can help, but cannot dictate what colleagues do, how they do it or when. BERT24, a collaboration to improve the BERT model family, illustrates the arrangement. Ben Clavié independently gathered people who had developed improved BERT models in the company’s Discord collaboration area. Benjamin Warner adapted a transformer implementation originally built for teaching into a readily modifiable BERT implementation; collaborators then added improvements and dataset mixtures.

Kerem’s adapter-training work supplies another example. Within a reported two or three weeks of the team’s earlier release, he converted its implementation to DoRA, added vLLM extensions and produced benchmarks for FSDP QDoRA. The guest reports training speed matching LoRA and performance exceeding a fine-tuning baseline. Without model, hardware, dataset or evaluation details, those remain workload-specific reported results.

Optional meetings spanning pairs of major time zones help colleagues discover projects and volunteer to learn or contribute. Austin’s work on making WebGPU programming easier prompted just such an offer. Broad freedom rests on a small set of shared priorities: - Reduce cost and complexity. - Make better use of existing models through fine-tuning. - Resist centralization. - Pursue research that might otherwise go undone and can produce practical outcomes.

Every recruiting candidate meets everyone in the company, a process considered manageable at the target size of 12. At recording time, all but one person entering the pipeline had been hired, and nearly all were already known through their work. Jono Whitaker’s course contributions and Benjamin’s fast.ai community work supplied that evidence. Clavié was an exception to prior acquaintance: the quality of his code, and the distinctiveness of what he chose to build, prompted the conversation.

Revisit architectures that fashion left behind

BERT24 connects that organizational freedom to a technical bet: BERT remained useful in industry while decoder-only models attracted disproportionate attention. With input from Colin Raffel, the group was considering fine-tuning a pretrained T5 encoder backbone and testing a full encoder-decoder model as a BERT baseline. These were proposed experiments, not announced results.

The architectural intuition starts with the task. For translation, an encoder builds a representation of the source sentence for a decoder to consume when producing output. For classification, the output need not be an arbitrary-length sequence of tokens, motivating an encoder-only model. The guest offers this as an intuitive, explicitly non-expert case for investigating alternatives.

He acknowledges that decoder models can compete with DeBERTa V3 in Kaggle competitions, while attributing their competitiveness to larger models and greater training investment; he supplies no controlled comparison. His broader strategy is to revisit promising methods that received little follow-up. A differently trained model might have commercially valuable strengths on particular tasks, but identifying those tasks remains part of the research.

Make the stack work—and measure it

The FSDP–QLoRA integration brings the discussion back to engineering. The training arrangement keeps the main model weights quantized and trains separate floating-point adapters to modify its behavior. Those adapters can then be distributed for use with the quantized base. Combining this arrangement with PyTorch’s FSDP library was the integration challenge; the discussion does not explain FSDP’s sharding mechanics. The guest built an initial proof of concept before the company launched, then worked with Benjamin, Jono and Kerem to make it usable. Much of the effort went into understanding poorly documented internals and getting the pieces to cooperate.

In his account, the quantization library bitsandbytes contained unused paths, duplicate implementations and CUDA code that was difficult to step through. PyTorch’s FSDP library required reading its implementation, while coupling among bitsandbytes, Hugging Face’s PEFT library and other components made isolation difficult. Creating a minimal script required substantial rewriting. Meta’s Llama recipes supplied a useful FSDP-and-LoRA example, but it did not support QLoRA.

Running successfully was only the first test. Benchmarking repeatedly exposed important cases where apparent progress increased memory consumption or slowed execution. Some failures came from dependencies: one Transformers regression reportedly broke FlashAttention for a particular set of models, and Benjamin traced it to an accidental change. Memory and speed measurements were essential to deciding whether the integration actually worked.

FlashAttention illustrates the gap between a compact idea—computing attention in tiles—and efficient execution across multiple systems layers. Inference work with vLLM and the HQQ quantization library brought similar integration challenges because both relied on custom kernels. Much of Answer.AI’s inference work remained unreleased at recording time, alongside overlapping performance work in fast.ai, the CUDA mode community and PyTorch’s Torch AO quantization project.

The preferred distribution artifact is a merged adapter over a quantized base: keep the main model weights quantized and distribute the floating-point adapter weights separately. This clarifies the guest’s objection to distributing complete merged models. If the recipient already has the base, the adapter is a smaller download. Keeping the main weights quantized also reduces the need to load and transfer full-precision weights. He expects faster inference and fine-tuning from this arrangement, but gives no measured speedup.

FastHTML addresses the application bottleneck

Answer.AI’s ambition extends beyond models: create thousands of commercially successful products with roughly 12 people. That is a goal, not a shipped-product count. Reaching it, the guest argues, requires a much more efficient stack for creation, deployment and maintenance. For a data scientist, learning frontend frameworks, backend development, security and communication protocols can become a substantial barrier to sharing a useful application.

His reference point is the immediacy of putting a short PHP script on a web host or a Perl file in cgi-bin and sharing its URL. FastHTML aims to recover that ease with a complete application in one Python file, working directly with web foundations. It targets the feel of a modern single-page application without requiring Jinja templates or separate CSS and JavaScript files.

FastHTML combines existing tools: Starlette provides the underlying web platform, HTMX is an optional part of the integration, and the interface follows FastAPI as closely as possible. Python components can wrap libraries such as Bootstrap, Shoelace or Tailwind, be installed through pip, and be developed interactively in Jupyter notebooks.

The described FastLight integration adds SQLite persistence by automatically passing database-ready objects to handlers for insert, update and delete operations. Session and security support are also mentioned, though their protections are not detailed. PicoCSS supplies default styling so users can begin without first assembling a stack of components.

The design draws on the guest’s earlier FastMail framework work and reprises the training-library philosophy: useful defaults with room to take control. He imagines a Gradio- or Streamlit-like layer on top of FastHTML, allowing users to move gradually toward the underlying web primitives as their applications grow. That higher-level package was a future possibility, intended to avoid a wholesale rewrite when an abstraction becomes limiting.

Craft a dialogue that produces an artifact

AI Magic brings AI into the development workflow itself. The in-development system is organized around dialogue engineering: crafting a conversation whose outcome is an artifact, such as an analysis, Python library or technical article. Its creator reports doing most of his work in it and becoming substantially more productive, without a quantitative comparison.

The interface argument identifies a gap between two familiar environments: - Chat applications follow a type–response–repeat loop. Hiding generated code can help complete beginners, but gives learners and programmers less direct control. - AI-assisted editors, such as VS Code and Cursor, add conveniences inside an already complex development environment. AI Magic is intended to sit between them, making the production of artifacts more interactive.

It is built on Claudette for Claude and Cosette for OpenAI, libraries intended to make the providers’ APIs more concise and usable. Claudette takes a provider-specific approach, in contrast to Simon Willison’s broadly provider-supporting LLM library. One capability worth preserving is assistant prefill: supplying the beginning of Claude’s response to guide its continuation. The goal is compatibility where useful without erasing provider differences.

A proposed course, How to Solve It With Code, takes its name from Polya’s How to Solve It. It would pair AI assistance with practical programming skills to address a specific failure: someone builds an application with ChatGPT, then cannot maintain, repair or extend it when the model falls short. The audience runs in both directions—programmers learning to use language models, and people who began with language models learning the process and fundamentals needed to sustain their work.

The closing outlook is aspirational: an international FastHTML community, wider adoption of AI Magic and dialogue engineering, and more people using these tools to build.

Reuse context and develop the answer before generating code

The final technical discussion starts with a concrete waste: repeatedly processing the same book or library documentation for every prompt. Jeremy wants the KV cache, the model’s cached context, to become a reusable application resource. For a new library such as FastHTML, he wants to upload the documentation once and keep it available throughout an application-building conversation. He was exploring that workflow for AI Magic.

This creates a broader design question: when should an application use retrieval, fine-tuning, persistent cached context or model state that can be updated? Jeremy points to state-space models and xLSTM as approaches to updatable state. How to combine these techniques remains open.

Asked about JEPA and diffusion for text, Jeremy focuses on the problem they might address: developing a sketch of an answer before emitting its tokens. He describes diffusion as gradually refining a picture of the solution, rather than generating it autoregressively, one token after another. He leaves open whether either approach will succeed for text.

Dialogue engineering serves a related purpose in his own workflow: use the conversation to develop the thought process before generating code. That is a personal assessment, not a measured comparison, but it makes the intended workflow concrete. The dialogue helps shape the answer before the final artifact is produced.