Practical fine-tuning with Axolotl

From mismatched datasets to repeatable experiments

Wing Lian wanted to mix Hugging Face datasets, but each used a different prompt format. Repeating an experiment also meant copying a long command line and editing its training settings, or hyperparameters. These problems became the starting point for [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl), a tool for fine-tuning language models. Lian moved settings into YAML files to make experiments easier to reproduce and move between environments.

In this December 2023 conversation, Lian is Axolotl’s maintainer and founder of the Open Access AI Collective. His programming career began with scraping apartment listings and building advertising-supported pages, which he recalls earning about $1,000 a month during college around 2004–2005. He later worked on blood-industry software, games at Social Gaming Network, and music products at United Masters and SoundCloud. By the interview, he had left SoundCloud and was spending more time on Axolotl.

A skiing injury in mid-March had given him the time to learn model training with the Alpaca LoRA repository. Dataset mixing and configuration management followed from those experiments. Lian reports that Axolotl then integrated QLoRA, a memory-saving training method explained below, seven days after its announcement. Making new methods reusable attracted users and established the project’s development loop: build models, discover missing functionality, improve the trainer, repeat.

What fine-tuning adds—and leaderboards can hide

A base model, in Lian’s introductory explanation, is a general text-completion system trained on a large corpus. Fine-tuning continues training on a chosen dataset to adapt it toward conversation, instruction following or a particular task. A host offers concrete targets: JSON output, shorter answers or a humorous response style. General-purpose community fine-tunes can themselves become starting points for business-specific adaptation.

Choosing that starting point requires more than reading a leaderboard. Benchmark contamination occurs when evaluation questions enter training data, allowing a model to learn test answers without acquiring the general ability being tested. Dataset mixtures can spread the problem indirectly: material enters one collection, then another inherits it through a merge. The participants describe accidental leakage and suspicions of deliberate leaderboard gaming, while acknowledging how difficult contamination is to establish.

The many model names become easier to follow as successive experiments in data curation. Capybara and Puffin are examples of releases shaped by particular curation techniques. The lasting output includes datasets, recipes and training skills as well as model weights. Local deployment supplies another motivation: a participant describes OpenHermes as offering roughly GPT-3.5-like quality offline on a laptop—an informal assessment without a specified benchmark or hardware configuration.

SlimOrca: validate answers before adding more

Orca illustrates how the content of training examples matters. It sampled tasks from Flan, asked GPT-4 for step-by-step explanations, and trained a smaller model on those responses. OpenOrca attempted an open reproduction because Microsoft Research had not released the original dataset.

The initial OpenOrca effort had not systematically validated GPT-4’s answers. SlimOrca added a filtering procedure: 1. Match each generated answer with its original Flan reference answer. 2. Give both to GPT-4 and ask whether the generated answer is mostly correct. 3. Filter the dataset and retrain.

The GPT-4-answer portion reportedly shrank from roughly 800,000 to 500,000 rows, retaining approximately the original performance. The speaker recalls a difference of about 0.1%, without specifying the benchmarks or exact measure. In this experiment, filtering questionable answers preserved performance with substantially less training data.

Compute cost also depends on how many experiments it takes to find useful settings. A rough estimate for Mistral 7B was 40 hours on eight A6000 GPUs, each with 48 GB of GPU memory, at a quoted cost around $500. That assumed a successful first run; the dataset and configuration were unspecified, and the spoken cost calculation did not reconcile with the total. Finding suitable hyperparameters commonly took three or four experiments, according to the speaker.

Axolotl supports small benchmark subsets during training to show whether capabilities are improving or declining. These are trend indicators, not full benchmark scores; full evaluations take substantially longer. A participant describes viewing the trends alongside hyperparameters in Weights & Biases. During a 40–50-hour run, an intermediate check can reveal an unproductive direction before more compute is spent.

Model selection includes the cost of testing

Access to weights does not guarantee affordable evaluation. The conversation describes a 120-billion-parameter model assembled from Llama 2 70B, tentatively identified as Goliath 120B, with Disco as its fine-tune. The speaker does not recall the exact stacking or merging procedure and has not tried Disco personally because of its GPU requirements. Its favorable reputation is anecdotal.

Datasets can outlast a model’s competitive position. OpenHermes carried a cleaned dataset from Llama training to a Mistral 7B base, where a participant reports substantially better results. Older model weights still matter because companies cannot replace deployments every week. Meanwhile, the compute requirements of releases such as Falcon 180B limit how many people can independently test them.

Informal recommendations need scrutiny too: several people may repeat one opinion rather than conduct independent tests. Benchmarks and hands-on use offer complementary evidence. The discussion clarifies that LMSys uses Elo rankings based on people choosing between model answers, while noting that arena visitors are a self-selected population. A business must assess whether the benefits justify testing and integration costs; size and novelty alone do not establish value.

Training-data provenance adds a separate concern. GPT-4 can write candidate answers or rank answers produced elsewhere. In the second pattern, it supplies preference labels: present a question with several responses and ask which are better. That distinction explains how the data was made, but does not settle permission to use it. The participants explicitly recommend checking licenses; their discussion leaves fair use, enforcement and restrictions across later generations of synthetic data unresolved.

LoRA reduces the state that training must keep

Model releases also tested the training product itself. [Manticore](https://huggingface.co/openaccess-ai-collective/manticore-13b) reportedly found an early audience in roleplay communities. Building it helped expose missing functionality, improve speed and turn debugging into useful model output.

LoRA makes such experimentation more accessible by freezing the base model’s weights and training a separate set of added weights, called an adapter. Inputs pass through the frozen base and the added weights; training updates only the latter. The adapter can then be merged into the base without changing its architecture. The base weights still occupy memory, but fewer trainable parameters mean less memory for gradients and optimizer state.

Related methods change different parts of this process: - QLoRA trains adapters with base-model weights quantized—represented at reduced precision—to four bits. - ReLoRA repeatedly trains an adapter for a chosen number of steps, merges it into the base, and restarts adapter training so the optimizer can seek a new direction. - GPTQ is a specific weight-quantization method, discussed here for roughly four-bit weights.

ReLoRA’s attraction was the prospect of gains resembling full fine-tuning while training relatively few parameters at a time. Early community experiments reportedly fell short; the speaker had heard of a potentially better release but had not tried it. Lian expects most users to choose between LoRA and QLoRA. He reports a community view that QLoRA can sacrifice final quality because quantization affects training updates, but offers no comparative measurements. It is a tradeoff to evaluate, rather than a quantified quality penalty.

Learning from examples and preferences

Supervised instruction or chat fine-tuning is framed here as teaching a model to express existing knowledge in a desired way or perform a task. Reinforcement learning with human feedback (RLHF) steers behavior using feedback on candidate answers. Responses to the same prompt can be ranked for helpfulness, accuracy or toxicity, then those preferences guide training. Starling is offered as an example of using multiple ranked responses.

The order of these stages remains an open question in the conversation. Supervised fine-tuning followed by preference-based training is described as customary, but the respondent does not know whether reversing the order works equally well. An unidentified Mistral-based model that apparently skipped the supervised stage is recalled as an interesting exception. The practical invitation is to experiment and report results, rather than treat the customary sequence as a requirement.

Configuration captures engineering experience

Axolotl builds on Hugging Face’s libraries: PEFT, for parameter-efficient fine-tuning, supplies methods that reduce the number of trained parameters; Transformers supplies modeling code and the trainer. Axolotl wraps these components and adds practical checks informed by experience using them.

Individually plausible settings can fail in combination, sometimes only during training or evaluation. Configuration validation catches incompatible or unintended choices earlier, while allowing some risky combinations with warnings. It turns knowledge otherwise passed through GitHub and Discord into rules the tool can apply for the next user.

Newcomers expose the remaining gaps. Most feedback reportedly arrives through Discord, including obscure errors and failures to start training. An Axolotl developer describes asking users about pain points because daily familiarity hides the learning curve. Early frustration can drive users away before they reach the capabilities that attracted them.

Pack examples without mixing their context

Sequence packing saves training time by placing several examples into the input space that one padded example would otherwise occupy. The discussion credits Hugging Face’s [StackLlama](https://huggingface.co/blog/stackllama) with demonstrating this approach. For illustration, if each example fills one-eighth of the available context, packing eight examples fills it. Training them separately would leave most of each input occupied by padding.

Simple concatenation creates a problem with attention, the mechanism through which later parts of the input can use earlier parts as context. A later example can now attend to unrelated examples placed before it. The developer recalls using this approach for Manticore, gaining training speed with a small accuracy tradeoff, though no measurements are supplied.

[Multipack](https://arxiv.org/abs/2107.02027) preserves the separation between examples while packing them together. An attention mask controls which parts of the packed input can see one another. After trying to construct that mask manually, the developer describes an implementation using Flash Attention that receives both concatenated examples and their boundaries. For equal-length examples, boundary offsets might be 0, 500, 1000, …, 4000. Those boundaries let the forward pass distinguish the component sequences: packing fills the input without turning separate examples into one continuous context.

The speaker estimates roughly a 2× improvement from Flash Attention and gains as high as 20× with Multipack in some cases, depending on the data. The accompanying Alpaca reproduction attempt used eight L40 GPUs with multiple optimizations enabled. It displayed an estimated completion time of about 30 minutes and a projected cost of $4–5, but was stopped before completion. The comparison was a recollection of eight A100s running for three hours at a reported $100 cost. Different hardware, projected completion and mixed time-and-cost comparisons mean 20× is not a controlled throughput result.

Axolotl’s goal is to expose improvements like these through configuration switches. An example YAML described as “Llama 2 FFT optimized” bundled several optimizations and reportedly ran successfully. Because optimizations do not always work together, reusable configurations and compatibility checks are part of making the performance work usable.

New architectures meet integration work

The final technical topic is the then-new Mamba architecture. The speaker tentatively describes an attentionless design whose memory and runtime grow closer to linearly with context length, making longer inputs especially interesting. The speed estimates and secondhand comparison with a small Pythia model remain preliminary; the conversation presents no completed evaluation establishing those gains.

The immediate obstacle was an interface mismatch. Mamba’s released modeling code did not yet fit Transformers cleanly, preventing it from dropping directly into Axolotl. Early integration attempts failed with training loss collapsing to zero. The developer believed the integration was finally working on the morning of the recording, but had no completed training evaluation to report.

A stable tool needs sustained contributions

Axolotl’s next priorities were to stabilize the feature set and write a roadmap. The intended path followed developers from learning model training through side projects to using the same tool at work. Organizational adoption would require reliable behavior and software companies could vet.

A proposed plugin system would expose hooks for company-specific workflows, including changing hyperparameters during training. Still a GitHub issue at the time, it aimed to let companies customize Axolotl without maintaining forks and repeatedly rebasing to receive upstream features.

The main request was for more core contributors who could own substantial work. Implementing Mamba between Uber rides and meetings illustrated the maintainer’s bandwidth constraint. A contributor referred to as Nano was credited with answering questions, managing the community and fixing issues while doing graduate research in Japan. The Open Access AI Collective, described as a nonprofit, also had a Discord community people could join.

Alex closes with a request to users, especially commercial users: contribute code, sponsor the work and make complaints constructive. Do as much supporting investigation as possible, and report where you get stuck instead of silently switching tools. That feedback can turn one person’s failed training attempt into an improvement for everyone who follows.