Axolotl

Axolotl develops a free, open-source framework for fine-tuning and post-training large language models. Associated with maintainer Wing Lian since 2023, it builds on Hugging Face’s model-training libraries and combines training methods, dataset handling and performance optimizations behind reusable YAML configurations. The Apache 2.0-licensed framework now supports full fine-tuning, LoRA and QLoRA, preference tuning, reinforcement learning, reward modeling and multimodal training.[1][5][4]

Axolotl’s configuration-centered design predates those broader capabilities: its early development addressed the practical problems of mixing training datasets and reproducing fine-tuning runs.[1]

In his December 2023 interview, Lian recalled learning language-model training after a skiing injury in mid-March left him unable to pursue his usual sports. He began with Alpaca-LoRA, then encountered datasets with incompatible prompt formats and training scripts that required long lists of command-line hyperparameters. He wanted configuration in a YAML file because it was “more portable and reproducible.” Lian recalled integrating QLoRA into Axolotl within seven days of its announcement.[1]

By that interview, Lian had left SoundCloud and was concentrating on Axolotl after earlier software work at Social Gaming Network and UnitedMasters. He named Nanobit, Amin/TMM1 and Casper Hansen among the framework’s core contributors. Lian also described releasing models such as Manticore as a way to exercise the tool, uncover missing features and improve training speed while producing something useful.[1]

By December 2023, Axolotl was used to train Teknium’s OpenHermes, OpenOrca’s Mistral-OpenOrca and Mistral-SlimOrca, and Nous Research’s Puffin, Capybara and NousHermes, alongside models from Pygmalion, Eric Hartford and DiscoResearch. Its prompt interfaces and formatters accommodated datasets in formats including Alpaca and ShareGPT. That adoption was model-specific: Lian explained that Nous Research used its own trainer for its YaRN models.[1]

Axolotl used Hugging Face Transformers and PEFT rather than replacing their modeling and parameter-efficient training implementations. It added configuration validation to catch incompatible choices before they caused failures during training or evaluation. In December 2023, Lian proposed training hooks and a plugin system so companies could customize workflows without maintaining difficult forks; this was a roadmap proposal at the time.[1]

On December 13, 2023, a16z announced Axolotl, identifying Wing Lian, as an Open Source AI Grant recipient. The program explicitly provided grant funding rather than investment. The later Axolotl company profile, observed September 18, 2026, lists a 2024 founding year and privately held status; the framework was already publicly in use in 2023.[2][3][1]

One early efficiency improvement was sample packing: filling a model’s training context with several short examples instead of spending computation on padding. Lian first implemented concatenation inspired by Hugging Face’s StackLlama, but noted that ordinary attention could let one example attend to another. He then incorporated a Multipack implementation from an OpenChat contributor that used Tri Dao’s FlashAttention and sequence-boundary information to keep packed examples separate.[1]

Lian’s OpenOrca work also tested improvements to training data. For SlimOrca, an OpenChat collaborator cross-referenced generated answers with the original Flan answers; Lian then used GPT-4 to judge correctness. He reported reducing the GPT-4-answer portion from roughly 800,000 to 500,000 rows while retaining performance within about 0.1% of the original model. This was his account of the experiment in December 2023.[1]

Axolotl’s FSDP + QLoRA integration combines distributed sharding with quantized adapter training and credits Answer.AI’s enabling work. The documentation gives a configured example of training a 70-billion-parameter model on two 24 GB GPUs and identifies Axolotl integration PR #1378. Its instructions specify supported model types—Llama, Mistral or Mixtral—rather than promising that any 70B model or training setup will fit.[6]

Axolotl expanded beyond conventional instruction fine-tuning during 2025. It added reward and process reward modeling in January, Group Relative Policy Optimization (GRPO) in February, beta multimodal fine-tuning in March and quantization-aware training in May. In July, it added composable context parallelism, tensor parallelism and Fully Sharded Data Parallelism within and across nodes.[5]

On April 3, 2025, Salman Mohammadi and Wing Lian introduced local training with Python interpreter feedback. Their GRPO tool executes generated code in a WebAssembly sandbox using VMware Labs’ precompiled Python runtime, with computational “fuel” and timeout limits. It separates rewards for error-free execution from rewards for passing test assertions: the latter uses 2 × accuracy³, giving increasingly correct solutions disproportionately larger rewards. A formatting reward helps extract executable code from model responses.[7]

The authors reported training Qwen2.5-3B-Instruct on AceCode-87K for approximately sixteen hours using four A100 GPUs, with two serving generations through vLLM and two training. They reported improvements on MBPP and MBPP+ over both the starting model and Qwen2.5-Coder-3B-Instruct, and on HumanEval and HumanEval+ over the starting model. Their optional reusable multiprocessing pool produced almost tenfold faster code execution in their benchmark; that figure concerned interpreter execution, not total training time.[7]

Dan Saunders and Wing Lian’s April 4 sequence-parallelism article addressed a different constraint: fitting long sequences into GPU memory. Axolotl splits each input sequence across GPUs through the ring-flash-attn library and its LLaMA-3 variable-length attention implementation. The configuration requires FlashAttention, a micro-batch size of one, and a parallelism degree that divides the GPU count, sequence length and attention-head count.[8]

Their LLaMA 3.1 8B benchmarks show why context capacity and throughput are different outcomes. With artificial data, Liger kernels and AdamW 8bit, without gradient checkpointing, full fine-tuning increased maximum context from 9,216 tokens on one H100 to 46,080 on eight, while throughput rose 2.19-fold. In the RTX 4090 QLoRA test, context increased from 4,096 to 32,768 tokens across one to eight GPUs, but throughput fell to 0.88 times the single-GPU rate. Distributing a sequence enabled longer training contexts while introducing communication overhead.[8]

On January 28, 2026, Axolotl announced its partnership with Arcee AI for the Trinity-Large launch and plans for a coding-focused tutorial on distilling Trinity-Large into Trinity-Mini. Axolotl was a launch partner; Arcee’s large-scale pretraining run was the work behind Arcee’s model.[3]

On April 30, the Axolotl team and FalconLLM’s Younes Belkada presented their Falcon BitNet collaboration. They integrated training of TII’s Falcon BitNet models using TII’s onebitllms tooling and released experimental supervised fine-tuned and DPO variants. The work builds on Microsoft’s BitNet research, which trains models to tolerate weights restricted to −1, 0 and 1.[9]

The collaboration distinguishes low-bit inference weights from training precision: ternary quantization is simulated during bfloat16 training, and activations and gradients remain bfloat16. The article describes practical two-bit weight packing, distinct from the theoretical 1.58-bit representation. Only full fine-tuning was supported at publication; BitNet LoRA and on-policy reinforcement learning remained research directions. The experimental 10B checkpoint approximated prequantized weights by injecting scales into ternary weights and saving them in bfloat16 because the original prequantized checkpoint had not been released.[9]

Axolotl also extended adapter training for mixture-of-experts models. Its April 2026 updates added SonicMoE fused LoRA support, and July updates added NVFP4 four-bit MoE LoRA through ScatterMoE and SonicMoE, including merging trained adapters into a plain NVFP4 checkpoint.[5]

Axolotl runs on local hardware or users’ own cloud infrastructure, with Docker images and Python packages, and accepts local, Hugging Face and cloud-hosted datasets. Its homepage, observed September 18, 2026, reported more than 170 contributors and 500-plus active Discord members, and named Teknium/Nous Research, Modal, Replicate and OpenPipe as users. These are company-reported community and adoption figures. Its LinkedIn profile at that date listed a company size band of 2–10 employees.[5][4][3]