Wing Lian

Wing Lian is the maintainer of Axolotl, an open-source framework for fine-tuning large language models, and founder of the Open Access AI Collective. His contributions include reusable training configuration, dataset curation, and integration of research techniques into practical training workflows. Axolotl, with Lian named, received an a16z Open Source AI Grant in December 2023; the program provided grants rather than investments.[1][3][4]

In his December 2023 interview, Lian recalled learning language-model training with Alpaca-LoRA after a mid-March skiing injury restricted his usual activities. Two practical problems shaped Axolotl: datasets used incompatible prompt formats, and experiments required copying and editing long command-line hyperparameter lists. He chose YAML configuration because it was “more portable and reproducible.”[1]

Axolotl used Hugging Face Transformers and PEFT rather than replacing them. Lian added value around those libraries through configuration validation: combinations of settings that might otherwise fail during training or evaluation could be caught earlier, or flagged with a warning. He treated this accumulated knowledge as a way to reduce the frustration of experimentation.[1]

Lian described model releases, including Manticore, as development exercises that exposed missing capabilities and opportunities to make Axolotl faster. Debugging could then produce something useful. He also sought pain points through Discord and conversations with users, recognizing that his everyday familiarity with the tool concealed the learning difficulties newcomers faced.[1]

For SlimOrca, Lian helped revisit the correctness of GPT-4-generated OpenOrca answers. An OpenChat collaborator cross-referenced those responses with the original FLAN answers; Lian then submitted the paired answers to GPT-4 to judge whether the generated response was mostly correct. He recalled filtering the GPT-4-only portion from approximately 800,000 to 500,000 rows and retaining performance within about 0.1 percent of the original model, though he did not specify the evaluation metric.[1]

Lian warned that benchmark performance can obscure a model’s actual abilities. If test questions enter the training data, deliberately or accidentally, a model may learn the answers without generalizing the underlying skill. His concern was the distinction between answering familiar test questions and being able to perform the task.[1]

Lian reported that the team integrated QLoRA into Axolotl seven days after its announcement. By the December 2023 interview, he was also adapting Tri Dao and Albert Gu’s newly released Mamba modeling code to the Transformers/Axolotl workflow. The code did not fit cleanly at first, and training loss fell to zero; he said he believed the integration was finally working by the recording. His contribution was making the architecture usable through the training framework.[1]

Lian’s work on sequence packing addressed wasted computation when short training examples occupy only part of a model’s context window. He initially concatenated examples, following StackLlama, to replace padding with useful training text. But concatenation alone lets later examples attend to earlier, unrelated examples. After trying explicit attention masks, he drew on an OpenChat collaborator’s Multipack implementation using Tri Dao’s FlashAttention, which supplied sequence boundaries so packed examples could be processed separately.[1]

To illustrate the potential gains, Lian described starting an optimized Alpaca reproduction on eight L40 GPUs. He stopped it before completion, using an estimated completion time of about 30 minutes to estimate a total cost of $4–5. His discussion of gains up to 20× depended on the dataset and combined optimizations; the comparison also involved different hardware and pricing assumptions. His usability goal was straightforward: “I just want to flip the knob and move on with my life and not figure out how to implement it.”[1]

In December 2023, Lian proposed plugin hooks so companies could customize training workflows without maintaining forks that were difficult to update. He also sought more core contributors, naming Nanobit, Amin/TMM1, and Casper Hansen among the existing group. He particularly credited Nano with community management, technical support, and fixes, explaining that more contributors would ease the pressure of fitting integration work between travel and meetings.[1]

Lian experimented with extending language models’ context windows, with evaluation help from Gradient, according to co-founder Mark Huang’s May 2024 interview. Huang reported apparent deterioration on needle-in-a-haystack tests around 500,000 to one million tokens, while emphasizing that evaluation remained unsettled. He also associated Lian with a model-delta methodology that Gradient was experimenting with to transfer capabilities between models.[2]

In July 2024, Lian published implementation guidance for QLoRA fine-tuning Llama 3.1 405B with Axolotl. To reduce system-memory demands, he described loading quantized weights on rank zero while using the meta device on other ranks, crediting the approach to answer.ai. He also identified a missing bitsandbytes quantization state needed for FSDP and credited an anticipated fix to @matthewdouglas. He provided guidance for addressing these loading issues without reporting a completed training run or measured memory use.[5]

Before his language-model work, Lian built software across several industries. Looking back in December 2023, he recalled scraping apartment listings, building search-optimized pages, and earning Google AdSense income during college around 2004–2005. He wrote in Perl, then moved through technical support and professional software and web development, including applications for the blood industry.[1]

Lian later worked at Social Gaming Network as it moved from Facebook applications into mobile applications, followed by other startups and work in music at UnitedMasters and SoundCloud. In the December 2023 interview, he said he was no longer working at SoundCloud and was concentrating on Axolotl.[1]