Dolly: cheap instruction tuning, harder questions

Start with a model people can get

The Databricks team had requested LLaMA weights and had not heard back. That obstacle helped determine Dolly’s starting point: [GPT-J](https://huggingface.co/docs/transformers/model_doc/gptj), a model Mike Conover had used before and considered a reasonable source of grammatical text. Accessibility was also a goal. He wanted other people to be able to build the experiment themselves.

Dolly adapts a pretrained language model to follow instructions. A base model may continue a prompt as though it were an arbitrary internet page; instruction tuning trains it on examples of requests and desired responses. The striking result was how much this additional training could change its behavior.

[Dolly 1.0](https://www.databricks.com/blog/2023/03/24/hello-dolly-democratizing-magic-chatgpt-open-models.html), released on March 24, used six-billion-parameter GPT-J and the Alpaca training dataset. Its rapid development drew on months of preparation: around ChatGPT’s arrival, Conover’s team built a hack-day integration for developers, received encouragement from the CEO, and continued alongside its infrastructure forecasting and optimization work. Dolly reused that code, and its release included some core components of the internal stack.

Nineteen days later, [Dolly 2.0](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm) moved to a 12-billion-parameter model from EleutherAI’s Pythia family. It also replaced Alpaca with instruction examples written by Databricks employees. Conover credits thousands of contributors, leadership support, and a core team, particularly Matt Hayes.

The campfire answer teaches more than campfires

The collection interface was Google Forms, organized around InstructGPT-style task families: content generation, brainstorming, question answering, and paraphrasing. Twice-daily email appeals and a leaderboard encouraged participation, alongside employees’ interest in making the dataset useful.

One answer explained how to build a campfire in four paragraphs. It demonstrated the kind of substantial response an assistant should give. Familiarity with ChatGPT may have helped contributors: they already had expectations about useful answers and wrote examples in that style.

Answer length is part of the training signal. The team observed that short instruction-tuning completions led to shorter model responses. The proposed mechanism involves the end-of-sequence token, which marks where a response stops: short demonstrations teach the model to stop earlier. Longer, thoughtful answers need corresponding demonstrations. No numerical comparison or experimental setup is given here.

That makes answer style distinct from task diversity. FLAN is described as a large instruction dataset spanning roughly a thousand or more tasks, often with brief answers such as a positive or negative sentiment label. Diverse tasks may act as regularization—discouraging the model from fitting itself too closely to one behavior—but do not necessarily demonstrate extended explanations.

Summarization was particularly hard to collect. Extracting people and places from a Wikipedia passage is comparatively mechanical; a good summary requires reading, comprehension, and synthesis. The idea that these examples provide richer training signal and take longer to overfit remained a research hypothesis.

The same difficulty motivates a zoom lens on language: show a business document’s high-level structure, then reveal details on demand. Image thumbnails can convey a collection’s character at a glance; language has no obvious equivalent. One speculative answer is personalized compression, adapting a long document’s presentation to the individual reader.

Discovering behavior inside a pretrained model

Conover compares instruction tuning to a phase transition: after pretraining on hundreds of billions of tokens, a small amount of new data can produce markedly different behavior. The analogy captures his surprise, rather than describing a measured physical transition.

Engineers choose the intended behavior and training examples, but optimization determines the learned parameters. Instruction tuning is therefore partly capability discovery. GPT-J was described as already two years old when this instruction-following potential was explored, raising the question of what other abilities might remain latent in existing models.

Pythia offers a way to investigate through a matrix of model sizes and checkpoints—saved states from different points in training. Comparing both dimensions could reveal when behaviors emerge and how they depend on scale. The practical goal is to estimate how much data and how many parameters a task requires. At the time, that process remained substantially trial and error; Chinchilla’s influence on training allocations was another example of teams revisiting their assumptions.

Human preferences provide a different training signal. In Richard Dawkins’s Biomorphs, users select appealing insect-like drawings, and variations of their rendering parameters produce new candidates. Aesthetic preference supplies the selection pressure. The analogy connects to reinforcement learning from human feedback (RLHF), where human preferences help shape behavior, rather than training solely to predict the next token.

Train and measure the behavior the business needs

A love letter to Edgar Allan Poe makes an engaging demonstration, but a support team needs useful ticket summaries and replies. Training data and token budgets should reflect those needs. One open question was how many summarization demonstrations a model of a given size would need to become useful for grounded question answering—answering from supplied material.

Choosing that data requires an evaluation that detects the resulting behavior. The team reported almost identical benchmark scores for GPT-J and the first fine-tuned Dolly, despite a striking qualitative difference in their responses. The comparison came without numerical scores or an exact benchmark configuration, but it exposed a practical problem: application behavior can change without the chosen metrics noticing. Assessing summary quality without a person remained unresolved.

One proposed solution uses active learning, selecting examples for human review according to what a model finds uncertain. A reward model would learn desired output qualities from employee or crowd-worker judgments. It would then send humans cases close to its decision boundary, where it struggles to choose a judgment, to obtain more informative labels. This was a proposed evaluation loop.

Classical machine learning also has a role in allocating work. A classifier can identify high-priority support tickets; a ranking model can decide which of tens of thousands of documents deserves a limited daily summarization budget. Generative models can then produce the summaries.

Prompting complicates the economics. Changing an instruction can turn the same support ticket into either a summary or a drafted reply, with further fine-tuning proposed to match the support team’s voice. The respondent estimates that comparable projects previously took a capable machine-learning team six to eight weeks. Making them accessible through prompting could save enough development time to outweigh substantial inference costs.

What the $30 run bought

The roughly $30 training cost covered instruction tuning of the original Dolly. It excluded the base model’s pretraining. The reported run used A100 hardware on a single Databricks machine, trained on roughly 50,000 Alpaca records, and took less than an hour. The estimate used standard cloud-provider pricing and may have been slightly below $30.

The open-source Dolly training notebook contains the distributed-training configuration. DeepSpeed provides options for distributing training across graphics processing units (GPUs): - Replicate the model across devices, process local batches, and aggregate their gradients to obtain a larger effective batch. - Shard model components, splitting them across devices when they cannot fit on one. Dolly reportedly needed sharding. The exact device count is unclear, but a single machine still used multiple GPUs.

Full pretraining over trillions of tokens can require more deliberate network-interconnect and data-center placement choices. Dolly’s surprise was the accessibility of adaptation: an afternoon and modest compute expenditure could reveal qualitatively different behavior. Microsoft and Hugging Face’s open tooling helped lower both the technical and financial barriers.

Evaluation needs a workspace

The team shared generated samples in Slack and reused roughly 50–100 reference prompts, including the Poe letter and backcountry-trip advice. Holding prompts fixed across models and checkpoints made differences easier to spot: shorter answers, stranger completions, or a stronger tendency to produce lists.

The desired workspace combined fast parallel generation and side-by-side inspection with three complementary methods: - Quantitative benchmarks for numerical comparisons. - Qualitative inspection of actual responses. - Human feedback at scale to collect judgments across examples. Sending a checkpoint to annotators every 1,000 training steps and collecting 100 judgments illustrated how these might fit together. The amount of feedback needed for statistical significance remained open.

A spreadsheet function might cover much of this workflow. The objections were about long-text editing and managing the evaluations: persistent records, locking completed annotations, collecting multiple judgments per item, and computing summary statistics. Scattered spreadsheet files make those responsibilities awkward.

Both sides nevertheless favored spreadsheet-style interaction: select a column or range, apply a prompt, and inspect the results. A dedicated interface should preserve that freedom to experiment. PromptLayer and HumanLoop came up as possible tools; Quadratic was suggested as a spreadsheet alternative by a participant who disclosed an investment. These were conversational recommendations, not comparative test results.

Spreadsheets also supplied an optimistic analogy for workplace change: automation can remove tedious work people once treated as their core contribution. Writing has its own boilerplate. One participant had built an email drafter using GPT-4 and sent many responses as drafted. The imagined next step was agents coordinating schedules or contract details and returning proposed actions. That requires faithfully representing the user’s interests; reliable handling of hallucinations remained unresolved.

Open data makes capabilities reusable

OpenAssistant points toward a possible division of labor: a few shared foundation models, specialized through fine-tuning. In that view, datasets become reusable investments in capabilities such as conversation or answering questions over retrieved passages. Writing suitable examples and producing them at scale becomes central engineering work.

One proposed augmentation keeps each answer fixed while paraphrasing its prompt, giving the model several formulations of the same request. Back-translation—English to Spanish and then back to English—was suggested as a way to produce variants. The expected benefit was analogous to image augmentation, but neither suggestion came with an experiment demonstrating improvement.

Spanish and Japanese translations of the Dolly dataset reportedly appeared within 24 hours, though their quality was not assessed. A larger proposed bootstrapping loop would generate prompts from examples, obtain large-model completions, and use human judgments or reward models to assess them. Corpora containing millions of examples were a prediction from this approach.

[RedPajama](https://www.together.xyz/blog/redpajama) addressed the earlier pretraining stage. It was introduced as a reproduction of LLaMA’s training dataset containing 1.2 trillion tokens. Producing that corpus required substantial work, including processing Common Crawl, removing near-duplicate material, and filtering for quality.

The Common Crawl filter was described as predicting whether a page would be referenced by Wikipedia. Its threshold creates a precision–recall trade-off: stricter filtering can exclude unwanted material while also discarding useful pages. Such choices can change the trained model’s quality and character. Reproducing the dataset therefore requires both methodological decisions and distributed-systems engineering.

By reducing that preparatory work, RedPajama motivated an April 2023 prediction: an open reproduction matching LLaMA’s quality could arrive within five months. This was an expectation about what a team might build, not an announcement of a completed model.

Commercial permission and independent scrutiny

Availability and commercial usability are separate requirements. Some businesses reportedly could not use Dolly 1.0 because of its training-data provenance. Clear commercial-use permission for both Dolly 2’s model and dataset was presented as a major reason for its positive reception, without interpreting specific legal obligations.

The appropriate model depends on the task and operating constraints. Public forum content might be processed by a specialized fine-tuned model or a general instruction-following model such as GPT-4. Other workloads may need to stay in-house because of intellectual property, restrictions on third-party subprocessors, or procurement friction.

Openness also enables independent scrutiny. Public research artifacts let specialists examine construction and behavior for bias, ethics, safety, and security problems. The argument is that broad inspection gives the field a better chance of finding and addressing problems than relying exclusively on a provider’s internal assurances.

A model migration is a behavior change

Moving between models can require different prompts, even between GPT-3.5 and GPT-4. A proposed migration tool would combine prompt regression tests—checks that expected behavior survives a change—with a model that iteratively rewrites instructions to preserve the previous system’s outputs.

An interactive storytelling book makes the problem concrete. A participant and a friend named Ben built a system where ChatGPT generates stories and branching choices, illustrated by an OpenAI image-generation model. The application needs exactly three choices labeled A, B, and C. It usually gets them, but occasionally the format changes despite emphatic instructions. That was an anecdotal failure pattern, not a measured reliability rate.

Regression testing therefore needs to cover format and output character across repeated generations. One successful response does not establish consistency. Evaluator models were proposed to help inspect outputs and manage migrations; a hypothetical hundred-run test illustrated the burden of checking everything manually.

Backcountry planning suggested another kind of evaluation: give a simulated character a generated packing list and observe the outcome. A simpler experiment would ask a model to critique a trip plan and inventory. The proposed explanation for self-critique is that the model can assess a complete passage, whereas generation selects each next token from the preceding text. This could help it recognize an error, but does not establish reliable self-review.

Two possibilities remain: self-reflection might exhaust knowledge already in the model and need further human feedback, or repeated interaction might produce continuing improvement. AlphaGo-style self-play supplies an analogy for the second possibility. Which trajectory language models would follow remained speculative.

Better tools, convincing avatars, and days outdoors

The closing product discussion returns to maps. Conover’s opening account of off-trail navigation involved relating a two-dimensional topographic map to nearby ridges, basins, and creeks. Google Maps later supplied a product example: it adjusts information density with zoom level and user intent, such as emphasizing parking while driving. The principle is adaptation subtle enough to go unnoticed. Hugging Face received similar praise for making complex infrastructure feel straightforward.

The predictions then turn toward representations of people. SkipFlag, described as a knowledge graph built from workplace communication, motivates a possibility: recorded statements could support avatars representing someone’s beliefs and intentions. Such systems might become useful and convincing before society has settled how to respond to them.

The Ghostwriter AI music example exposes the distinction between immediate plausibility and sustained coherence. One reaction was that it sounded radio-ready; another found it less coherent after repeated listening. Participants hoped human creativity would move toward unfamiliar ideas while generation served formulaic needs such as background study music. Whether markets would reward that distinction remained open.

The immediate request for builders was evaluation tooling that combines sample generation, quantitative tests, personal inspection, and human assessment. A second proposal returned outdoors: represent map tiles through geographic context and descriptions of places, then match a person’s desired day to suitable locations. The proposed representations were analogous to word2vec embeddings, encoding contextual relationships. Better discovery could spread visitors beyond the familiar trailheads where shared recommendations concentrate demand.

The final extension was fresher information about those places: coordinate sensor and camera inputs from multiple agents, perhaps low-power gliders surveying trails, to report current backcountry conditions. Both outdoor proposals had uncertain business potential. Their motivation was concrete—help people discover public lands and spend richer days outside.