NeurIPS 2023 Recap, Part 2 of 4: Better Data, Visual Instructions, and Search

QLoRA: Preserve quality with four-bit fine-tuning

Can four-bit fine-tuning match a 16-bit baseline? Part 2 of the NeurIPS 2023 recap opens with QLoRA’s closing results. Across several Llama model sizes, the presenter compares accuracy on the MMLU evaluation after instruction fine-tuning. Ordinary four-bit float falls short of the bfloat16 baseline; four-bit normal float matches it in the reported comparison.

Cheaper fine-tuning also permits more experimentation. After more than a thousand experiments, the team reports selecting a dataset and building Guanaco, a chatbot fine-tuned for 24 hours on a single consumer GPU. The model size behind that hardware claim is unspecified.

Guanaco’s evaluation uses 80 Vicuna prompts in a tournament: two randomly selected systems answer the same prompt, and humans or GPT-4 choose the better response. Across repeated pairings, human judges place Guanaco slightly above ChatGPT; GPT-4 judges them approximately equal. Comparable quality on these prompts does not establish general parity with ChatGPT, as the presenter explicitly cautions.

The presentation closes with a claim of 18-times-cheaper fine-tuning, without specifying the cost accounting behind that multiplier. At the time, QLoRA was available through Bits and Bytes and integrated into Hugging Face Transformers.

DataComp: Fix training, vary the dataset

Image–text datasets enabled open-vocabulary classification: models could classify images using new textual class descriptions. Yet, as Samir explains, the experimental workflow largely stayed the same—fix a dataset and iterate on models. DataComp benchmarks dataset development by holding CLIP training fixed while participants change which data it receives.

CLIP aligns image and text representations by training an encoder for each with a contrastive objective. For classification, it compares an image’s features with encoded descriptions such as a photo of a plane or a car, then selects a label. DataComp evaluates these models zero-shot: without fine-tuning them separately for each downstream task.

The workflow follows five steps: 1. Choose a scale to set the architecture and training exposure. 2. Select data for the run. 3. Train with the fixed recipe, including its prescribed hyperparameters. 4. Evaluate zero-shot on the downstream suite. 5. Submit results to the leaderboard.

The crucial control is samples seen, rather than unique dataset size. At the small scale, selecting 6.4 million examples means seeing each twice to meet the 12.8-million-sample training budget. The four configurations are: - Small: ViT-B/32, 12.8 million samples seen. - Medium: ViT-B/32, 128 million samples seen. - Large: ViT-B/16, 1.28 billion samples seen. - Extra large: ViT-L/14, 12.8 billion samples seen.

Two tracks separate filtering from broader dataset construction: - Filtering: select from Commonpool, a Common Crawl-derived pool whose size matches the chosen scale’s exposure budget. Preprocessing includes near-duplicate checks against evaluation data and filtering of content deemed not safe for work. Metadata includes dimensions, captions, checksums, CLIP features and scores, and face bounding boxes for automatic blurring. - Bring your own data: add other sources, or edit and generate Commonpool images and captions. A separate track preserves comparisons among methods restricted to filtering the supplied pool.

Evaluation covers 38 downstream tasks, including ImageNet and variants, distribution shifts, fairness, and retrieval. Alex reports that better datasets remain largely better across variations in training recipes, suggesting that dataset rankings can carry beyond one exact recipe.

Filtering: Which examples survive matters

Two complementary baselines select data using a pretrained CLIP model: - CLIP-score filtering: compute cosine similarity between each image and its caption, then retain high-scoring pairs—for example, the top 30%. Similarity serves as a proxy for pair quality. - Image-based filtering: cluster image features and retain clusters assigned at least one ImageNet image. This filter uses image features without captions or class labels.

The best baseline keeps examples that pass both filters. Applied to the larger pool, it produces DataComp 1B, containing 1.4 billion samples. Across more than 300 pretraining experiments, the team reports higher accuracy with this dataset than with alternatives including OpenAI’s WIT and LAION 2B, under compute-matched model training.

With CLIP-score selection, performance peaks around retaining 30% of the original pool. Random subsampling does not produce the same benefit. Under the fixed exposure budget, carefully selected examples outperform a larger, noisier selection; reducing dataset size alone does not explain the improvement.

Filtering rankings are relatively stable between small and medium scales, though imperfectly correlated. The team uses that relationship to screen methods cheaply, then scale promising ones. By the presentation, community methods had surpassed the original baseline. Data Filtering Networks (DFN) investigate what makes an effective filtering model; in that context, the presenter reports 84.4% zero-shot ImageNet accuracy using a ViT-H14.

Beyond one dataset: Tools for continuing experiments

At the poster, participants emphasize that DataComp 1B is an artifact of the benchmark. Constructing a 12.8-billion-pair pool and a broad evaluation suite required substantial collaboration. The reusable contribution is tooling for cleaning, filtering, and benchmarking, allowing other builders to run their own dataset experiments. The project’s open resources and smaller scales are intended to support participation with only a couple of GPUs.

A participant reports strong correlation between zero-shot ImageNet accuracy and performance across the wider suite, including retrieval, while acknowledging variation by benchmark. That makes ImageNet a useful signal within these experiments, but leaves a research question: what distinguishes a good filtering model from a good CLIP model? At billions of images, curation means building models that select examples.

Proposed extensions bring different constraints: - Image generation: evaluate data selection through the quality of a trained Stable Diffusion model. Repeated diffusion training is the compute obstacle; this is a proposed experiment. - Other modalities: extend the benchmark idea to language, audio, video, or medical images. Video poses particular difficulties because of its data volume and the many dimensions a benchmark could vary.

One participant traces DataComp to discussions within the Institute for the Foundations of Machine Learning about improving LAION. The ambition grew from producing a better dataset to opening tools so others could do the same. The host closes this discussion by arguing that dataset and model research both matter, then introduces LLaVA as a possible companion to DataComp in future multimodal systems.

LLaVA: Create visual instructions with a text-only teacher

Recognizing objects does not by itself answer what is unusual about a scene. Haotian illustrates the gap with a GPT-4 Vision demonstration: a man is ironing clothes while standing on the back of a taxi. He motivates LLaVA, an open visual instruction-following model, by arguing that earlier systems such as BLIP2 and Flamingo struggle with complex instructions and long conversations. An open model would make visual reasoning easier to study and improve.

Instruction tuning means fine-tuning on instruction–answer pairs. Haotian describes Self-Instruct’s approach of expanding a small seed set using a text teacher and examples supplied in its prompt, with Alpaca as an example of this broader approach. For visual instructions, the team lacks a powerful multimodal teacher. Its workaround is to give a text-only teacher a textual representation of the image.

Captions from the COCO dataset describe the overall scene; object categories and bounding boxes add regional context. Manually written instruction–answer examples demonstrate the desired output. Given another image’s annotations, the teacher generates new instructions and answers. These are paired with the original image to form image–instruction–answer triplets.

LLaVA-Instruct 158K trains three complementary behaviors: - Conversation: sustain multi-turn engagement about an image. - Detailed description: attend to visual details. - Complex reasoning: infer implications beyond the visible objects. For the last behavior, an example asks what challenges people face in a scene with luggage and an SUV. The intended answer infers that the luggage may not all fit in the vehicle.

The architecture connects a CLIP vision encoder to a Vicuna language decoder through a linear projection layer. This connector maps image features into a representation the decoder can use alongside the user’s instruction to generate text. The presenter attributes its effectiveness to the semantic information already present in CLIP’s features.

Training separates two tasks: 1. Align the features: train only the projector, keeping the vision encoder and language model frozen. 2. Learn visual instruction following: train the projector and language model on the generated triplets. For limited compute, the presenter suggests LoRA, QLoRA, or projector-only training, reporting useful visual chat without measurements for these alternatives.

LLaVA evaluation: Separate demonstrated behavior from supplied context

Demonstrations show capabilities beyond the explicit visual instruction data: - Reading text in images: LLaVA recognizes “NeurIPS 2023” on a slide and relates it to artificial intelligence and machine learning, despite having no explicit optical character recognition (OCR) data in its visual instructions. - Multilingual responses: it recognizes the French Quarter and describes it in Chinese, despite English-only visual instruction data.

The evaluation pipeline also uses a text-only GPT model. It receives detailed textual image annotations, a user instruction, and two model responses, then returns scores out of ten with explanations. LLaVA-Bench in the Wild uses detailed annotations to evaluate accuracy and hallucination on questions requiring subtle visual details, multilingual understanding, and knowledge beyond the training data.

The presenter reports that LLaVA 1.5 trains within a day on one node and performs well across 12 benchmarks while using less than one percent of the data used by other approaches. The comparison approaches, scores, and node hardware are unspecified. Its simple design is presented as compatible with most language-model training and deployment optimizations; the demo, code, data, and model are described as open source.

A live demonstration presented as running on a MacBook Air interprets standing attendees in a conference photo as evidence that a presentation is popular. A follow-up supplies the event name and the user’s positive experience, then requests a tweet. The interaction combines visual inference with user-provided context; the event name and personal experience come from the follow-up instruction.

Adapting LLaVA: Match the change to the capability gap

The poster Q&A clarifies two roles in the pipeline. GPT-4 helps generate data; Vicuna generates LLaVA’s responses. Bounding boxes likewise belong to the data-generation context: LLaVA learns from images, instructions, and answers, so its training task does not require consuming those boxes.

Haotian describes developing the data engine through manual inspection and qualitative iteration: changing the number of in-context examples, revising reference instructions and answers, and clarifying the teacher’s task. Text-only GPT-4 produces plausible spatial layouts and left–right relationships, but that ability was not quantitatively evaluated. The supplied COCO context depends on the accuracy of its human annotations; it does not guarantee correct generated answers.

For adaptation, the suggested starting point depends on the images: - Similar domain: use the instruction-tuned model to retain its acquired visual knowledge. - Substantially different domain: consider new stage-one alignment or a domain-specific encoder, such as biomedical CLIP. Starting over may sacrifice useful visual reasoning and object localization. The speaker recommends testing that tradeoff experimentally.

Training-data needs depend on what the model must learn. In one reported experiment, 100 examples suffice to adapt LLaVA to generate Stable Diffusion prompts in a desired style. The model already understands the visual content; training mostly reorganizes how it expresses that knowledge. The result is qualitative. Unfamiliar reasoning tasks may need more examples, and the discussion’s suggestions of 10,000 or 50,000 are informal estimates, not universal targets.

Replacing the vision encoder with bounding-box inputs would shift several responsibilities elsewhere: - Locate objects: another system must obtain the boxes. - Read image text: an OCR system must supply it. - Describe attributes: additional visual information must reach the language model. The speaker favors a model trained directly on vision that can also coordinate auxiliary detectors, taggers, and OCR systems.

Unfreezing the vision encoder helps when low-level detail matters. The team reports little difference for ordinary object and attribute questions, but benefits for tasks such as judging whether a line is straight. VIP-LLaVA explores visual prompts: users circle an object with a scribble instead of identifying it precisely in words. Recognizing those small marks can require unfreezing the encoder or using earlier layers that retain the detail.

The discussion closes by returning to the project’s motivation: Vicuna’s performance at its size, together with its open availability, inspired an open visual reasoning model that a community could improve. The training-time exchange reiterates roughly one day on a single node for LLaVA 1.5, without specifying hardware.

Tree of Thoughts: Search beyond the first reasoning path

Shunyu introduces Tree of Thoughts through the Game of 24: combine four numbers with addition, subtraction, multiplication, and division to obtain 24. For 2, 9, 10, 12, (12 × 2) × (10 − 9) works. But a GPT-3.5 demonstration on 4, 5, 6, 10 reaches 48 and declares it to be 24. GPT-4 achieves only 4% task success in the reported evaluation using a five-example chain-of-thought prompt, which asks it to reason through successive steps.

The diagnosis is that local token choices can commit a model to a dead end. Explaining successive calculations does not supply a mechanism for exploring alternative initial moves. Drawing on the analogy of fast versus deliberate cognition, Tree of Thoughts adds a search controller. The language model proposes and evaluates reasoning moves; the controller determines which paths to explore.

A thought is a coherent piece of text representing one reasoning move. Choosing its size balances generation against evaluation: - Individual tokens: easy to generate, difficult to judge for future usefulness. - Complete solutions: easier to evaluate, difficult to generate correctly in one attempt. - Intermediate equations: a workable middle ground for the Game of 24, making each move relatively easy to generate and assess. Other tasks may use words, sentences, or paragraphs as thoughts.

The framework separates four responsibilities: - Define the thought space: choose what counts as a reasoning move. - Generate candidates: prompt the language model to propose next moves. - Evaluate candidates: estimate how promising each resulting state is. - Control the search: maintain candidates and decide which to explore. For the arithmetic game, a generation prompt proposes equations. A separate evaluation prompt considers the remaining numbers, briefly simulates routes to 24, and assigns high, medium, or low promise.

Evaluation supplies a heuristic, not a guarantee of solvability. The model rates 5, 5, 9 as plausible even though Shunyu says it cannot reach 24. Its role is to steer exploration toward promising branches; search can benefit without every local judgment being correct.

The Game of 24 search has three levels: four numbers become three, then two, then one. At each level, the system generates and evaluates moves, keeping only the highest-valued candidates. This is a thought-level beam search, with breadth specifying how many candidates survive. Reported success is 45% at breadth one and 74% at breadth five, compared with the 4% five-shot chain-of-thought baseline. Evaluation-set size and inference costs are not supplied.

Backtracking, planning, and agents that interact

Crosswords call for a different search policy. With five horizontal and five vertical clues, the system pursues promising moves until the board appears unsolvable, discards that branch, and returns to earlier choices to try alternatives. Shunyu identifies this as depth-first search. The reported result rises from 1% for chain of thought to 20% for Tree of Thoughts, then falls to 5% when pruning and backtracking are removed. The exact scoring metric and evaluation-set size are unspecified, but this comparison supports the importance of revisiting earlier choices.

For creative writing, the candidates are plot plans. The model generates diverse plans, evaluates them, and writes from a selected plan. Shunyu reports that human readers found the results more creative than chain-of-thought writing, without providing numerical results or an evaluation protocol. These applications adapt the units of generation, evaluation, and search without retraining the underlying model.

The presentation closes with search and language models as complementary tools: search supplies deliberate control, while language models supply flexible proposals and heuristics for open-ended problems. Follow-up work connects this approach to cognitive architectures for agents that reason internally, interact with the external world, and learn through those interactions.

The closing commentary values Tree of Thoughts as a general framework that could be specialized to recover approaches such as a backspace-token model, Skeleton of Thought, and Chain of Thought—an interpretation rather than a demonstrated relationship here. Part 2 ends as the poster discussion begins, returning to how language models and search can compensate for each other’s limitations.