From Reusable Vision Features to Actions Learned from Video — Part 5 of 6

DeCAF: reusable features need usable software

A visual representation may need to locate an object’s parts as well as recognize the object. The DeCAF retrospective opening this part of the ICML 2024 recording reports that DeCAF improved part localization out of the box, with further gains from straightforward localization techniques. Its scene representations also grouped into indoor/outdoor and man-made/natural categories when visualized, despite not having been trained on those labels.

The presenter credits open-source dissemination as DeCAF’s main impact. Other contemporary papers demonstrated transfer, but the DeCAF and Caffe releases helped the community adopt it quickly. DeCAF made inference with pretrained features accessible; Caffe and GPU acceleration subsequently made training models of that size more accessible. Together, they challenged the perception that useful deep learning required infrastructure on Google’s scale.

DeCAF reused frozen or relatively frozen AlexNet features—representations whose parameters stayed largely unchanged during adaptation. Retrospectively, that resembles a foundation-model approach: pretrain a reusable representation, then adapt it. The presenter offers a deliberately simplified historical arc: end-to-end task training became fashionable around 2015–2020, enabled partly by Caffe, before pretraining regained dominance in the early 2020s, with BERT and CLIP as examples.

Prompting need not involve text. The presenter distinguishes fine-tuning a pretrained model from directing it through prompts, citing the group’s NeurIPS 2022 visual-prompting work and subsequent large-vision-model work. Vision-and-action models, with or without language, are presented as a more open research opportunity than competing with companies on the next vision-language model. Future architectures might also lower the compute barrier, though that remains speculation.

One robotics project adds action pretraining so prompts can specify a control scheme, task, and desired trajectory. Humanoid Control as Next Token Prediction uses a transformer without an underlying text language model, pretrained on human and humanoid action data from the wild and simulation. The presenter reports locomotion in new environments, including San Francisco, without quantitative evaluation details.

A closing question compares prompting with fitting a linear prediction layer over frozen features and asks whether language changes the picture fundamentally. The answer separates two issues: - Fine-tuning versus prompting: their formal connection remains unsettled. Research into how models learn from examples in their context may help explain it. - Text versus other token sequences: the presenter doubts that text is uniquely special. Tokenizing a modality and predicting its tokens could apply broadly, including to vision and motor control.

Language replaces fixed class IDs

Lucas Beyer describes language as the API for vision: the input/output interface through which users communicate with a model. Traditional workflows pretrained on fixed class IDs, then fine-tuned on a smaller labeled dataset for a task such as flower classification. CLIP and ALIGN changed both supervision and use: train on image–text pairs, then classify by scoring free-form candidate descriptions against an image, without task-specific fine-tuning.

That changes what the data can teach. A hand-selected class vocabulary encodes its creator’s perspective and limits the concepts explicitly supervised. Web image–text pairs offer more noise and more descriptive detail. Alt text and page titles may contain useless thumbnail metadata, but they may also describe a Frankfurt Airport skyline in 2017 or a London barge race—specificity unlikely to emerge from a short class list.

Beyer introduces SigLIP, his group’s downloadable CLIP variant, through matching examples. A photo of two people wearing coffee-themed shirts scores 100% against a description about needing caffeine and 4% against one about needing water. A cow-on-a-beach example scores 99% for that context, 36% for a cow, and 1% for a cow in a prairie. These are individual matching scores, not accuracy measurements. Beyer uses them to challenge the claim that an unusual background necessarily defeats recognition.

Benchmark incentives narrow the data again

A separate multilingual SigLIP checkpoint learned from multilingual web pairs without special translation procedures. Examples include French and German descriptions of a cow on a beach. A more revealing case concerns the dish called ants climbing a tree: the English phrase matched literal ants, while the Chinese dish name matched the food. Language carries culturally specific associations, beyond literal translation. Beyer notes that the paper did not thoroughly evaluate multilingual performance; he also mentions a larger multilingual release available by the time of the talk.

The reason for separate English and multilingual models is benchmark incentives, Beyer argues. English-only training improves ImageNet zero-shot scores—classification without task-specific fine-tuning—and other English benchmark results. He describes LAION-5B as roughly two billion English pairs, two billion non-English pairs, and one billion with unknown language, while CLIP-style papers commonly use only the English subset.

Several filters can reinforce that narrowing: - English-caption filtering removes other languages at the outset. - CLIP-score filtering favors pairs an existing English-trained model already matches well. - ImageNet-oriented filtering retains text containing ImageNet class names or images resembling ImageNet examples. Beyer argues that ImageNet-like evaluations reward these choices even as they restrict the data’s breadth. He highlights InternVL for explicitly combining English LAION, multilingual LAION, and Chinese data.

Matching a caption can leave its relationships unlearned

Better captions alone do not settle the learning problem. Beyer’s central example is an image captioned as a cat sitting left of a dog. In contrastive matching, the model learns to associate an image with its caption against other examples in a training minibatch. What it must understand depends on those alternatives.

The caption permits different shortcuts: - No other cat or dog in the batch: recognizing either animal may identify the correct pair. - Another cat, but no other cat-and-dog pair: recognizing both animals may suffice. - The same animals in reversed positions: sufficiently precise examples and captions, without another distinguishing shortcut, can force attention to the spatial relation. Thus, the objective can reward less understanding than the caption contains. In Beyer’s account, SigLIP shares this limitation.

Captioning directly supervises the relational word. An image encoder feeds a text decoder trained to predict the caption. At the position requiring left, predicting right, above, or below incurs a loss. Beyer advocates this objective and disputes the characterization of captioning as much less efficient to train, although he gives no compute comparison in the talk.

Evaluation needs the same scrutiny. ARO—attributes, relations, and order—tests competing captions that differ along one of those dimensions. Beyer reports substantially stronger captioning results when both approaches were extensively optimized on the same data in otherwise matching setups. But choosing between a horse eating grass and grass eating a horse can exploit language alone. His group therefore trained a blind decoder, a captioner that never sees the image, to probe that shortcut.

A subsequent benchmark uses plausible alternatives: a yellow racket with a blue ball versus a blue racket with a yellow ball. Beyer reports that its authors’ blind baseline achieved random accuracy, while captioning models outperformed equivalent CLIP models—and even the strongest CLIP models—on almost all tested categories. These compositional comparisons underpin his recommendation to move beyond CLIP.

One output stream for questions, boxes, and masks

The Pali model family takes an image and task text as input and produces text. A prompt can request a Romanian caption or ask a pointed visual question. Beyer illustrates the distinction with a coin image: asking for all coins produces twelve; asking specifically for one-dollar coins produces two.

The same interface accommodates structured vision tasks: - Bounding boxes become coordinates. Encode two corners, normalize coordinates by image dimensions, and multiply by 1,000 to obtain integers independent of pixel resolution. - Segmentation masks become discrete codes. A mask encoder compresses a mask into a short sequence from a small vocabulary. Append that vocabulary to the language vocabulary, predict the codes, then decode them into a mask. - Task text specifies the target. A request can distinguish a left hand from a right hand, with the interpretation of left and right inherited from the training data.

PaliGemma pairs a SigLIP image encoder with the decoder-only Gemma two-billion-parameter language model. It is downloadable, subject to license restrictions. Its training separates acquiring broad capabilities from adapting those capabilities to a particular use:

1. Reuse independently pretrained models. Start with image and language checkpoints, avoiding their initial pretraining cost within this pipeline. 2. Train them together on multimodal tasks. Supply image and text inputs with text outputs. 3. Increase resolution. Begin around 224×224 for efficiency, then use a shorter, more expensive stage at a higher resolution such as 448×448. Fine detail, especially document text, can require it. 4. Transfer to the intended use. Apply supervised fine-tuning or instruction tuning to focus the pretrained capabilities and adapt the interface.

Task prefixes make different pretraining skills explicit. The supervision comes from several sources: - Multilingual captions: language detection supplies a label for requesting a caption in the appropriate language. - Reading and answering: existing text-recognition systems provide text-reading targets, while classifiers support synthetic questions about image contents. - Question generation: producing a question for a supplied answer exercises a different skill from answering it. - Detection and segmentation: existing models supply pseudo-labels—training targets generated by those models.

Later fine-tuning removes awkward user-facing syntax such as answer EN. Beyer argues that this adaptation needs relatively little data because it largely changes how existing capabilities are requested. Pretraining aims to impart broad skills and knowledge; transfer focuses them and makes the interface suitable for use.

Reward tuning aligns sampling with task quality

A unified token interface introduces another mismatch: token errors do not measure detection errors well. Predicting coordinate 299 instead of 298 is wrong under exact next-token supervision, even when the box displacement is acceptable. An entirely spurious box may incur a comparable number of token mistakes to several small coordinate errors, despite being much worse for detection.

Beyer starts with supervised training to obtain a capable model, then samples alternative predictions and scores them with a task reward. Reinforcement learning (RL) tuning increases the probability of higher-scoring outputs. The reward need only return a number: it can be nondifferentiable, computed by a complicated metric, or supplied by a human. Generative vision systems such as Pali and Unified-IO provide the multiple plausible samples this procedure needs, unlike the conventional detection and segmentation systems he contrasts them with.

For object detection, Beyer reports COCO detection scores rising from 39 to 54, measured with the mAP metric used as the tuning reward. He does not specify the model, evaluation split, or tuning budget. He also describes segmentation experiments and a playful colorization example: define an image-flashiness score, tune against it, and obtain flashier outputs.

His closing interpretation is that good outputs become more frequent. In the sampled reward distributions, the untuned model already produces similarly strong outputs, but rarely. Tuning reduces low-reward outputs and favors high-reward ones; Beyer interprets this as redistributing sampling probability rather than teaching new capabilities.

Simply sampling more and choosing the highest-likelihood candidate need not improve task reward. Before tuning, likelihood may be poorly aligned with the reward. The key distinction in Beyer’s closing comparison is the reward of the highest-likelihood sample, rather than the highest reward available among all candidates.

The recording then closes its vision discussion with a pointer to the oral presentation for MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision Language Benchmark in the show notes. Its planned discussion was cut for time.

From broken robot hands to actions inferred from video

Ashley Edwards turns to learning actions, policies, rewards, and environments from video. Her starting frustration was having to design a new reward function for every environment. In summer 2016 at Waseda University, she planned to teach a robot sign-language gestures from videos. Its hands did not work, so she switched to facial-expression imitation.

Human and robot faces looked very different, and she had only a few examples and one GPU. Motion templates aggregate frames to represent where and when movement occurs. Comparing movement structure across different appearances let the robot learn approximate smiles and surprised expressions. The approach still required a hand-specified reward comparing features from the human and robot templates, and it remained confined to a narrow task.

In 2017, Edwards moved to publicly available game playthroughs, mostly speedruns, to learn behaviors across environments. The project generated motion templates for unseen scenes but did not reach its intended agent-control stage. It also produced an unexpected qualitative result: a model trained only on games could separate animals from unfamiliar scenes by predicting motion.

Predicting multiple futures exposed recurring action patterns. Candidate futures could represent movement left, right, up, or down; training minimized the loss of the candidate closest to the observed outcome. Edwards interpreted those recurring patterns as shared action representations. This led to ILPO, which learns actions and policies from video without recorded action labels.

ILPO separates two learned objects: - A latent action represents an inferred cause of a transition, such as moving right or jumping. - A latent policy gives the probability that the expert takes each latent action in a given state.

A generative model predicts possible next states. A minimum-loss objective selects the latent action whose prediction most closely matches the observed transition, grouping transitions by their outcomes. The policy then weights those possible futures so their expectation matches the expert’s expected future. For example, if an expert moves right half the time and jumps half the time, the policy should place its probability on those futures rather than on staying still.

Learning from video precedes mapping actions to controls. Environment interactions connect the learned latent actions to executable actions. Edwards reports rapid adaptation after 200 interaction steps in a demonstration, without specifying its environment, success metric, or baseline. Interpreting generated next frames as available actions also has an explicit condition: the dynamics are assumed deterministic.

Choose desirable next states without action labels

The next project asks what can be learned from poor demonstrations. With a supplied reward function, Edwards replaces a value function over state–action pairs with one over current-state–next-state pairs: it scores how valuable a transition is. A policy selects the next state to maximize that value. During environment interaction, an inverse dynamics model supplies an action for making the transition.

Edwards reports learning optimal trajectories from suboptimal videos, including random rollouts that sometimes reach the goal, without a quantitative optimality evaluation in the talk. The generated next frames describe a plan for reaching valuable states, connecting policy learning to video generation.

Without a supplied reward function, another approach uses expert-video ordering as a progress signal. Assign reward one to the final frame, then discount backward through the sequence to create targets for a value function. This assumes that the expert trajectory’s temporal ordering represents useful progress.

The learned video value function supplies targets for supervised policy training, replacing an RL bootstrapping step. In a pouring example, predicted values increase over time. Edwards also reports using the learned function to train an agent, without giving performance metrics.

Genie learns an interactive environment

Genie extends video learning from behavior to environments intended to be playable by humans and AI agents. Its three components connect visual representation to control: - A video tokenizer represents video through discrete codes. - A latent action model infers changes from frame sequences and supplies the representation used for control. - A dynamics model predicts future frames conditioned on those latent actions.

After training on a large platformer-game dataset, Genie could turn text-generated images into interactive environments, Edwards reports. Training agents inside those generated environments remains a motivation and future direction in this account.

The audience presses on a consequential ambiguity: visible change does not identify what an agent controls. Edwards acknowledges that separating actions from other environmental dynamics without supervision is difficult—the model might potentially let a user control clouds. Reward information, policy learning, text, and scale are proposed aids, not demonstrated solutions.

Edwards says Genie’s architecture has no explicit restriction to 2D platformers and mentions experiments on robotics data. She treats broader real-world video learning as a direction for more data, larger models, and updated architectural components, rather than an established result at that scale.

VQ-BET turns continuous behavior into tokens

The final poster interview introduces Seung Jae Lee’s VQ-BET, which adapts token prediction to behavior generation. The project began with difficulty training strong behavior-cloning policies—policies that imitate demonstrated actions—on large action datasets using traditional methods.

The bridge is quantizing continuous actions into discrete tokens. A vector quantizer converts action data into discrete representations. An architecture resembling a language model then predicts those tokens to generate behavior conditioned on the current observation.

Lee describes three sources of training data: - Downloadable datasets from mostly open-source environments. - Human demonstrations collected in some cases with virtual-reality equipment. - The team’s own real-world recordings, gathered with small manipulation equipment and an iPhone.

The team validated the framework in simulation before moving to real-world experiments. Lee calls the model lightweight, but Part 5 ends before he completes the resource explanation; no model size, compute requirement, or quantitative evaluation is given.