Teaching Robots More Cheaply—and Automating RL’s Hidden Labor — Part 6 of 6

Collect demonstrations, then act fast enough

The opening poster discussion in the source recording puts two robotics costs side by side. The presenter reports collecting 45 human demonstrations per real-world task in roughly one to two hours, then achieving performance similar to diffusion-based policies with inference taking about one-fifth as long. Reported control rates exceed 100 Hz on GPU and reach roughly 20 Hz on CPU. Hardware models, benchmark tasks and the performance metric are unspecified.

The proposed next step is to scale the architecture toward more generalizable agents, including robots that perform tasks from language instructions.

From task-specific policies to the data bottleneck

Chelsea Finn begins with early robot policies—models that choose motor actions from observations—with just 92,000 parameters. They mapped camera images directly to motor torques to insert a block into a shape-sorting cube or lift an object into a bowl with a spatula. Having to train from scratch for every task led her toward few-shot learning and meta-learning: using previous experience to learn new tasks faster. That work also found applications in education and drug discovery.

Those policies were also tied to particular tools and environments. Seeking broader generalization led Finn to study distribution shift, where conditions differ from those represented during training, and to help develop WILDS, a benchmark examining that problem across real applications.

She identifies three persistent obstacles: - Motor-control data is scarce. Robotics lacks an internet-scale equivalent of demonstrations for tying shoelaces or opening bottles. - Physical safety is difficult to establish. Learned policies are hard to interpret, and mistakes can directly harm people. - Formal guarantees are lacking. Compared with approaches such as optimal control, learned policies offer fewer assurances about their behavior.

Explicit physical modeling has its own difficulties. Tearing tape and applying it to a box requires accounting for adhesion to the dispenser and fingers, tearing against a metal edge, and coordination of 14 motors. ALOHA, a teleoperation interface developed by Finn’s student Tony, lets a human puppeteer the robot to collect demonstrations. An end-to-end transformer learns from those demonstrations to map camera images to control of 14 joints, with examples including applying tape and putting on a shoe.

Finn describes related advances across several robot bodies: - Mobile manipulation: a two-armed mobile robot learns from teleoperation to cook and serve shrimp and put a pot in a cabinet. - Surgical tools: learned policies on a da Vinci robot tie knots and transfer a needle between tools; the reported robustness is qualitative. - Humanoids: transformer policies coordinate arms and legs using demonstrations from a harder-to-build shadowing interface.

These examples underpin Finn’s view that supervised learning already produces useful robot behavior, supported by better architectures, optimizers and debugging practices. She sees data scarcity as a more direct explanation of robotics’ difficulties than the observation that intuitive human motor skills are hard for machines. Scarcity includes unfamiliar situations inside otherwise large datasets. Her response is to seek cheaper information through natural supervision, data outside robotics and experience gathered at test time.

Language corrections train the choice of skill

Once architecture and optimization are well tuned, improving a failing model often means collecting and labeling more examples of its failure scenarios. That cycle is expensive. Natural supervision expresses task knowledge directly: a person can explain that water should be hot before adding pasta, without specifying arm movements. Preference feedback also avoids exact target outputs, but still requires many labeled outcomes. Richer verbal guidance may convey more per example.

The robot example is a long task using both arms: putting objects into a bag. If the opening is too narrow, a person might suggest using a sponge to open it wider. The correction should rescue the current attempt and improve future autonomous behavior.

A hierarchical policy connects two responsibilities through language: - Choose a skill: the high-level policy maps observations to an instruction, such as picking up the sponge. - Execute the instruction: the low-level policy combines that instruction with observations to produce motor commands. The hierarchy is prior art. The training idea is to freeze the low-level policy and update only the high-level policy using language supervision. This can improve the whole system provided the low-level policy already follows a sufficiently broad range of instructions.

Yell At Your Robot, or YAY Robot, uses a correction loop resembling DAgger, the dataset aggregation algorithm: 1. A human instruction overrides the high-level policy’s choice. 2. The low-level policy receives the correction and acts on it immediately. 3. The correction enters a dataset used to train the high-level policy, while the low-level policy stays frozen. Because the high-level policy outputs language, the correction supplies a training target without requiring a new motor trajectory.

Before correction training, the robot places a Sharpie underneath the bag and struggles to recover. After a failed grasp, it keeps requesting the same skill rather than adjusting the gripper. After fine-tuning, the narrated demonstrations show repeated recovery attempts, lateral adjustments after missed grasps, and a change of strategy with the sponge: releasing it and poking it into the bag after pushing fails.

Corrections can prevent a failure, too. In a trail-mix task, a scoop sits behind the bag, apparently about to spill peanuts. A human redirects the left arm, raises the scoop and moves it inside before autonomous pouring resumes. After fine-tuning, a cranberry-scooping example shows the robot repositioning a scoop that had missed the opening before proceeding.

The reported average task-success improvement is 20%, closing much of the gap to live human corrections; the talk does not distinguish relative percent from percentage points. Low-level execution remains a ceiling: performance is still limited even when humans choose the instructions. Language feedback improves the high-level policy without new demonstrations, but the system still needs robot data and a capable instruction-following controller.

Describe a classifier’s failure, then rebalance its data

Natural supervision can also target image-classification errors. A bird classifier may rely on background trees; a shape classifier may rely on color. Users compare correct and incorrect predictions and describe the apparent pattern. Here, one concept can identify many errors, making feedback more economical than labeling each image separately.

The interface uses CLIP’s text–image similarity scores to check whether a description separates correct predictions from failures. Users can revise their wording until the system connects the concept to the relevant images. It then follows a rebalancing approach attributed to prior work called DFR: balance the identified groups and retrain or fine-tune. For tree-related failures, balancing images with and without trees reduces the incentive to depend on that background feature.

A study recruited 26 crowd-platform participants with minimal qualifications, including native English proficiency, to use the interface on Waterbirds and CelebA. Many identified the relevant concept; some descriptions matched or exceeded lead student researcher Yunho’s reference phrases, while others had less effective wording. Four cases involved difficulty identifying the correct failure. Feedback took an average of two to three minutes.

Rebalancing and retraining with those descriptions reportedly improved robustness over original-dataset training and zero-shot prompting. The reported gain over original-dataset training was 7–10%, with neither the exact metric nor the relative-versus-percentage-point distinction specified.

In a separate ImageNet extension, Yunho identified failures in 31 classes, including sliding-door images associated with cars. Reweighting improved performance on the minority split while preserving overall performance. This extension had no user study. The broader lesson is that observed failures make supervision easier to target, though the demonstrated correction method addresses one kind of failure; extending it to other types remains future work.

Make robot control resemble pretraining

Data outside robotics offers another route to better generalization. An ImageNet-pretrained encoder reportedly helps somewhat, especially in familiar scenarios, but leaves a substantial gap on unseen objects, backgrounds and environments. The proposed improvement is to make the downstream task resemble the pretraining task.

A model trained for visual question answering answers questions about images. Robot control can use the same format: pose a task question and output tokens representing actions, including gripper translation and rotation. Those tokens become executable commands. This vision-language-action model retains the pretrained architecture and is fine-tuned on robot tasks alone or a mixture of robot tasks and the internet question-answering data used in pretraining.

The reported recipe generalizes better than an ImageNet-pretrained encoder. Comparisons involving RT2X and OpenVLA, two vision-language-action models, also show better average performance across two robot platforms on generalization-focused evaluations than models using standard pretraining or none. The spoken comparison supplies no numerical scores or detailed task conditions, and does not isolate the contribution of task formulation from vision-language pretraining.

Use a failed attempt to choose the next action

No training set can anticipate every deployment scenario. A human door-opening example illustrates the alternative: after placing a key incorrectly, the person withdraws it and tries again successfully. Test-time adaptation uses experience from the current situation to change subsequent behavior.

The robot method passes recent attempts and the current observation to a vision-language model to guide skill selection. An unchanged image can warrant a different action once an attempt has failed. Crawling near an obstacle may initially look reasonable; after trying it, turning may make more sense. This use of attempt history as model context allows the reported system to change strategy where a system without history repeats its mistake.

In an outdoor example, the robot’s rear legs become stuck on an unstable step outside its current field of view. Without history, it keeps walking forward. With history, it backs up and tries climbing over the step. The reported improvements exceed 50% in success rate and time to complete a test scenario, but baseline values and the precise calculations are unspecified.

Connecting language to locomotion remains unresolved, including whether language is always the right abstraction for retries. Future directions include general learning regimes built around natural supervision, pretraining designed for easier transfer, and adaptation during interactions with people or web environments.

These approaches complement larger robot datasets. Finn mentions co-founding a company in March of the talk’s year to explore scaling data and models across robot platforms and real-world uses. Her closing observation concerns evaluation: benchmarks can exclude information available in the actual application. When natural supervision or test-time experience is unavailable under benchmark rules, the benchmark may sometimes be harder than the problem it represents.

Expose the manual engineering inside automatic control

Younghyo Park’s closing position talk proposes an automatic behavior generator: specify a robot, environment and task, then obtain a controller through time and compute alone. Practical reinforcement learning (RL) still requires substantial human work around training. When learning fails, practitioners often change the environment using domain knowledge and trial and error. Park calls this environment shaping; modifying rewards is only one part of it.

Using the robotics benchmark Isaac Gym Envs, Park illustrates two other forms of shaping: - Action-space shaping transforms outputs into motor commands. The example applies scaling, clamping, moving-average filters and a final motor controller, adding task-specific choices and hyperparameters. - Observation-space shaping engineers inputs. A door-opening environment adds handcrafted task features to simulator state and changes representations of quantities such as rotations to suit neural networks.

Park reports that the tested RL algorithm, PPO, failed to solve the tasks after action-space shaping was removed. Removing handcrafted observation terms could also break learning. The task list, training budgets and scores are unspecified, so the findings apply to those experiments. They illustrate how algorithm performance can depend on surrounding human optimization, which Park calls graduate student descent.

He proposes two complementary directions: - Automate shaping to reduce the human effort needed to construct task-specific heuristics. - Develop algorithms that need less shaping, evaluated on unshaped environments without those heuristics. The team created a collection of unshaped robotics environments, with APIs and tools to support both directions.

Search over code—and handle interactions between components

Shaping requires searching over diverse functions. Fixing their forms and tuning coefficients with conventional hyperparameter optimization limits that search. Representing shaping functions as Python code opens a more flexible code-optimization problem for large language models. Park cites Eureka as an example of using a language model as a sampling-based optimizer for shaping.

The team’s GPT-4 experiments expose a difficulty beyond generating individual functions: - Individual components: action and observation shaping reached performance similar to human shaping. - Joint optimization: asking the model to shape multiple components together caused performance to drop dramatically. - Sequential optimization: optimizing components one at a time often reached only a local optimum. Park provides no numerical evaluation details and presents full automation as unfinished work.

Park closes by considering foundation models trained on large, manually collected datasets as another route to robot controllers. He argues that RL remains valuable for generating robust, generalizable and especially superhuman behaviors that imitation learning may struggle to produce. RL-generated behavior could itself train foundation models. Making RL easier to use could therefore create a reinforcing cycle: more generated behavior supplies more data for better embodied models.