What Makes a Thousand-Layer RL Network Trainable?

Why was deep RL still using two layers?

Kevin’s starting puzzle was concrete: reinforcement learning (RL) algorithms he encountered in a Princeton research seminar used simple two-layer networks. Language and vision had benefited from much larger models, yet RL trained from scratch still relied on shallow ones. He wanted a recipe that would make additional capacity useful.

That project became 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities, which received a NeurIPS 2025 Best Paper award. Its authors are Kevin Wang, Ishaan Javali, Michał Bortkiewicz, Tomasz Trzcinski, and Benjamin Eysenbach. Kevin started and led the work in Ben’s independent research seminar, collaborating with Ishaan; Michał joined later after pursuing related questions.

Ben was skeptical: his earlier attempts at deeper RL networks had failed, as had previous work he knew. But infrastructure Michał had developed over the preceding year made another attempt relatively inexpensive. It was a research bet worth testing.

Depth needed a different training recipe

The approach was self-supervised RL: learning from relationships within the agent’s experience, without a human-crafted reward signal. It learns representations of states, actions, and future states. Representations associated with the same trajectory—a sequence of experience through the environment—are pulled together; those from different trajectories are pushed apart. This contrastive objective supplies a signal for learning to reach goals.

Even with that objective, adding layers initially degraded performance. Kevin then combined depth with residual connections and other architectural components, including layer normalization. In one environment, one doubling of depth did little, but another produced a sharp improvement. The result suggested a critical depth, though the conversation gives no numerical threshold for that environment.

The interaction mattered. Ben explains that depth alone hurt, while residual connections alone did not help. Tuning one variable at a time could therefore discard the ingredients before testing their successful combination. The team borrowed residual-network techniques to address vanishing gradients and tested what happened when those connections were removed.

Ben’s central qualification is that architecture and objective must work together. The result does not justify inserting a huge network into an arbitrary existing RL algorithm. The successful objective itself does not use rewards.

Learn future-state relationships through classification

The classification task is straightforward: given a current state and action, decide whether a candidate future state belongs to the same trajectory or a different one. The network learns representations that support this distinction. A participant argues that shifting the learning burden from noisy, biased value-function regression to classification helps explain why the method scales.

This places the method between familiar categories. One participant questions the RL label because the code does not directly maximize rewards. Another describes it as a goal-conditioned actor-critic algorithm: it learns to act toward a specified goal, but uses an objective resembling self-supervised methods in language and vision. The architecture adapts earlier work; the useful contribution is its combination with the contrastive objective.

Many evaluated tasks involve robotics. That motivates a possible alternative to imitation learning, where agents learn from human demonstrations: improve goal-reaching capabilities without increasing manual supervision alongside them. Extending this approach to broader robotics applications remains a research ambition.

Depth saves parameters, but costs time

The depth-versus-width experiments start from four hidden layers with 256 neurons each. At fixed width, adding layers grows parameter count roughly linearly. Widening successive layers increases both their input and output dimensions, so parameter count grows approximately quadratically. At approximately matched parameter counts, depth produced faster performance gains in the team’s experiments. They also report better sample efficiency—learning more effectively from collected experience.

The practical rule is to favor depth when parameter count is the constraint. Inference speed is a separate tradeoff: at some depths, doubling the layers can roughly double the time needed for a forward pass through the network.

A thousand layers is not necessary for every task. A team member says most environments reached nearly perfect performance much earlier, offering 64 layers as an example of sufficient depth. At that scale, collecting experience can be a larger bottleneck than inference.

The experiments use a JAX-based, GPU-accelerated environment described as Jax GCRL, which collects thousands of trajectories in parallel. This supplies enough experience to make larger networks useful, with GPU simulation enabling large-scale collection within hours.

Substantial experience was necessary before the gains appeared. In the recording, a speaker places the large performance increase after roughly 50 million environment transitions. The companion article instead says deeper networks became particularly beneficial beyond 15 million. These differing accounts should not be treated as a universal training threshold.

An implicit world model

The language-model analogy concerns the learning objective: next-token classification in language, trajectory-membership classification here. The intended transfer is to bring lessons from language-model scaling into RL. Changing language-model pretraining in the opposite direction comes up as a research suggestion, not an established method.

A participant interprets the state, action, and goal representations as an implicit world model: learned information about the environment without explicitly predicting a high-dimensional next frame. Another extends the idea with a poker analogy: consider possible hands and narrow the range as an opponent’s actions reveal information. Classifying candidate possibilities offers a way to think about world modeling, though this analogy is not a reported experiment.

The inference cost leads to a proposed deployment strategy: deep teacher, shallow student. Train a deep model to advance capabilities, then distill those capabilities into a shallower model for cheaper inference. Distillation is already among the team’s future directions; retaining the deeper model’s performance at standard-network efficiency remains the goal.

Scale capacity and batches together

The future-work discussion begins with trajectory stitching: learning shorter sub-behaviors that can be combined at test time to generalize beyond those individual behaviors. It then returns to network capacity. Width also improved performance, and increasing depth made larger batch sizes useful.

The proposed explanation is that small networks cannot fully exploit the signal in larger batches. The team says its experiments support this capacity-bottleneck hypothesis: successfully training deeper networks unlocked benefits from larger batches. Whether this explains weak batch-size scaling throughout traditional value-based RL remains open.

A next experiment would scale depth, width, and batch size together with a larger compute budget. All the team’s experiments, including the thousand-layer networks, are said to be runnable on a single H100 GPU with 80 GB of memory. That specifies hardware feasibility, not runtime or total experimental cost. Distributed training would let researchers explore the combined scaling dimensions further.

From representations to actions

The closing technical discussion turns to vision-language-action models and robotics world models. A participant is exploring how representation learning could apply to them. Another argues that industry attention has favored text generation over action outputs, describing tool calling as structured text. An impressive model preview prompts discussion of screen-based applications, while transfer to embodied uses remains uncertain.

Two architectural directions make the possibilities concrete: - Action experts: keep a pretrained vision-language model frozen and train additional experts on top to produce actions. - Hierarchical planning: use a larger, slower network to emit higher-level plans or chunks less frequently, while a second system operates faster. Hierarchical planning offers another way to accommodate expensive inference: different parts of an acting system can run at different frequencies.

Asked about the poster session, a team member recalls visitors finding the work eye-opening: a simple objective challenged expectations about RL scalability, with continued improvement reported at a thousand layers. The closing question is how far goal-reaching agents can advance when this approach scales across several dimensions together.