RWKV: Recurrent Language Models Meet a Website-Sized Problem

First generate the test. Then explain why it failed.

UIlicious had a concrete use for a language model: let users describe a browser test or a code change, then generate the code. Eugene Cheah describes building that system on Salesforce CodeGen. Fine-tuning—adapting an existing model with task-specific examples—changed the economics. He estimated that roughly 1,000–10,000 examples might yield a functional, if imperfect, system, rather than the millions of records he had previously expected to need.

Deployment mattered too. UIlicious packaged its product in containers for enterprise customers to run on premises. Cheah valued what he regarded as CodeGen’s clearer licensing position, particularly when European customers asked about those obligations.

Then customers asked for the next step: when a test failed, could the AI inspect the website, explain the failure, and recommend a change? The team got this working on toy websites before trying basic interactions on popular sites: visit a page, find an element, click, add something to a cart. Amazon was the first failure. An e-commerce page could contain megabytes of HTML even after stripping CSS and JavaScript. The practitioner estimated that such inputs could exceed a million tokens—the text units a model processes.

One response was to change the input. The team explored a custom tokenizer but considered converting HTML into a more token-efficient representation more practical: it would preserve compatibility with existing models. The other response was architectural. [RWKV—Receptance Weighted Key Value](https://huggingface.co/blog/rwkv) offered a recurrent approach, carrying information forward in an updated state as it processed each token. Its appeal was a processing cost that grew linearly with input length, though its memory still had limits.

The context search finds a multilingual community

In the August 2023 conversation, the practitioner places his initial search in the preceding year, when 4K tokens was the commonly understood context limit. Vanilla transformers’ quadratic scaling with context length looked prohibitive for the million- or ten-million-token workloads he imagined. The interviewer raises a separate concern: a larger context window does not guarantee that a model can retrieve and use everything in it. Cost and useful memory were already two different problems.

The guest tentatively recalls finding RWKV through GitHub, forums, Discords, and blogs, possibly through lists of models useful outside English. Multilingual support was a discovery route, rather than the main requirement of the website-testing application. For other community members, it was the point: they wanted something they could use and experiment with in Korean, Japanese, Thai, Arabic, and other languages.

Their needs exposed two obstacles. Inefficient tokenization makes a language expensive to represent; inadequate training coverage leaves a model poorly equipped to use it. The guest argues that predominantly English evaluations encourage model builders to prioritize English data. RWKV’s multilingual usefulness came from different data and tokenizer priorities, rather than an inherent advantage of recurrence for particular languages.

The strategy was to include languages in the foundation model, then let users fine-tune it for their applications. Adapting a model that already supported a language was expected to be easier than teaching it that language from scratch. Initial coverage included English, Chinese, Japanese, and Korean, reflecting community participation. But language coverage does not ensure balanced knowledge: the guest expected substantial US or English-language bias to remain and said it had not yet been fully evaluated.

An alternative to attention, available to run

RWKV’s lineage reaches back to Apple’s 2021 [Attention Free Transformers](https://arxiv.org/abs/2105.14103) work. Its creator, Bo Peng, known as Blink, applied that approach to language modeling and scaled it up. The proposition was a recurrent neural network, or RNN, that could train with transformer-like parallelism and compete at comparable model and dataset sizes.

RWKV replaces transformer multi-head attention with time mix, and feedforward networks with channel mix. Its weights and layers still select information that influences the output, but do so without conventional attention heads repeatedly scanning preceding tokens. The guest dislikes the community’s term light attention because it turns discussion toward whether the mechanism counts as attention, obscuring how its computation differs.

At the time, the main downloadable models had 7 billion and 14 billion parameters. Releases served several purposes: - Raven was instruction-tuned: trained to respond to instructions, using data described as typically generated by GPT-4 and scrubbed of model self-identification language. - World was still being trained, focusing on a new tokenizer and broader multilingual coverage. - Smaller 0.1B and 0.4B models primarily supported research, although users also tried practical applications.

The companion article describes RWKV as Apache 2.0 licensed while raising separate concerns about training-data provenance, including the Pile and Alpaca/ShareGPT material. In the interview, Open Assistant is suggested as another source of instruction data; its subsequent inclusion is recalled but not confirmed. A permissive project license and training-data provenance are separate considerations.

World: native-speaker feedback and a simpler tokenizer

World’s collection of roughly 100 languages was pragmatic. Wiki material went into the dataset when the download tool worked for a language; failed downloads were skipped. Translation pairs added coverage. The code data was already present—the deliberate change was to increase multilingual material.

When the interviewer describes this as intentionally accepting worse benchmark scores, the respondent pushes back: dataset additions answered user feedback, rather than targeting a benchmark loss. He also clarifies that he is a community member, not the person training these models. Intermediate checkpoints, released at stages such as 10%, 15%, and 20% of training, let Discord members report improvements and regressions. The informal rule was that native speakers should judge language quality.

Japanese- and Chinese-specific tunes were available, while users of smaller languages were asked to test the base World model. Testers joined through requests to the community, rather than appointments as representatives for each language.

The tokenizer—the component that splits text into model inputs—used a concrete rule: choose the longest matching string in its vocabulary, using a trie data structure to find it. It did not assume spaces as delimiters, which the guest says helped Japanese and Chinese text. The aim was to change the tokenization approach without substantially enlarging the vocabulary. He recalls roughly 50,000 entries in the previous tokenizer and believes the replacement was similar in size.

His intuition separates understanding text from representing it efficiently. He recalls a model understanding text laid out vertically, one character per line, despite the extra tokens. That anecdote illustrates why character-level understanding can coexist with a strong computational reason to group characters into larger tokens.

Recurrent inference, overlapping training

The architectural challenge was a training bottleneck: conventional RNNs needed one token’s state before they could compute the next. In the guest’s account, that sequential dependence made large-scale training unattractive. Bo Peng began working publicly in the Eleuther AI forum to make recurrent training scalable and parallelizable. Sponsored GPU compute helped move RWKV from small experiments to 14B models and the comparisons discussed in the [RWKV paper](https://arxiv.org/abs/2305.13048).

During inference—running the model to make predictions—the recurrent loop is straightforward: 1. Process a token using the current state. 2. Produce an updated state and a prediction. 3. Carry the state into the next token’s computation. RWKV’s time-mix and channel-mix operations sit inside layer blocks. Each block passes its state to its counterpart at the next token position, carrying information forward through the sequence.

The training explanation emphasizes overlapping work across layers and token positions. A lower layer can advance to the next token before all upper layers finish the previous one. Some operations work independently per token; others retain dependencies on earlier positions. This is an approximate account of execution, with a practical goal: expose enough concurrent work to keep the GPU busy, even though dependencies remain.

The guest explains the two mixing operations through their memory roles. Time mix combines input and state, using learned weights to retain or discard information; it is expected to preserve longer-term information. Channel mix is strongly biased toward the next token, moving short-term information upward through layers until it disappears. This division is intended to ease the burden of maintaining near-term and long-term information together.

The cost argument follows from the recurrent loop. With the model held fixed, each token requires a state update whose work is constant relative to sequence length: O(1) per token, or O(n) across n tokens. The transformer comparison involves an increasing amount of interaction with prior tokens—token 1,000 must account for 999 predecessors. These are qualitative scaling claims; the conversation supplies no hardware, caching configuration, or matched timings for a reproducible speed comparison.

Working memory and model weights are different costs. Token-by-token inference can retain the current and next states without storing a history of states, but a 14B model still needs storage for its parameters. Community members were porting RWKV to C++ and running it on Raspberry Pis. Large-model training nevertheless required substantial hardware; lower compute costs did not make a 20B training run a single-GPU proposition.

Architectural parity and the best available model

Claims of competitive reasoning performance have an essential condition: comparable parameter counts and training data. The guest says the benchmark comparison used the Pile, not RedPajama. A model such as Falcon, trained on much more data, could make more sense for an English-language application. The discussion provides no task-level scores supporting a universal claim of no trade-offs.

An architectural comparison asks whether recurrence can compete at matched scale. An application builder asks which available model works best on their computer. RWKV still needed larger training datasets to compete on that second question. Proposed 40B- or 60B-parameter models were ambitions requiring data work and GPU time, not completed runs.

That also complicates RWKV’s limited visibility. The guest points to its noncommercial open-source structure, weak marketing, and late paper publication; the interviewer challenges marketing as a complete explanation. Meanwhile, RWKV had attracted people underserved by English-focused models. The guest estimates that about half its Discord community had joined for languages other than English.

Generating indefinitely is not remembering indefinitely

An unbounded generation loop does not provide unlimited useful context. The guest describes models trained around 8K context, with some community work reaching 16K. They could continue generating beyond those lengths, but performance deteriorated outside the range they had been trained to handle. That weakness mattered directly for question answering over large documents and summarization.

When the interviewer suggests that longer context simply needs more funding, the contributor disagrees. The existing way of scaling memory was inefficient; reaching 100K would require changing it. Experiments targeted effective retention around 4K–8K, with a roadmap toward 16K, 32K, and 64K. The aspiration was to retain useful information at larger scales, with remnants surviving beyond what a sliding context window would discard.

This returns to the website problem: the goal was to process inputs such as 100,000 tokens or four megabytes of HTML while extending the portion retained without loss. Those remained targets. The companion article adds two weaknesses, citing the [RWKV documentation](https://wiki.rwkv.com/): sensitivity to prompt formatting and poor performance on lookback tasks, which require using earlier information. Cheap traversal and dependable recall remained separate engineering requirements.

Where software engineers can contribute

The closing advice distinguishes application building, model training and dataset curation, and deeper model research. Application engineers should experiment with available models, develop their prompting skills, and compose prompts into sequences or agents. Choose providers and open models according to usefulness. Understanding every layer’s mathematics is not a prerequisite for delivering value to users.

Going deeper means collecting data, training or fine-tuning, inspecting loss, and developing intuition about learning rates and training order. The guest reports roughly six months experimenting with CodeGen and another six with RWKV, with GPU costs a real constraint. Helping with music generation then overturned assumptions formed in language-model work. Domain experts could contribute by identifying foundational and advanced material when designing datasets and deciding training order.

First-principles study remains useful alongside experimentation. The guest recommends Zero to Hero for showing components built incrementally, even when practical work uses Torch. He describes discovering a fundamental misunderstanding despite already having working models with good loss. During RWKV V5 experiments, collaborators informally predicted which changes would produce better loss curves and were often surprised. Understanding the code did not make the outcome predictable.

The final invitation is to bring familiar engineering skills into model development. The guest points to Stable Diffusion community optimization and memory mapping in GGML and Llama-related inference code as examples of established systems techniques improving working implementations. He jokes about researchers repeatedly rediscovering multiples of 32 when tuning Nvidia GPU workloads—a workload-dependent anecdote, rather than a universal batching rule.

There is room to contribute through datasets, training practice, shader code, memory mapping, caching, documentation, and code organization. Getting a model to work leaves substantial engineering work in making it efficient and useful.