Making Transformers Sing: Suno’s Bet on Audio Tokens and Musical Participation

What should a music model predict?

A speech model can produce intelligible words while losing the qualities that make a voice sound natural. That problem helps explain Suno’s approach to music generation: learn a representation broad enough to capture speech, singing and background music, then generate sound through successive predictions. A representation designed for a narrow task may help initially while limiting what the model can eventually express.

In the March 2024 Latent Space conversation, Mikey Shulman describes audio generation as unsettled—roughly one or two years behind images and text, in his estimation. Both diffusion and transformer approaches were in use. Suno preferred next-token prediction: represent audio as discrete units called tokens, predict a probability distribution over the next token, and repeat to generate successive pieces of sound. The language-model operation remains familiar; the tokens represent audio rather than text.

Shulman wants to impose as little explicit musical knowledge as possible, allowing models to learn structure themselves. He accepts worse early performance in exchange for the greater generality he expects later. This does not eliminate design choices: the biases and training data of the models that encode audio remain consequential.

Audio tokens and the speed of listening

Audio tokenization—encoding sound into discrete representations—is where Shulman locates much of Suno’s proprietary work. Separate models learn these representations, which must accommodate speech, background music, vocals and other sounds. Suno can then apply transformer sequence modeling to those tokens. The detailed tokenizer design and training-data mixture remain undisclosed.

The training data extends beyond music. Shulman compares this to code models learning useful patterns from natural-language text. Realistic vocals were a particular weakness, and he suggests that nonmusical human vocal audio could help, without claiming a measured improvement. The team’s understanding of audio scaling was also preliminary: whether higher-quality music makes disproportionately better training data remained an open question.

Model size meets a concrete constraint: generation must keep pace with listening. Shulman describes Suno’s models as relatively small, with larger models generally performing better. But streaming requires enough token throughput to sustain playback. He speculates that this might keep Suno from reaching 175 billion parameters; it is a forecast, not a demonstrated ceiling or a disclosure of actual model size.

Small models also forced useful research. Early time and compute constraints pushed the team to find improvements it might have missed with a much larger starting model. Shulman sees scaling as both a straightforward route to better performance and a potential crutch, so Suno preferred smaller models where possible.

From Kensho’s speech work to Bark’s unexpected music

Suno’s co-founders worked together at Kensho, primarily on text, before a financially focused speech-recognition project drew them into audio AI. Music experiments gradually became the work they could not put aside. Advisers encouraged a speech business, and the enthusiastic GitHub reception for Suno’s first public release, the open-source text-to-speech model Bark, reinforced that advice. The team nevertheless chose music for its emotional impact. The Parakeet work raised in the interview was described as an NVIDIA collaboration alongside that focus.

Bark used self-supervised training: learning from audio itself before adaptation to text-to-speech, rather than relying entirely on scarce, high-quality data dedicated to that task. It borrowed substantial code from Andrej Karpathy’s NanoGPT. The team describes its central contribution as turning audio into tokens so existing transformer machinery could model the sequences.

The team attributed Bark’s natural-sounding speech to training on natural speech, arguing that a narrow emphasis on intelligible words could sacrifice other qualities. That was a qualitative assessment. A more surprising product signal came from Discord users: a musical-note prompt could elicit early forms of music, and people kept trying to extract songs from a speech model. Suno took that persistence as evidence of demand for accessible music creation.

From a coffee-shop joke to a song worth refining

Suno’s two interaction modes served different creative impulses. A single description box let someone turn an event into a song quickly. Expert mode supported repeated changes to words, line breaks and ad-libs. The team reported that more than half of usage occurred in expert mode, though it did not specify a measurement window or define the usage metric.

A co-founder’s name miswritten on a Starbucks cup became a quickly made song and a shared joke that stayed in the team’s heads. The other use case was more deliberate: listen, tweak and regenerate until the result matched an imagined song. Both suggested intimate sharing—music made for yourself or a few friends, much as people text photographs. A personalized Valentine’s song for the interviewers illustrated that possibility.

The intended audience shaped the controls. Suno expected ordinary consumers to describe a song’s overall feel, rather than request precise changes to an isolated drum sound. Professionals were already reportedly using it for inspiration and samples, but models customized to their music or voices remained possible future directions. The broader consumer workflow was still being discovered.

The comparison to Midjourney captured the pleasure of creating and sharing, but the response emphasized music’s distinctive social property: listeners hear the same moment together over minutes. Collaborative creation could extend that synchronized experience. Suno’s opportunity was to bridge the gap between musical taste and musical ability while making participation social.

The demo separates fidelity, composition and control

The live demonstration uses a development server and a new model still receiving finishing touches. Its first request is a country song about a cloud provider running out of GPUs. The lyrics incorporate CUDA cores and falling back to a CPU. A participant finds generation strikingly fast, but no latency is measured. The demonstrator rejects the first candidate as unimpressive and plays another: topical responsiveness does not guarantee a satisfying song.

Next comes house music about podcasting and AI music generation. The demonstrator hears the drop arriving at the wrong time, introduces a beat-drop marker, strengthens the house-style instruction and regenerates. The marker’s exact syntax is not given. The team says users have taught it a great deal about prompting, tentatively describing their techniques as inventive ways to communicate with the model.

Keeping the lyrics while changing the music creates another constraint. An interviewer points out that a hip-hop rhyme scheme may depend closely on its beat. The demonstrator agrees, while suggesting that the same words could still work in country rock. Suno evaluates these outcomes along three axes: - Sound fidelity: Does the audio sound crisply recorded? - Song quality: Is the composition interesting and memorable? - Controllability: Does the result respond to the prompt? Generating the same lyrics in different styles is one stated test. It probes responsiveness, but the new setting must still suit the words.

The country-rock attempt retains the beat-drop marker. After playback, the demonstrator hears a possible electronic influence and wonders whether the marker caused it—explicitly a speculative connection. The last premise, a sad AI wearing an Apple Vision Pro, becomes a blues song. Subsequent prompts request melancholy, Chicago blues and slide guitar, followed by another house attempt. Repeating a genre word is described as a way to emphasize style, although excessive repetition can push the input out of domain.

During the blues example, an interviewer asks whether Suno uses RAG for music. The demonstrator says no and calls it problematic, without explaining the term or the concern. The exchange does not establish how the system handles new world knowledge.

Original songs, collaborative concerts and personal taste

Restyling lyrics leads to a question about existing songs. Suno’s stated policy at the time disallowed regeneration from someone else’s lyrics because the user lacked publishing rights. Authorized remakes were a possible future direction; the immediate focus was new, original music.

The argument against synthetic artist covers also concerned lasting appeal. A hypothetical Metallica rendition of the Beatles’ “Come Together” might spread quickly, but the response compares its novelty to repeatedly requesting Shakespearean sonnets from ChatGPT. There is a relational cost as well: an imitation weakens the connection to the original artist without creating an authentic connection to the imitated performer, who did not participate.

The model roadmap emphasized better fidelity, composition and control, including opportunities to borrow from open-source transformer work. The product ambition went beyond converting text into music: alternative inputs and collaborative creation could make listeners active participants. Against the backdrop of the UMG–TikTok dispute over existing music revenue, Suno framed its goal as expanding participation. Gaming supplied an analogy for the appeal of active experiences.

An interviewer proposed a continuous Twitch music stream shaped by chat. The response explored a collaborative concert, with or without a central artist, where generative tools help audience members communicate the sounds they imagine without needing musical training. These were exploratory ideas. Continuous DJ-style transitions received an optimistic prediction, without a release date.

The methods for steering models remained undisclosed. Asked whether DPO, mentioned in connection with V3, meant Direct Preference Optimization, the respondent described several experiments without confirming that method. The interviewer then raised a practical feedback problem: a metallic voice could be intentional, while an odd pause could be a defect. A thumbs-down cannot explain the distinction, and casual listeners may not want to annotate it.

Ratings, sharing and play telemetry were proposed as future learning signals. The ambition was a model users could teach through their preferences, like a producer directing a flexible artist. At the time, everyone used the same model. The song discussed as “Return To Monkey” illustrated the appeal of an audience of one: music can resonate intensely with its creator without affecting others the same way.

The team also reported positive reception among blind and vision-impaired users. It saw sonic creation as an accessible creative experience with some of the pleasure people find in image generation, while acknowledging that the analogy was imperfect.

Aesthetics are an evaluation requirement

The broader audio landscape divides into music, speech and sound effects, with overlap and a mixture of specialized and general models. Within music, the discussion identifies four markets: stock background tracks, AI covers, new songs and professional production tools. Suno chose new-song creation while recognizing room for tools such as production-software plug-ins. Generating unusual sounds was framed as an extension of instrument making, with the theremin as a historical analogy.

How, then, does a team know its model is improving? Small benchmark differences may have little consequence in use. Suno’s principle is aesthetics matter: if the goal is music that makes listeners feel something, evaluation requires listening. The team considers audio benchmarks even less adequate than text benchmarks. Better objective measures are desirable, but they cannot capture every quality that matters or become the sole optimization target.

That concern also shapes research judgment and hiring. Economists are praised for questioning incentives and measurements; recruiting at economics conferences reportedly produced some of Kensho’s best hires. The value extends beyond interpreting human feedback. Powerful models can overfit and respond to steering in poorly understood ways, making intuitive, first-principles reasoning useful alongside mathematical analysis.

The closing example is the Stanford Question Answering Dataset (SQuAD), a benchmark for answering questions about passages. In the account given, models reached human-level performance on the original task, but adding questions whose answers were absent from the passage reopened a substantial gap. The decisive move was to ask what the test left out. Music demands the same judgment: crisp audio, a memorable composition and faithful prompt response are distinct qualities, and a convenient score cannot establish that a song delivers the experience its listener wanted.