Mark Huang

Mark Huang co-founded Gradient with Chris Chang and Forrest Moret and contributed to its 2024 work extending Llama 3’s context window beyond one million tokens. The 1048k model’s citation credits him alongside Leonid Pekelis, Michael Feil, Moret and Tiffany Peng. In his May 2024 public discussion of that work, Huang emphasized evaluating what longer context lets users accomplish, rather than treating window size alone as the achievement.[3][2][1]

Huang said the team chose Llama 3 partly because its original 8,000-token context seemed too short. One million tokens was a target, not an assured outcome: the team evaluated intermediate extensions and continued as the results held. Gradient’s model card reports extending the 8B model from 8k to more than 1040K tokens through progressive training on increasing context lengths, an initial RoPE-theta schedule and empirical optimization. Huang credited Crusoe for providing the substantial compute the project required.[1][2]

The positional-encoding decision was to adjust RoPE theta, which governs the frequency of the rotational encoding used to represent token positions. Increasing theta as the context grew helped the model accommodate longer sequences. Huang described the choice as easy and well understood by the team, while stressing that its success was empirical: “it's not like a mathematical tautology or proof”.[1]

Training those sequences also required efficient use of GPU resources. The team combined Flash Attention and Ring Attention, then adapted EasyContext to its cluster’s network topology. Huang credited Zhang Peiyuan’s open-source contribution and said EasyContext was the first PyTorch implementation they tried that worked well for them. The model card documents training on Crusoe Energy’s L40S cluster, with additional parallelism and a custom topology to address bottlenecks in moving attention’s key-value blocks between devices.[1][2]

The team separated continual pre-training from chat fine-tuning. Huang described filtering and concatenating SlimPajama texts to reach the target lengths, then filtering and reformatting an UltraChat-derived dataset for chat. GPT-4 rephrased and reformatted parts of that chat data. His central data-design warning was concrete: if the last 200 tokens always contain the answer, a long training sequence will not teach the model to attend to earlier information. For the 8B 1048k variant, Gradient reported about 830 million training tokens in the final stage and 1.4 billion across all stages.[1][2]

Huang treated needle-in-a-haystack retrieval as a preliminary check, rather than proof of comprehensive long-context understanding. The team also used RULER, whose more demanding tasks included retrieving multiple values, tracking variables through chains of assignments and counting common words across a context. These tests mattered to him because useful behavior can depend on connecting information scattered throughout the input.[1]

In the May 30, 2024 interview, Huang called four-million-token context “the next incremental checkpoint”. He wanted evaluations that connected larger windows to practical capabilities, especially tracking changing state through a long coding or investment-management session. He distinguished these evolving conversations from document synthesis, where retrieval-augmented generation or hierarchical recursive summarization could sometimes work around the need for a larger window.[1]

His view of agents likewise centered on outcomes: adding nodes to a workflow should improve its probability of success, rather than merely adding language-model calls. He identified evaluations, post-training techniques and synthetic-data construction as useful research priorities for Gradient, and invited collaboration on long-context evaluation and paired multimodal datasets. His product principle was to “build technology that people care about and will actually 10x their value rather than build technology that's just 10x harder.”[1]

Before this long-context work, Huang had worked in quantitative finance and enterprise machine learning. Those earlier roles supplied the experience with production systems and operational friction that he drew on when explaining Gradient’s founding.[1][4]

Huang recalled leaving quantitative finance partly to find more collaboration and learn how big data and machine learning scaled outside that environment. At Box, he worked across product analytics and go-to-market; afterward, at Splunk, he focused on streaming analytics, search and deep learning. A September 2023 conference biography he shared described his earlier work as algorithmic trading at quantitative hedge funds and identified him as Gradient’s co-founder and Chief Architect at that time.[1][4]

Huang and his friend Chris Chang had wanted to build something together. In Huang’s account, their experience with internal tooling helped shape the enterprise focus: replacing an ML platform could mean rebuilding workflows over multiple quarters or years. With fellow co-founder Forrest Moret, they sought to reduce that friction in shipping useful workloads. Huang described his vision of an assistant as a system that could improve alongside its user and remain flexible when unfamiliar data arrived.[1][3]