Gradient.ai

Gradient.ai developed an enterprise platform for fine-tuning and deploying specialized language models, alongside research extending Llama 3's context window beyond one million tokens. Its co-founders were Chris Chang, Mark Huang and Forrest Moret. In a May 2024 interview, Huang described the company's ambition as moving enterprise robotic process automation and codified workflows toward more autonomous agents.[3][4][1]

That agent-oriented vision followed Gradient's earlier work on model customization and deployment: at its public launch in October 2023, the platform focused on helping enterprises adapt multiple language models to their own data and tasks.[3][1]

Gradient emerged from stealth on October 5, 2023, with $10 million in seed funding led by Wing VC, with participation from Mango Capital, Tokyo Black, The New Normal Fund, Secure Octane and Global Founders Capital. TechCrunch's launch account identified Chang as CEO.[3]

Huang later traced the enterprise focus to the founders' experience with internal tooling and disruptive ML-platform migrations. He had moved from quantitative finance into product analytics at Box and streaming analytics, search and deep learning at Splunk. In his account, he and Chang wanted to reduce the operational friction of shipping AI workloads, rather than repeatedly rebuilding systems through migrations that could take quarters or years.[1]

The launch platform hosted open-source models, including Meta's Llama 2, for customer fine-tuning. It could serve models through an API or deploy AI systems into customers' Google Cloud, Azure or AWS environments. Gradient also offered models for tasks such as data reconciliation and paperwork processing, and for industries including finance and law. Its infrastructure integrated with vector databases, including Pinecone.[3][2]

The central product decision was to support multiple specialized models. Chang argued that a single generalist model imposed tradeoffs in task-specific performance; he said Gradient could integrate thousands of models into one system, a capacity claim rather than a count of deployed models. In October 2023, Gradient reported around 20 enterprise customers and thousands of combined users to TechCrunch. The company had 17 full-time employees and aimed to reach 25 by year-end.[3]

Gradient's long-context project addressed Llama 3's original 8,000-token window. Huang recalled that one million tokens was a “North Star” rather than a result the team knew it could achieve. The 1048k model used Meta-Llama-3-8B-Instruct as its base and extended the window beyond 1.04 million tokens. Its development citation credits Leonid Pekelis, Michael Feil, Forrest Moret, Mark Huang and Tiffany Peng.[1][4]

The method combined positional-encoding adjustment and progressive training. Gradient used NTK-aware interpolation to initialize a schedule for RoPE theta—the parameter governing the frequencies of rotary positional embeddings—then optimized theta empirically as context grew. Training proceeded through roughly 65k, 262k, 524k and 1,048k-token sequences. Gradient reported approximately 1.4 billion training tokens across those stages, including about 830 million in the final stage.[4][1]

Crusoe sponsored the compute, and Gradient built on Zhang Peiyuan's EasyContext implementation of Blockwise RingAttention. Ring Attention distributes attention computation across GPUs while overlapping computation with communication. Gradient adapted the implementation to its cluster topology and added parallelism and load balancing. The final two training stages each used 512 NVIDIA L40S GPUs; Gradient listed wall-clock training times of 61 and 87 minutes respectively, excluding preparation and evaluation.[4][1][6]

In a technical account hosted by Crusoe, Gradient's Michael Feil and Leo Pekelis attributed the L40S choice to availability in a constrained GPU market and FP8 performance relative to A100s. Using a price of $1.45 per GPU-hour, they estimated fine-tuning costs of $12.83 per million tokens for the 8B model and $167.56 for the 70B model. These were calculated training-cost estimates, not total project expenditures.[6]

Gradient generated long training sequences by augmenting SlimPajama and then fine-tuned on an UltraChat-based chat dataset. Huang said the team used GPT-4 to rephrase and reformat parts of the chat data. He emphasized that sequence length alone was insufficient: examples answerable entirely from the last 200 tokens would not teach a model to connect information across the full context.[4][1]

Gradient evaluated retrieval with Needle in a Haystack and used RULER for harder tasks, including multiple-value retrieval, variable tracking and word counting. Its model family included 8B variants reaching 4 million tokens and 70B variants reaching 1 million. The May 2024 interview discussed benchmark degradation at 4 million tokens; Huang suggested large theta values and numerical precision as possible explanations. Passing retrieval tests did not establish equivalent reasoning quality throughout the advertised window.[4][1][6]

Gradient also developed Albatross, a family of finance-specific language models. Its public v-alpha-tross demonstration used Llama 2 70B with additional finance pretraining, supervised fine-tuning and direct preference optimization. Corpus selection combined public-repository crawling, a likelihood-ratio filter adapted from LiRA membership inference to identify material unlikely to be in the base model's training data, and review by finance professionals.[5]

Gradient reported scores of 73.28 on H6, 61.79 on GSM8k and 100.0 on its own sec_tables_v1 financial-table metric, matching GPT-3.5 on that last evaluation. The company described v-alpha-tross as an early, limited-capability demonstration, warned that it had not been specifically aligned for safety, and said it did not plan continued internal development of that legacy model. That decision applied to the demonstration; the company separately described Albatross release versions with expanded training and architecture augmentation.[5]