Sebastian Raschka

Sebastian Raschka is an LLM research engineer and the author of Build a Large Language Model (From Scratch) and Build a Reasoning Model (From Scratch). His contributions include educational PyTorch implementations, experiments on efficient finetuning, and explanations of reasoning-model development. He previously taught and researched machine learning at the University of Wisconsin–Madison and subsequently worked as a senior engineer at Lightning AI.[4][5][7][8][9][11]

Raschka describes a hands-on approach to learning: read a paper, inspect its configuration or reference implementation, and often build a smaller version in PyTorch. He values the manual work because it exposes details that diagrams can omit. As he puts it, “Working code doesn’t lie”.[5]

His 2024 book Build a Large Language Model (From Scratch) develops a GPT-like model through text processing, attention, model implementation, pretraining, and finetuning for classification and instruction following. Its official companion repository uses PyTorch without external LLM libraries. The main-chapter code is designed for conventional laptops without specialized hardware; the project builds a small educational model and also provides code for loading larger pretrained weights for finetuning.[7]

Raschka makes a concrete maintenance choice for that repository: he restricts contributions that would extend the main-chapter contents, keeping the code consistent with the printed book. He still welcomes feedback and questions through GitHub Discussions and the Manning forum.[7]

In [Build a Reasoning Model (From Scratch)](https://www.manning.com/books/build-a-reasoning-model-from-scratch), published in June 2026, he begins with a pretrained LLM. The book first develops generation and evaluation tools, then implements inference methods that leave model weights unchanged—such as self-consistency, response scoring and self-refinement—before moving to reinforcement learning with verifiable rewards and distillation. It addresses cost and latency tradeoffs, including cases where refinement makes answers worse.[8][7]

In [Understanding Reasoning LLMs](https://magazine.sebastianraschka.com/p/understanding-reasoning-llms), published February 5, 2025, Raschka organizes the field into four approaches: inference-time scaling, pure reinforcement learning, supervised finetuning combined with reinforcement learning, and supervised finetuning or distillation. Using DeepSeek’s models as examples, he distinguishes spending more computation during inference from changing a model through training.[11]

Raschka argues that reasoning models suit complex mathematics, puzzles and coding challenges, while using them for every task can add expense, verbosity and errors from “overthinking.” His advice is explicit: “Use the right tool (or type of LLM) for the task.” He considers distillation attractive for smaller, more efficient models, while emphasizing that it depends on a stronger existing model to supply training data. He also distinguishes instruction finetuning on generated outputs from classical distillation using teacher-model logits.[11]

Latent.Space’s revised 2025 AI Engineer Reading List recommends his reasoning survey, calling it the “best literature review.” On February 26, 2026, Raschka participated alongside Nathan Lambert and swyx in SAIL Live #6, a discussion of Anthropic distillation and model cheating.[2][3]

Before these reasoning-model publications, Raschka’s university teaching and work on efficient finetuning supplied earlier examples of his research and implementation practice.[5][9][10]

As an Assistant Professor of Statistics at UW–Madison, Raschka researched machine learning and deep learning methods, with applications in computer vision and computational biology. He taught statistics, machine learning and deep learning before moving to industry full time. At Lightning AI, he worked as a senior engineer; in his November 19, 2023 LoRA experiments article, he described co-maintaining Lit-GPT with his colleagues there.[6][5][9]

Those experiments examined practical choices for finetuning a 7B Llama 2 model on Alpaca instruction data. Raschka reported 21.33 GB of memory and 1.85 hours for default 16-bit LoRA, compared with 14.18 GB and 2.79 hours for 4-bit QLoRA: lower memory use at the cost of longer training. Doubling iterations over the 50,000-example dataset reduced performance in his experiment; he proposed overfitting as an explanation requiring further investigation. He treated LoRA rank and alpha as settings to test, with alpha equal to twice the rank serving as a conditional heuristic.[9]

On February 18, 2024, Raschka published a from-scratch LoRA and DoRA tutorial. He explained LoRA, credited to Hu and colleagues, as learning a weight update through two smaller matrices. DoRA separates weight magnitude from direction, applies LoRA to direction, and trains magnitude separately. Raschka’s contribution was the explanation and PyTorch implementation of these methods; he credited the DoRA researchers and acknowledged inspiration from the catid/dora implementation.[10]