I've got a question. I'm having a bit of trouble phrasing it, so please don't assume I'm against learning for learning's sake, that's not the point.
Whenever I've tried to learn about LLMs, machine learning, etc. I always come out with a bad feeling, compared to learning about software.
Theoretical courses make me feel like it's all baby steps, unless you assume a level of commitment equivalent to a math degree. Practical/applied stuff seems to require levels of compute, training data, etc. that make it unapproachable for a home hacker for everything beyond basic exercises like a handwritten digit classifier.
Now this feeling of "impracticality" is perfectly fine. It would be the expected outcome if I was taking a course on classic literature or modern art. But machine learning stuff feels like it could have room for learning-by-doing applications that aren't entirely pointless, so I wonder if there's some way to approach the topic I'm simply not finding.
Did anyone have the same experience? any tips if so?
I think this is an outcome of playing with systems that we don’t actually understand very well on a deep level. We have found recipes that work, and have used graduate descent (so many PhD theses) to understand and optimize these systems, but we’ve built a model of language that also encodes reasoning, and those things are extraordinarily complex. Providing proper mathematical rigor for their explanation requires, as you say, the level of commitment equivalent to a math degree. Also, because language is so complicated and subtle, you need highly precise statistical models of it, and that just takes a shitload of data; hence the scale. You can train toy models at home, but they will suck because you actually need massive amounts of data to capture the nuance that language requires. And that’s to say nothing of multimodal models.
You say "...because you actually need massive amounts of data to capture the nuance that language requires..." but humans don't require that! Why do LLMs?
How many books did you read in your education? Far fewer that what LLMs read, no doubt.
Of course we "have a life" so we interact with hundreds of people, too.
But in any case there's are fundamental differences here between these two systems (LLMs and humans) that we have not yet discerned clearly.
I completely understand where you're coming from. While understanding the math and architectures of these large models can be edifying, it lacks some of the liberating/empowering spirit of traditional software. You can build a bespoke to-do app that is better suited to your needs than anything produced by some productivity software company, but you're totally beholden to one of the few big players for access to an effective LLM.
But there is so, so much interesting work and research to do at a much smaller scale. I work on drug discovery models, and none of my work requires data or infrastructure that are not generally accessible to the public (it's nice to have the computing resources of my institution but there are other ways to access a GPU).
So if you are really interested in this stuff, I would suggest looking at the endless supply of interesting problems in domains that are not dominated by huge frontier labs.
Get Sebastian's book and build an LLM from scratch. The problem with most textbooks (on any subject, it's not unique to ML/AI) is they usually try to build the theoretical foundations first. In many cases things are better taught top down.
I'm OK going through the learning itself (will definitely check out the book though, since it's so well liked!). The question is more, what do you do with that once the book or course is done?
For a junior coder, there's usually a path to recommend, joining some open source project or doing some home scripting, etc. Something where you can be useful even if it's not your full time job.
I'm wondering whether there's some similar hobbyist path available here, or whether there's too much of a jump between the educational/pop sci stuff and the professional realm. If it's the second, that's also fine, I just want to know if I'm missing something.
One practical-ish thing you could reasonably do with this (beyond just better understanding how models work so you can make better choices/tradeoffs), is post-train some of the small-to-medium models, fine-tune them on your own data or priorities.
You don't need to change many weights, relatively, to add capabilities or modify behavior. Look up LoRas for instance.
I'm of the opinion that post-training existing models is an area with a lot of benefit and fun in the near future (it's already being done here and there of course).
I doubt you _really_ have to know everything in this course to know how to do that, but I'd want this as background personally for it.
I have the books on my shelf for 5 years now and never encountered the need to build an AI so i haven't touched it yet. Don't feel bad about using AI to learn but not build one.
You can google stuff to learn without the need to build a search engine too.
I have however learned a lot about math! And other related stuff. I'd argue if you extend your low level skills in math and code when the need for an AI comes up, simple weather / sales forecast, who knows, you'll have an easier time building one.
if you are willing to put in the work you can devise a hyper specialization for which you can create an RL playground (which is likely absent in the frontier labs), then you can obtain cutting edge results on it by post training an open weights model using minimal resources.
I think it's one of those things where you have to work in the field to really get it and be productive. You just don't have the time or resources otherwise.
But machine learning stuff feels like it could have room for learning-by-doing applications that aren't entirely pointless, so I wonder if there's some way to approach the topic I'm simply not finding.
That's what Karpathy's videos are for. You end up with a GPT-2.0 equivalent if you follow them to the end.
This is still pointless though. I mean it's valuable from a learning perspective but the parent is saying they want to contribute somehow. You can only learn so much from toy examples
The prereqs aren't as extensive as people fear, but they are important. Starting out by building a scratch-trained GPT3-scale model is always an option, but anyone who's going to succeed at that doesn't need to watch any YouTube videos.
For the rest of us, the sense of discovery that we get from training and running toy models is a big part of the reward. This stuff is not an exact science, at least not yet, so time spent building intuition with small-scale models is an incredibly important part of the learning process. Not pointless at all.
I feel the same. I feel machine learning is 1) you need strong math knowledge but also intuition. 2) You need actual production level data, a lot of the nuances of machine learning, applied scientists' roles are how to build strong models with real world, messy, inconvenient data. You need to apply various techniques and not just memorize a specific formula.
Compared to software coding, you build websites, apps etc. The barrier to entry is much higher.
I am trying to pivot into machine learning myself, so far my understanding is try to find a software engineer role where you deal with data, model infra, data pipeline, etc. Then within the same company, try to do more modeling work and build intuition and resume credentials in this area. Someone correct me if I am wrong.
I feel there's a misconception in a lot of the comment threads about what an LLM-from-scratch tutorial is for. Of course you're not going to come out of this kind of tutorial being able to train a useful LLM from scratch. (It requires too much compute, and datasets you don't have access to, and there are a lot of tricks and operational problems to solve for real training runs.) If that's your bar for "impracticality", then sure, this is impractical.
But that's not the point; the point is understanding these things better by seeing how they are made (or grown, if you prefer). Building something at a level further down the stack than you will ever go forces you to confront details that shape the layers above that you do deal with. This is one reason why Crafting Interpreters is such a beloved book here on HN.
LLMs are now at the core of tools we all use everyday... don't you want to know how they are made? Maybe not everyone has this itch to understand the platforms and tools they use, but I sure do, and it's helped me over and over again. Sure, LLMs aren't like operating systems or compilers. They're a quasi-organic mass of billions of parameters learned during training and honestly nobody (not even the mech interp folks) really deeply understands why they work so well. But that's no reason to completely give up and there is still a lot you can learn.
Take RLVR ("Reinforcement Learning from Verifiable Rewards"). This is the training technique that made LLMs get so good at programming and math in 2025 and 2026. A lot of people who confidently talk about LLM capabilities either haven't heard of it (and thus end up parroting out-of-date factoids about LLM training derived from the pre-2025 era) or have only a very dim idea of what it is. In this course you actually do RL training of an LLM to get it solving math problems better. You're right down in the details. It's interesting for its own sake but it will also give you a grounding for discussing questions such as: "why did LLM writing get so weird and bad in mid-2026?" and "why is it so hard to train LLMs to be good at things that aren't like math and programming?"
Anyway, Sebastian's two books are great and highly recommended; I worked through the first one a year ago and it was one of the best investments of my time in all of 2025. If you prefer video for learning this is great as well (although I'd still say - try the books first).
I got his book Build Your Own LLM From Scratch and it's super readable and does a great job at holding your hand through the entire process...need to finish reading it though!
I did the same. I got "Build Your Own LLM From Scratch" last christmas and it is a great resource for learning how LLMs work, from tokenization to generating text. The author makes a great job explaining each new concept. If you know PyTorch (or at least know about tensors), the material will be approachable for you.
Sebastian, if you are reading this, this is an idea for another book: how to orchestrate agents for adversarial code review. I've been looking for learning resources to make something similar to this [1] at a smaller scale. I am aware of codex subagents, though. However, given that Sebastian already has a book for LLMs and another book on planning, I think adversarial code review is an interesting next step.
I don't like working with neural networks and I always found everything overcomplicated or oversimplified, sometimes both at the same time.
But this book is almost perfect. Every new concept is well described and every is well founded. And every step moves you forward (if you know, you know)
I like it too. Haven't got very far through it yet but just learning about how to use PyTorch and finally getting how back propagation works via autograd etc. was really fun and enlightening.
What would be the computing cost to go through the book (or the course)? Do you need Cloud GPUs, or can this be run locally, if so how powerful your computer has to be?
I've been wondering about getting a PC to poke around LLM stuff (not a practical use, but more for learning purposes) but I don't have any intuition about how much is enough and whether it is worth it to me.
All of these build your own model guides are like someone putting out a build your own CPU guide 10 years ago. Interesting, but not of any practical use.
I’ve read, worked through, and watched his LLM book as well as this one! Things finally clicked, can’t recommend both books enough—his pedagogy is great (at least for me).
I just started reading his book, and he makes it easy to grasp difficult concepts. As a matter of fact, I have been working on platform where you can learn how AI models work in an interactive manner.
This is an essential resource and his book is a must read from this author. I have the upmost respect for Raschka for his research and detailed explanations in AI including him dissecting research papers.
I'm a bit confused as to who this is for - apart from learning how the tech works that is.
If you're truly going to build your own model (and commit to the cost of that), then video tutorials will not cut it.
And if you don't really, then I wonder if this is really the best style of conveying the information you'd be interested in?
I mean I might be wrong and maybe this helps emerging players or something like that, but to me it is more likely that this is more of a monetisation strategy of a personal brand. Living off selling to people that think they _should_ but then in the end don't.
Why teach people to play guitar once a week if they won't be a new Jimi Hendrix? Why teach people to run if they won't be a new Usain Bolt?
Maybe because they don't want to be an "emerging player" or something, they only want to understand how a new technology that is changing the world works and prefer to learn by doing.
If Sebastian is doing it by its personal brand, he really deserves it. I would trust him because I know he is a reference on his area of knowledge and I know he is because he shares his knowledge instead of only making fancy talks.
Let’s not be this negative, we might be going towards a new direction with technology that could allow small teams to build their own models. Maybe not frontier models. Maybe not close to the others in generalized benchmarks. But perhaps they could be really specialized into solving one particular problem better than the others.
If you're truly going to build your own model (and commit to the cost of that), then video tutorials will not cut it.
That makes no sense. It's like saying if you want an engineering degree, going to lectures will not cut it. Of course not, you have to do the homework and projects too.
You're right about me not being the target audience, but you're wrong about the second sentence I think.
The target audience of this is for people that like to _think_ they're doing stuff by paying for it, to then not do anything.
It's a purchase of an identity, basically.
Which.. I guess is fine as something to cover Sebastian's expenses to enable him to keep delivering this free stuff. If you want to take money from someone, taking from this group is likely the most ethical thing to do.
"I know I'm not the target audience and I don't have any interest in how things work, but let me tell you how you're wrong by telling you who the actual target audience is"
What's next? Are you going to give us your opinion of music genres you don't ever listen to? And tell us who should listen to them too, I suppose.
You are completely off base. His initial book was released almost two years ago. I post train models, for work, and I learned more from his book on LLMs than any other source on LLMs. There is a lot more to it than that book, but you have to start somewhere, even if you have a decent theoretical foundation. There are a bunch of people who learned how to build them from his book.
How do you think people learn to build/train models? It comes down from the sky?
You sound like you are projecting a bunch of motivations and assumptions about character onto other people. But you projecting such things onto another person is a terrible way to begin to understand something.
There are genuinely people who enjoy learning how something works by doing smaller scale versions of it themselves. This site happens to have a much higher average percentage of them than most other places.
Whenever I've tried to learn about LLMs, machine learning, etc. I always come out with a bad feeling, compared to learning about software.
Theoretical courses make me feel like it's all baby steps, unless you assume a level of commitment equivalent to a math degree. Practical/applied stuff seems to require levels of compute, training data, etc. that make it unapproachable for a home hacker for everything beyond basic exercises like a handwritten digit classifier.
Now this feeling of "impracticality" is perfectly fine. It would be the expected outcome if I was taking a course on classic literature or modern art. But machine learning stuff feels like it could have room for learning-by-doing applications that aren't entirely pointless, so I wonder if there's some way to approach the topic I'm simply not finding.
Did anyone have the same experience? any tips if so?
How many books did you read in your education? Far fewer that what LLMs read, no doubt.
Of course we "have a life" so we interact with hundreds of people, too.
But in any case there's are fundamental differences here between these two systems (LLMs and humans) that we have not yet discerned clearly.
But there is so, so much interesting work and research to do at a much smaller scale. I work on drug discovery models, and none of my work requires data or infrastructure that are not generally accessible to the public (it's nice to have the computing resources of my institution but there are other ways to access a GPU).
So if you are really interested in this stuff, I would suggest looking at the endless supply of interesting problems in domains that are not dominated by huge frontier labs.
For a junior coder, there's usually a path to recommend, joining some open source project or doing some home scripting, etc. Something where you can be useful even if it's not your full time job.
I'm wondering whether there's some similar hobbyist path available here, or whether there's too much of a jump between the educational/pop sci stuff and the professional realm. If it's the second, that's also fine, I just want to know if I'm missing something.
You don't need to change many weights, relatively, to add capabilities or modify behavior. Look up LoRas for instance.
I'm of the opinion that post-training existing models is an area with a lot of benefit and fun in the near future (it's already being done here and there of course).
I doubt you _really_ have to know everything in this course to know how to do that, but I'd want this as background personally for it.
You can google stuff to learn without the need to build a search engine too.
I have however learned a lot about math! And other related stuff. I'd argue if you extend your low level skills in math and code when the need for an AI comes up, simple weather / sales forecast, who knows, you'll have an easier time building one.
That's what Karpathy's videos are for. You end up with a GPT-2.0 equivalent if you follow them to the end.
For the rest of us, the sense of discovery that we get from training and running toy models is a big part of the reward. This stuff is not an exact science, at least not yet, so time spent building intuition with small-scale models is an incredibly important part of the learning process. Not pointless at all.
Compared to software coding, you build websites, apps etc. The barrier to entry is much higher.
I am trying to pivot into machine learning myself, so far my understanding is try to find a software engineer role where you deal with data, model infra, data pipeline, etc. Then within the same company, try to do more modeling work and build intuition and resume credentials in this area. Someone correct me if I am wrong.
But that's not the point; the point is understanding these things better by seeing how they are made (or grown, if you prefer). Building something at a level further down the stack than you will ever go forces you to confront details that shape the layers above that you do deal with. This is one reason why Crafting Interpreters is such a beloved book here on HN.
LLMs are now at the core of tools we all use everyday... don't you want to know how they are made? Maybe not everyone has this itch to understand the platforms and tools they use, but I sure do, and it's helped me over and over again. Sure, LLMs aren't like operating systems or compilers. They're a quasi-organic mass of billions of parameters learned during training and honestly nobody (not even the mech interp folks) really deeply understands why they work so well. But that's no reason to completely give up and there is still a lot you can learn.
Take RLVR ("Reinforcement Learning from Verifiable Rewards"). This is the training technique that made LLMs get so good at programming and math in 2025 and 2026. A lot of people who confidently talk about LLM capabilities either haven't heard of it (and thus end up parroting out-of-date factoids about LLM training derived from the pre-2025 era) or have only a very dim idea of what it is. In this course you actually do RL training of an LLM to get it solving math problems better. You're right down in the details. It's interesting for its own sake but it will also give you a grounding for discussing questions such as: "why did LLM writing get so weird and bad in mid-2026?" and "why is it so hard to train LLMs to be good at things that aren't like math and programming?"
Anyway, Sebastian's two books are great and highly recommended; I worked through the first one a year ago and it was one of the best investments of my time in all of 2025. If you prefer video for learning this is great as well (although I'd still say - try the books first).
Sebastian, if you are reading this, this is an idea for another book: how to orchestrate agents for adversarial code review. I've been looking for learning resources to make something similar to this [1] at a smaller scale. I am aware of codex subagents, though. However, given that Sebastian already has a book for LLMs and another book on planning, I think adversarial code review is an interesting next step.
[1] https://bun.com/blog/bun-in-rust
[2] https://learn.chatgpt.com/docs/agent-configuration/subagents...
But this book is almost perfect. Every new concept is well described and every is well founded. And every step moves you forward (if you know, you know)
I've been wondering about getting a PC to poke around LLM stuff (not a practical use, but more for learning purposes) but I don't have any intuition about how much is enough and whether it is worth it to me.
If you're truly going to build your own model (and commit to the cost of that), then video tutorials will not cut it.
And if you don't really, then I wonder if this is really the best style of conveying the information you'd be interested in?
I mean I might be wrong and maybe this helps emerging players or something like that, but to me it is more likely that this is more of a monetisation strategy of a personal brand. Living off selling to people that think they _should_ but then in the end don't.
It's also not a bad foundation for people who want to understand how these things are trained and work.
Maybe because they don't want to be an "emerging player" or something, they only want to understand how a new technology that is changing the world works and prefer to learn by doing.
If Sebastian is doing it by its personal brand, he really deserves it. I would trust him because I know he is a reference on his area of knowledge and I know he is because he shares his knowledge instead of only making fancy talks.
some people enjoy learning how the tech works
It's a critique of the people and the lack of substance. But not of the author. The author is highly capable.
That makes no sense. It's like saying if you want an engineering degree, going to lectures will not cut it. Of course not, you have to do the homework and projects too.
Something is off here. It indeed makes no sense
The target audience of this is for people that like to _think_ they're doing stuff by paying for it, to then not do anything.
It's a purchase of an identity, basically.
Which.. I guess is fine as something to cover Sebastian's expenses to enable him to keep delivering this free stuff. If you want to take money from someone, taking from this group is likely the most ethical thing to do.
What's next? Are you going to give us your opinion of music genres you don't ever listen to? And tell us who should listen to them too, I suppose.
How do you think people learn to build/train models? It comes down from the sky?
He is covering expenses. Probably barely.
It seems that you want to oppose something really badly today, but you're struggling to find something that makes sense for more than 5 minutes.
There are genuinely people who enjoy learning how something works by doing smaller scale versions of it themselves. This site happens to have a much higher average percentage of them than most other places.