This matches my instincts too. When trying to debug ML learning pipelines or research projects, LLM suggestions I've received are almost always techically correct, but they point out small, tedious details which don't really matter. In my experience, they miss the forest for the trees.
I'm bearish that continued scaling will fully solve this. In the meantime though, as a commentor in the OP points out - by solving the menial stuff (building diagnostic tools, producing visualisations, scaffolding experiments) they're still proving to be extremely useful research assistants.
Llms are far from being ingenious or innovative. The main way they can surprise you today is pulling in information that’s new TO YOU - I impressed with the algorithms they pop out sometimes simply because never formally studied compsci. But when it treads on my home turf I find it very basic, and for bigger picture stuff like system architecture its terrible, by its nature it doesn’t have a “vision” of how the whole system fits together
Strong agree. My 30 years of experience building computers and breaking them has made me better because I don’t trust my knowledge or myself as a first principle. If I didn’t have that and I could easily see myself in an alternate reality, coming in late to the game to leverage ai to build a car out of seatbelts; harnessing all the power of 1000 suns seeing that idea to fruition.
The very best LLM can answer is the answer you can expect from an online forum like Reddit or a book/paper published more than a year ago.
If you know how they work and how they are trained this makes perfect sense.
Going out and searching the web gets around model cutoffs, so fixes the latency of knowledge issue a bit, but they’re still finding statistical modes of what they’ve seen before.
“Stochastic parrot” is seen as a crude and dismissive jibe at LLMs, but it’s actually a concise observation of the reality.
That's not really true any more, since given tool use and iterative 'chain of thought' reasoning, it's more like an answer refined by conversation with the asker or the second best answerer, from the best answerer, is the best that you can expect.
E.g. consider the best answer describes how to achieve something with some CLI, but the CLI's changed and the command is now broken. It can discover that, combine with the 'best answer' about what changed in the CLI, and fix it to work for the original problem even if that's completely novel and there were no extant answers about solving the problem with the new CLI form, or that it had broken that use case.
It doesn't change fundamentally how they work of course, but 'thinking', creating that feedback loop, made them significantly more useful IMO.
The “conversation” is just one between some redditors or book publishers.
It’s going to narrow in on a statistically better answer but it’s not going to create novel insight beyond the arguments made from compressed snapshots of existing knowledge.
There is no “thinking”. Just more attention, which is apparently, all you need.
> The very best LLM can answer is the answer you can expect from an online forum like Reddit or a book/paper published more than a year ago.
Hard disagree.
Claude is able to come up with solutions in very niche areas where there is little discussion on the web, because it takes steps to digest the source of (for instance) frameworks and libraries, and can then use/advise on lots of esoteric options, edge-cases and all sorts of stuff that doesn't get discussed on the open web.
I work in a niche where it is utterly hopeless at this, and everyone in the niche agrees: it produces things that look convincing to non-experts but that is fundamentally derivative dross.
It’s great at bread and butter stuff, but not at specialised ideation.
If you know differently, please share examples as maybe we can learn something here.
I'd argue it is the other way around - LLMs are great at pure creativity, creating something that never existed before, but not always at "pulling in information that’s new TO YOU". I use them for entertainment - jokes, short flash fiction etc. and they are excellent at it.
Exactly. If you're an expert on the topic, framework, or library, even the best LLMs are still underwhelming. Even for 15 line snippets, if you're very familiar with the programming language, you'd have written it differently.
An LLM is like a low IQ person with vast knowledge.
I have a PhD and Opus 5.5 came up with a novel solution to solve a problem in my area of research that I, and other experts in the field, had been unable to. This was from very general prompting it to find improvements over published methods.
They're solving them in a way that isn't particularly interesting because it's near impossible to read and understand, and follow, what the LLM did. It's this way not because it's so smart, but because it's so stupid and still reached the goal by pure force and effort.
I am sorry but your argument looks to contradict itself: They solve hard mathematical problems because they are stupid and use brute force? Then why a human skilled in mathematics guided them for many turns in their conversation?
If they were really stupid, a Prolog program, a genetic algorithm or a SAT solver would be enough to reach the goal. Clearly it was not the case.
Can't you kind of think of them as a genetic algorithm that has a very good hypothesis generator? Humans could, in principle, help guide that (already very impressive) hypothesis generator to make better guesses to find the proof more easily.
I think it does gel. You're right that the comment you're replying to is weakened by that fact.
It gels because they solved about 700 problems out of 4000 attempts. And they threw a ton of resources at each. So "per problem" it's less than 1 in 5, "per token" it's probably not a great record. I mean it's astonishing that it's possible at all of course. But on average, even an expert with a top model will probably have an underwhelming response to the vast majority of prompts.
and I'm guessing it was with "tool calling" ie. they can write a bunch of Lean proofs and hit compile and basically brute-force a whole bunch of hypotheses quickly?
Ie. the LLM itself isn't as "smart" as the combination of LLM, Lean, looping, search and vast gpu/cpu-time.
Look cmon, this is pure cope. The problems it has solved in TCS are far from trivial, with plenty of smart people making their careers in trying to solve them. The techniques these LLMs are using are non-trivial, and not just brute force.
On the other hand, using AI to trawl through all the grunt work, experiments, and well trodden areas means I can crawl up pretty close to that giants shoulder and spend way less effort myself answering the actually important questions so I can stand on it.
(This isn't supposed to sound self aggrandising - the giants whose shoulders I'm standing on aren't very tall, and they probably have bigger brothers and sisters, but the idiom was too good of a match to pass up using)
The thing is, without doing the grunt work you usually don't have enough understanding to answer the big questions.
Ask anyone who has an advanced degree in math or physics. They will tell you that doing the boring practice problems over and over is usually the key to understanding, not reading the high level theorems.
The proper term is: "On the shoulders of a very large pyramid of hobbits." The concept is the very core of human intelligence, also known as cultural evolution. I can't resist to recommend Joseph Henrich's book "The Secret of Our Success" every few years here, to anyone interested in human intelligence, and how it is different from other primates.
There was a critical review article with a title that summarizes the book well: "On the shoulders of a very large pyramid of hobbits, or why we are not just very smart chimps. The Secret of Our Success: How Culture is Driving Human Evolution, Domesticating Our Species, and Making Us Smarter." Obviously I got triggered by your comment, which was eerily close to that title.
True. History of Science isnt part of most curiculums. So most people dont see the path and time it took to work out why the magnet points north. But LLMs can actually show you the whole path and a map of different paths people have taken.
Using Fabel to improve some non-AI ML models a while back, I had a similar result - it could implement perfectly any suggestion I made, but it could not work autonomously due to making very poor suggestions itself, or getting distracted by the wrong things, making dumb assumptions, failing to see the big picture, etc.
These seem pretty easy to solve problems though if I could just afford 100 or 1000 agents. Let them explore and compete and work out their errors for themselves.
At a practical level though, I do a lot more 'bonus' R&D work now by just keeping the agents running and testing ideas in the background.
I feel like at this point in time only agent swarms can have truly innovative behavior. We'll see if this behavior can be distilled into something smaller and more cost effective.
The ceiling on LLM intelligence might be in sight. Sure, they are squeezing out the last juice now with RL, for example on design Opus 5.5 is clearly an upgrade. But overall, true intelligence (to think big picture, small detail, ingenious, innovative, novel, or even to just use short simple language to get the fundaments across) seems lacking just as much now as 2 years go.
How does this comment fare with the recent drop of 700 or so novel math proofs? Is math intelligence possibly the outlier because math is the most self-contained knowledge where absolute truth is the easiest to establish?
You're only looking at the symbolic part: deriving a statement from prior statements. I agree that the derivation either is correct or isn't.
But then there's the rest: the philosophy and intuition behind the symbolic manipulation. You can skip that if your goal is to use the results, but I don't think you can skip that if your goal is to develop the field.
I feel great sympathy for the mathematicians. This must be gut wrenching. It will lead to progress eventually, for sure, but right now a lot of them are in a bad place.
I've also found it very hard to leave AI agents up to their own devices for long periods and get much productive benefit from it.
I suppose it's possible that frontier labs have internal models that are so far beyond that autonomous research becomes possible, but I'm not sure that's it.
It seems to me like these models are like a very competent, important part of some hypothetical digital brain, but alone they don't have all the facilities they need to be useful, like a genius with severe ADHD and short-term memory loss.
For anything where quality isn't objective in a clear-cut way, very intentional context management is needed to get good outputs. As context size expands, performance degrades. As the number of tasks increases, performance degrees. As the number of independent decisions based on subjective criteria increases, performance degrees.
If you can get a harness that makes sure every decision is made one at a time, focused with only the context it needs, things work great. But making those harnesses is still hard, a lot of work, and something that agents seem to have no clue how to help you with.
That's a kind of bitter lesson thinking, that we take this generalist model, which already mastered everything from English to Javascript with unsupervised training, and stuff it inside an expert system so it can do long division for it. That's much similar to the last AI winter, when the researchers realized their logics couldn't cope with certain logical scenarios, but still thought they could salvage their flawed reasoners as dispatchers to specific algorithms.
LLMs will either solve issues like this at the architecture level, or declare them out-of-scope entirely.
You can try adapting to this reality in your methodology though. Divide up your project into clear cut and ambitious modules, with well-defined objectives to evaluate each part.
Codebase porting, implementing sophisticated algorithms, performance tuning, model training, reverse engineering, fuzz testing / benchmarking suites, refactor to support a new interface, etc, scale so much better than 'fix a then b. don't break c' style prompts.
I don't think its the frontier models, my opus 5.5 has no problem working on a project fairly autonomously, for example if I prompt it in the evening it will work the whole night without issues...
It's interesting that we still need to go into these "try again until you find the correct answer" loops, and often still need to tell the model to ground their claims in facts. Models have gotten better at this but I still see myself asking "are you sure?" way too often.
If they tackle that, I think models will be perceived as much smarter than they are now - even though now with the right direction they can already do so so much.
Part of the problem here is how people use these models and expectations are formed.
If you want autonomous work of such systems you can not expect that: a) each reasoning step will make sense and be correct; b) results will come very quickly - especially in the first stages of the project.
I have very positive experience in non-trivial problems with using AI as an autonomous researcher or engineer. But it worked since an agent had a separated, fully dedicated environment and enough time to build up (emerge) some strong patterns of work. Early days were underperforming and slow progress. Now it is overperforming and frequently surprising me with new stuff.
I'm old enough to have realized that I've done plenty of R&D in my life that wasn't formally named that. R&D is about the substance, the mode, the impact, not the names or titles.
Silent Bob guy just "rebuts" the original poster by spouting how many loc Claude spits out without bugs and unread tests. Which doesn't truly counter anything op said. I'm sure silent on guy can be smarter than me but come on. And yes ai report writing is often verbose and full of crap. Shouldn't need special prompts for it to be focused get to the point
This reads like someone who tried Codex a few times with a blank system prompt and no context and came to the conclusion LLMs are dumb. All of those four examples are handled well by Opus 5.5. Especially with context and competent direction.
The worst part of his premise is that they can be overcome by increased intelligence. He’s not describing fundamental limitations. Meaning that it’s inevitable that his examples will be “solved.” At least better than humans. Meaning the R&D is obviously valuable.
It reads to me like he’s using Astra. This is based on me reading the comment, and seeing him say he’s using Astra several times, eg:
For example, I improved the SOTA on "listing all legal Scrabble moves" by 10x just by prompting Astra over and over to try harder, to aim for <specific performance level>, to "systematically try every method in the literature that seems kind of related," etc.
Perhaps you should instead say something like “This seems amenable to Stock Dismissal #4”.
LLMs are great at doing work quickly and they don’t say something is impossible or discourage you, so you can really tackle problems that previously would have taken years of work.
They aren’t capable of generating truly innovative ideas though. Ask it to come up with a name, it will come up with the same weird slop names over and over. Always gives people the surname Okafor. Because it works by generating probable answers, it can only generate probable names.
I think the reason models' default creative output is so bland is sort of like why if you combine all color crayons it just turn out brown.
These models actually can create content of myriad styles very well. But artistic styles are mutually incompatible. It can't create something that matches all of them, and it doesn't have an opinion on which is best, so bland requests just give bland results. To get good results you need to direct the model into the region of its latent space that makes stuff you thing is good.
> Because it works by generating probable answers, it can only generate probable names.
That's not true anymore. I've asked it to generate names for some LLCs and it came up with plenty of novel names that sounded vaguely Latin and had a certain sound that I had asked for.
I'm bearish that continued scaling will fully solve this. In the meantime though, as a commentor in the OP points out - by solving the menial stuff (building diagnostic tools, producing visualisations, scaffolding experiments) they're still proving to be extremely useful research assistants.
Such a good turn of phrase.
If you know how they work and how they are trained this makes perfect sense.
Going out and searching the web gets around model cutoffs, so fixes the latency of knowledge issue a bit, but they’re still finding statistical modes of what they’ve seen before.
“Stochastic parrot” is seen as a crude and dismissive jibe at LLMs, but it’s actually a concise observation of the reality.
E.g. consider the best answer describes how to achieve something with some CLI, but the CLI's changed and the command is now broken. It can discover that, combine with the 'best answer' about what changed in the CLI, and fix it to work for the original problem even if that's completely novel and there were no extant answers about solving the problem with the new CLI form, or that it had broken that use case.
It doesn't change fundamentally how they work of course, but 'thinking', creating that feedback loop, made them significantly more useful IMO.
It’s going to narrow in on a statistically better answer but it’s not going to create novel insight beyond the arguments made from compressed snapshots of existing knowledge.
There is no “thinking”. Just more attention, which is apparently, all you need.
Hard disagree.
Claude is able to come up with solutions in very niche areas where there is little discussion on the web, because it takes steps to digest the source of (for instance) frameworks and libraries, and can then use/advise on lots of esoteric options, edge-cases and all sorts of stuff that doesn't get discussed on the open web.
It’s great at bread and butter stuff, but not at specialised ideation.
If you know differently, please share examples as maybe we can learn something here.
They literally generate stuff that's averages of other stuff.
It's not interesting in the least
Usually commentators say that they have heard so and so version of the joke, or that it's a repeat, but they did not
Both from opus. Sol/astra could not be funny. The versions that worked I had fed top posts from r/jokes as references and gave abstract directions.
However, the core premises were not new even if the construction was. And it's a lot of work to get something useful out.
https://youtu.be/wBdkmI03V6g
An LLM is like a low IQ person with vast knowledge.
How many people on this planet are experts on every topic?
>An LLM is like a low IQ person with vast knowledge.
Are you aware that frontier models have scored in the 135-195 IQ range when they have been tested? How smart are you?
If they were really stupid, a Prolog program, a genetic algorithm or a SAT solver would be enough to reach the goal. Clearly it was not the case.
It gels because they solved about 700 problems out of 4000 attempts. And they threw a ton of resources at each. So "per problem" it's less than 1 in 5, "per token" it's probably not a great record. I mean it's astonishing that it's possible at all of course. But on average, even an expert with a top model will probably have an underwhelming response to the vast majority of prompts.
Ie. the LLM itself isn't as "smart" as the combination of LLM, Lean, looping, search and vast gpu/cpu-time.
On the other hand, using AI to trawl through all the grunt work, experiments, and well trodden areas means I can crawl up pretty close to that giants shoulder and spend way less effort myself answering the actually important questions so I can stand on it.
(This isn't supposed to sound self aggrandising - the giants whose shoulders I'm standing on aren't very tall, and they probably have bigger brothers and sisters, but the idiom was too good of a match to pass up using)
Ask anyone who has an advanced degree in math or physics. They will tell you that doing the boring practice problems over and over is usually the key to understanding, not reading the high level theorems.
There was a critical review article with a title that summarizes the book well: "On the shoulders of a very large pyramid of hobbits, or why we are not just very smart chimps. The Secret of Our Success: How Culture is Driving Human Evolution, Domesticating Our Species, and Making Us Smarter." Obviously I got triggered by your comment, which was eerily close to that title.
These seem pretty easy to solve problems though if I could just afford 100 or 1000 agents. Let them explore and compete and work out their errors for themselves.
At a practical level though, I do a lot more 'bonus' R&D work now by just keeping the agents running and testing ideas in the background.
Did you see the chart on model performance over time on LifeVal? Ceiling in there. https://openai.com/index/lifeval/
You're only looking at the symbolic part: deriving a statement from prior statements. I agree that the derivation either is correct or isn't.
But then there's the rest: the philosophy and intuition behind the symbolic manipulation. You can skip that if your goal is to use the results, but I don't think you can skip that if your goal is to develop the field.
I feel great sympathy for the mathematicians. This must be gut wrenching. It will lead to progress eventually, for sure, but right now a lot of them are in a bad place.
Edit: for readability.
Citation needed.
I suppose it's possible that frontier labs have internal models that are so far beyond that autonomous research becomes possible, but I'm not sure that's it.
It seems to me like these models are like a very competent, important part of some hypothetical digital brain, but alone they don't have all the facilities they need to be useful, like a genius with severe ADHD and short-term memory loss.
For anything where quality isn't objective in a clear-cut way, very intentional context management is needed to get good outputs. As context size expands, performance degrades. As the number of tasks increases, performance degrees. As the number of independent decisions based on subjective criteria increases, performance degrees.
If you can get a harness that makes sure every decision is made one at a time, focused with only the context it needs, things work great. But making those harnesses is still hard, a lot of work, and something that agents seem to have no clue how to help you with.
LLMs will either solve issues like this at the architecture level, or declare them out-of-scope entirely.
Codebase porting, implementing sophisticated algorithms, performance tuning, model training, reverse engineering, fuzz testing / benchmarking suites, refactor to support a new interface, etc, scale so much better than 'fix a then b. don't break c' style prompts.
If they tackle that, I think models will be perceived as much smarter than they are now - even though now with the right direction they can already do so so much.
It’s a dance, not an orchestration.
If you want autonomous work of such systems you can not expect that: a) each reasoning step will make sense and be correct; b) results will come very quickly - especially in the first stages of the project.
I have very positive experience in non-trivial problems with using AI as an autonomous researcher or engineer. But it worked since an agent had a separated, fully dedicated environment and enough time to build up (emerge) some strong patterns of work. Early days were underperforming and slow progress. Now it is overperforming and frequently surprising me with new stuff.
The worst part of his premise is that they can be overcome by increased intelligence. He’s not describing fundamental limitations. Meaning that it’s inevitable that his examples will be “solved.” At least better than humans. Meaning the R&D is obviously valuable.
They aren’t capable of generating truly innovative ideas though. Ask it to come up with a name, it will come up with the same weird slop names over and over. Always gives people the surname Okafor. Because it works by generating probable answers, it can only generate probable names.
These models actually can create content of myriad styles very well. But artistic styles are mutually incompatible. It can't create something that matches all of them, and it doesn't have an opinion on which is best, so bland requests just give bland results. To get good results you need to direct the model into the region of its latent space that makes stuff you thing is good.
That's not true anymore. I've asked it to generate names for some LLCs and it came up with plenty of novel names that sounded vaguely Latin and had a certain sound that I had asked for.