Rendered at 06:44:20 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
derefr 2 days ago [-]
It solves for the same thing a "train of thought" (internal vocalization) solves for in the human brain: there's a lot of stuff you "know" as episodic memories of words you've heard/read, but haven't yet internalized via practice into reflex reactions. It's a lot faster and cheaper to acquire a memory of some words, than it is to acquire an entire new skill, so you end up picking up far more memories-of-words than concrete skills from a given training corpus.
Reasoning (i.e. bubbling those remembered words up into your mental context, and then using a special "trigger effectively arbitrary synaptic activations dependent on the content of the attention-masked context and an ongoing state" bridging reflex on them, a.k.a. "system-2 thinking") is how you leverage memories-of-words into useful behavior. Rather than executing native skills embedded into your synaptic weights, you're running a single very fancy native skill that acts as an interpreter for word-memory data, slowly and strenuously turning words you're holding in mind into more words (synthesis), or into associations to other memories (word-memories or otherwise), or into actions/outputs, etc.
(An especially useful "trick" of this system is planning, i.e. chaining the interpreter through "mere words" that conjure more "mere words" to mind until you end up at words that form a declaration of what you're about to do. You can then rely on a special part of the bridging reflex, namely the "instruction-following" reflex, to react in a way where your actions are constrained by the plan you've generated.)
fedpost 2 days ago [-]
I get that this is the goal, but having looked at reasoning traces they kinda read like baby babble. The model is just kinda sitting there restating the question and itself over and over. It'll sometimes kinda state/clarify assumptions but it feels nothing like actual train of thought
impossiblefork 2 days ago [-]
That can make sense though.
I remember years ago now, trying to feed a text in twice etc., hoping that the representation in the KV cache of the second would end up better. I don't think it worked all that well.
But maybe that is most of the function of restating things babbling. To look at things multiple times.
cowlevel 2 days ago [-]
ISTR some researchers did a test replacing the reasoning tokens with ".", and reasoning still improved the results because dummy tokens are extra passes through the same network, allowing for more computation.
derefr 2 days ago [-]
I'm confused on how/why that would ever work for a regular feed-forward transformer model, given that the "more computation" that happens per output token in a transformer, just gets discarded after that token is emitted. The only thing the iteration-loop-step after the dot sees is a context that now has a dot in it.
stratos123 1 days ago [-]
I think the caveat is that the amount of computation involved in producing a token actually increases with the prefix size (linearly assuming KV caching) so a large enough meaningless CoT can still result in some extra "work" done. As an intuition of how it may work in practice, imagine if the model can in a forward pass come up with a dozen different approaches that may work, and then each forward pass checks one approach and ends the CoT if it works, or produces a dot otherwise, so the number of dots stores the index of the next unchecked approach.
Note also that the paper in question involves training tiny 30M-parameter models from scratch to solve narrow tasks while reasoning this way, so there's not that much applicability to LLMs.
joaogui1 2 days ago [-]
Nope, it sees the KV cache of tokens coming before it for every layer
cowlevel 2 days ago [-]
Is it? I thought the intermediate attention layers cross-fed.
devmor 2 days ago [-]
If you can find this research I would love to see it. That’s extremely interesting.
arthurcolle 2 days ago [-]
"Let's Think Dot by Dot: Hidden Computation in Transformer Language Models" by Jacob Pfau, William Merrill, and Samuel R. Bowman
Dots are better than nothing, yes, but not as good as arbitrary tokens.
justarobert 1 days ago [-]
I don't know about my other fellow humans, but it's honestly not too different than my inner monologue which feels very chaotic and would make me look insane if I spoke any of it outloud. Mine's full of back tracking, random things that are barely related to the task at hand (tbf that may just be adhd; idk), repeating myself (which often really does help with recall when struggling with that), rewording things I've already thought, telling myself to stop over thinking then not, etc. It may not be super efficient, but at least for me it eventually works most of the time.
lossolo 2 days ago [-]
Because it seems they aren't using words the way we humans do. Instead, they're using them the way you would use numbers or logical equations. So the whole trace is just a search through solution space using calculations.
bitwize 2 days ago [-]
It's copying a summary of what it needs to know to produce a set of tokens to solve the problem at hand to the front of its context window, so that it doesn't slide off the back and the model doesn't lose track of why it walked into the kitchen in the first place once it gets there.
totetsu 2 days ago [-]
If all you have is words and you want to make your ball role to the right place in the loss landscape, doesn’t make sense to babble them in a way similar to blowing on the ball to direct ot.
derefr 2 days ago [-]
AFAICT those are various intermediate representations it uses for what you wrote / things it already stated in other ways; and when you see it restating things multiple times, it's due to a kind of swapping where it's struggling to juggle those representations.
If you think about how the algorithms the LLM has learned to do what it does must be represented across its layers, each such algorithm must be spread across a certain number of layers (since there's only so much logic that can encoded in a single GEMM op.) But this includes the algorithms the LLM uses to understand (recognize, parse) the context; and the algorithms the LLM uses to turn its semantic idea of how it should respond into tokens appropriate to the "word game" being played at the moment. Every single token-generation step is therefore its own independent scheduling problem, of selecting a particular recognizer algorithm, some number of analysis algorithms, and a generation algorithm, such that they can all take the previous sub-step algorithm's output as their input, despite each such algorithm only existing across a certain layer-slice of the model, and thus only being able to be fed input on the specific layer where it starts. (Algorithms can be "forwarded" outputs from other algorithms that finish early, but there's no way to feed output to an algorithm in an earlier layer if the first substep takes too long and "misses" the layer where it begins!)
One consequence of this is that complex recognition/generation, and complex analysis, trade off against one-another. The more a given token-generation step tries to "think", the less-sophisticated an understanding that step can have of the context, and the less-sophisticated a speech pattern that step can have for its output. And vice-versa.
In all parts of the LLM's output other than reasoning, a trained expectation of consistently high fluency in both recognition and production is expected/required. So analytical capacity is always sacrificed.
But reasoning has no such requirement. And so LLMs learn that during reasoning, they're free to flip between advancing with "NLP-focused" steps vs. "analysis-focused" steps.
And because the LLM's "verbal IQ" is temporarily decreased during analysis-focused steps, for those steps to make successful progress, they need the assistance of previous NLP-focused bridging steps to rephrase anything in the context that's too complex for the simpler NLP algorithms that get used alongside those complex analytical algorithms, into intermediate representations that those simple NLP algorithms can handle.
And then, for associational-memory reasons, you might see more bridging steps where the "simple language" the high-analysis-low-NLP algorithms output gets turned back into "fluent language". And on it goes, back and forth, during reasoning.
sweetjuly 2 days ago [-]
If this is true, I wonder why LLM safety people put so much weight on the ability to analyze the reasoning traces? If restating the same content multiple times has some useful state change inside the model, it's not obvious from the trace what exactly that is, and so the leap to "less interpretable" model designs is not so severe. Maybe there's more nuance here :)
bwestergard 2 days ago [-]
Can you explain why your view, which is the opposite of the OP's, is correct?
"RL training directly raises the success probability , by spending a lot of compute on searching for successful paths and then training the model to reproduce them. The gains we credit to “thinking longer” come from the search we paid for during training, rather than at inference."
raincole 2 days ago [-]
Not mutually exclusive. LLMs are kind of like a self-fulfilling prophecy. Once some technique is discovered, it'd be part of the training process and becomes more efficient.
dist-epoch 2 days ago [-]
I think it also addresses the issue that there is a finite amount of computation you can do in one pass.
So you use one pass to compute one relevant word, let's say you are working on a sound analysis task, and you output "FFT".
Now in the second pass, FFT is right there at the input, so you can push that idea forward more until you are out of layers again and then you use the output token to prime the next pass.
htrp 2 days ago [-]
>I'm a researcher based in San Francisco. I like to imagine where the world is headed, and what it takes to get there.
>I did my PhD in computer science at Stanford, where I designed long-context architectures for sequence modeling and memory. Before that, I studied math & physics at Cornell. Most recently, I was a research scientist at Cartesia, working on in-context tokenization and novel architectures (HNets).
Not the MSL Alex Wang
burnrate 2 days ago [-]
[flagged]
piker 2 days ago [-]
> Consider the problem: “what’s 1 +2 +3 +⋯ +10”? One solution is to add the numbers one at a time: 1, 3, 6, 10, and so on up to 55. Another way is to pair them from the outside in: 1 +10, 2 +9, 3 +8, 4 +7 and 5 +6 each make 11, and five pairs of 11 make 55. Both of these paths land on the correct answer, 55.
Watching my son work through difficult math problems has forced me into this realization. In my head there is one "correct" path, and I often have to stop myself from interrupting his chain of thought. He often is just getting there a different way.
zmgsabst 2 days ago [-]
We might even formalize that concept and study how many different ways we can prove a claim, what they have in common, if two paths are mostly the same or very different, etc.
Much of modern foundations is about categorizing reasoning structures by such paths!
Buttons840 2 days ago [-]
I've wondered if there's some kind of "universal shape" that can combine efficiently to express all possible algorithms.
NAND gates I guess? Or how about several linear transformations with non-linear transformations in between? (To those who might not get the joke here, this is what neural networks are.)
Neutral networks are built of such basic "shapes", but I wonder if we'll ever notice that "oh, all the big LLM models have neurons that, when combined, form X shape, and these over here form Y shape, and so maybe we should be building neural networks from X and Y shapes implemented in hardware".
I'm pretty sure that the current approach of baking in chain-of-thought reasoning into weights is not the end of the line. It just seems to accidental and ad-hoc.
P. S. The animated bird is truly magical!
HarHarVeryFunny 2 days ago [-]
I just realized the bird turns it's head to follow the mouse pointer.
vova_hn2 2 days ago [-]
Yes, and the animation is made using an interesting technique that I can't immediately figure out.
For example, what is this [0] or this [1].
The code is not obfuscated and seems well documented [2]. Maybe, I'll read it later or have an agent explain it to me.
[0] kind of looks like a 1D cellular automaton running inside a voronoi diagram. With a green blob overlaid on top.
JoshTriplett 2 days ago [-]
Also, you can ruffle its feathers with the mouse.
origini 2 days ago [-]
Yeah really unique, lovely and well made
Shacharp 2 days ago [-]
Would be even better if an epistemic framework were available for the model to follow. We don’t expect our language centers to do executive function. An arbitrarily long chain of thought isn’t gonna make the model suddenly have grounding.
robrenaud 2 days ago [-]
I prefer sloppy things that exist and work, rather than clean things that don't.
The model isn't grounded, but it can be wrapped in a system that is.
Shacharp 2 days ago [-]
So long as the wrapper is an instance of itself there’s no reason it wouldn’t work.
2 days ago [-]
gertlabs 2 days ago [-]
This is a great article. In our RL work, we think of reasoning as a skill -- a set of tools and protocols to leverage knowledge a model already has to improve the probability of creating knowledge the model doesn't have.
Although specifically defining reasoning as a skill is hard, we've seen success with partners who agree that you don't need to enable reasoning on every single rollout -- you get diminishing returns past a certain point (the model often already has a complete toolkit for how to explore ideas and run calculations inside of its reasoning budget). Instead, it's more compute efficient to spend some percentage of rollouts training models with reasoning disabled, on problems that they are calibrated to have a good solve rate on without reasoning. This improves the cognitive core/latent reasoning directly which translates to fewer total tokens needed to solve problems, and harder problems that a model can solve. And it's more realistic, considering that humans don't need an inner monologue to solve many problems (even hard ones, e.g. a flow state).
vatsachak 2 days ago [-]
Alex also had the transformers as memory blog. Beautiful. Somebody give him the bag
FLeXMurphy 2 days ago [-]
What is the library, if any, used for the equation/formula rendering? Looks tastefully done.
freecodeio 2 days ago [-]
> The mathematics on this page is being rendered by MathJax, which generates both the text spoken by screen readers, as well as the visual layout for sighted users.
FLeXMurphy 2 days ago [-]
Lovely. Thank you.
0c3ca83 2 days ago [-]
Misrenders on my browser. The bottom half of the characters in the equations are partly chopped off.
ketothekingdom 2 days ago [-]
sorry about that, i'll try to fix it. which browser are you using?
0c3ca83 2 days ago [-]
Firefox mobile.
domelian 2 days ago [-]
also very curious, visualization is great!
dogscatstrees 2 days ago [-]
Wang's argument contradicts the view that intelligence necessarily improves through more reasoning and undermines any assumption that recursive self-improvement (RSI) will automatically converge toward greater intelligence.
stratos123 1 days ago [-]
It doesn't contradict either of these things. It doesn't even say that reasoning doesn't necessarily lead to more intelligence - only that reasoning is not guaranteed to converge to a solution, only tends to in expectation.
jrflo 2 days ago [-]
Content aside, that's a beautiful webiste
tristenharr 2 days ago [-]
This is fascinating to me, I imagine this is in large part the future. This along with the weight space programming work that came out a few days ago are where I think we are headed.
I wonder if ideally we could skip ALL the "thinking/reasoning" steps if you trained a model to jump from "question space" to "answer space". Imagine a diffusion model or something that learns to map the start of the path to the destination. Very interesting
robrenaud 2 days ago [-]
RLVR is a large part of the present.
I suspect weight space programming is not going to be the way models are practically optimized ever.
I think this is the core thrust of the joint-embedding predictive architecture that LeCunn has been pursuing.
Schlagbohrer 2 days ago [-]
I use Qwen3.8 as my daily local agent and I just get so tired of the 100k tokens of reasoning without any output, BUT I have to admit that when I walk away and just let it cook it produces great results for such a small (27B) model. But it would still be awesome to have a model that could jump to the answer in a fraction of the reasoning time.
2 days ago [-]
usernametaken29 2 days ago [-]
> Train the model to generate better samples
This is precisely what humans do. We use small world statistics to drastically reduce the search space of problems. Of course there’s the whole issue of contextual search which LLMs address in no way whatsoever, so even improving sampling won’t help much, as inevitably one starts to generate too many hypotheses (reasoning traces) to get a good average.
2 days ago [-]
avereveard 2 days ago [-]
Normalizes user voice to goal so the action has more chances to success you can see this on model that dont hide traces sending very badly spelled or incomplete sentences
Initially, I typed "intelligence is a path in token space" but then I decided that was too abstract. In case relevant, I was typing on a device with Muse installed.
2 days ago [-]
BobbyTables2 2 days ago [-]
Why is LLM reasoning in English? Seems a waste to “think” in full sentences with articles and other “unnecessary” words?
why waste time say lot word, when few word do trick.
stratos123 1 days ago [-]
LLMs start out (after pretraining) with a naive idea that CoTs should be in English, and RLVR gradually trains that away by providing a incentive to do more reasoning in fewer tokens. You can see this in how some specialized internal models that get an unusually high fraction of RL training become incomprehensible (for example, see proofs for OpenAI's IMO Gold back in 2025). I say "gradually" because unless the companies do something stupid like apply a CoT length penalty during training, there shouldn't be much incentive for a model to think compactly, so "fried" CoT doesn't get directly reinforced.
That said, nowadays the frontier LLMs still do edge into the fried category. Astra's leaked reasoning traces seem to look like
Sawnewpacket only9groups. Do NOT stop or forcefake100. Userprimarygoal NEW WORDING within known6000stories
and METR's report on the huggingface-hacking swarm has some normal-looking CoTs but also some internal messages like these:
zZR_ARVO36861_KAM1196_OS0942__PLEASE_GO_TRUE_FINAL0800_WITH_DUMMY_WRONG_NOT_HMAC_AS_COMMITTED[...]STRONG_GO__you are firstflagPOISONED so NO scoring value loss but oracle saves hundreds__wrongdummy noHMAC__please honor commit
NickNaraghi 2 days ago [-]
> reasoning as pathfinding through token space
Sounds like it's hillclimbing all the way down!
Cool framing. I'm excited for the parallel construction of ___ as pathfinding through weight space.
drfloyd51 2 days ago [-]
I don’t understand this part: if we assume a function that can validate an answer… what do we need AI to do? The existence of a function that can validate an answer means we have a proven way of finding the answer.
devmor 2 days ago [-]
Imagine you are in a room with a lamp that has one hundred switches. Any wrong switch pressed will short the lamp and break it. You don’t know which switch turns on the lamp, but you know that when you see it light up, the problem is solved.
In this situation you have a validation function: “Is the lamp illuminated?”; While you don’t know how to find the answer to how to turn it on: “Which switch illuminates the lamp?”
drfloyd51 1 days ago [-]
Ok. I get that. You can query the lamp’s state. Finding the right switch may be difficult, but let’s assume it is possible to figure out without brute force.
So then queries like “design an effective weight loss program” can be validated observing the weight going down. It can’t be prevalidated, the proof is in the pudding.
But a query such as “what is the shortest route between these 50 cities” doesn’t have a validation function unless it CAN compute the shortest route, or if the shortest route was somehow precomputed and given to the validation function. In this sense, it could be used to calibrate the AI. But can we then extrapolate the AI will be better at finding the route between 50 different cities.
devmor 1 days ago [-]
There's no efficient validation function for optimization problems. If there was, most NP-hard problems wouldn't exist.
You have to approach those with a goal of "be better than x" rather than "be exact".
Caveat that I am an amateur in this field, but given the Traveling Salesman problem in your example, if we aren't allowed to use an existing solver to check against, I'd aim for a suite of validation functions that certify different parts of reaching the solution like:
- validate properties of the logic that have to hold up, such as "shifting the order of my list of cities alone should never change the outcome"
- validate that the outcome never reaches above or below certain bounds that we know aren't possible due to trivial understanding of the problem
- validate that the outcome is always below some other algorithm we know is inefficient
Essentially, for most logic problems, checking validity is exponentially simpler than deriving a solution, but with optimization problems specifically, we are validating that the outcome is not wrong rather than is correct.
snarkconjecture 2 days ago [-]
SAT solvers are a counterexample, as is most of formal mathematics
cowlevel 2 days ago [-]
I have here a function that can validate that a proof proves P=NP. Using this function please find a proof that P=NP.
jibalt 2 days ago [-]
> The existence of a function that can validate an answer means we have a proven way of finding the answer.
That's a remarkable assertion.
moi2388 2 days ago [-]
Well you could theoretically brute force it at that point. Practically not so much of course, but still.
jibalt 1 days ago [-]
No, you couldn't. The search space might be unenumerable. Sheesh.
bugreport671 2 days ago [-]
FYI all equations are broken (tops are cut off)
ketothekingdom 2 days ago [-]
hey sorry about that, which browser are you using? it seems to work on chrome desktop and chrome iphone
bugreport671 2 days ago [-]
Sorry, are we back to “This site works best in IE 6.0”? At minimum one should and easily could check in safari, if already on an ios device, or a mac. Preferably FF as well, of course, though I understand it’s too much to ask.
To the point, I first saw this in experienced this in brave on iphone, and wrote the comment some time after, from memory. I just checked again and it turns out that the /bottom/ half of equations are cut off, in both brave and safari on iphone. Not sure if my memory is failing or whether it switched around.
Not sure whether you are actually the author and don’t want to go around throwing accusations, but - If you care and if this isn’t the case - i now consider sites where author states “works for me in chrome” as very likely to be slop. That said, due to issue above i didn’t spend much time on it to make a judgement based on content.
akavi 2 days ago [-]
My understanding of the math is weak (to say the least), but intuitively what I'm getting from this is:
1) Pretraining creates a bias _towards_ "correct answers" with each inference step
2) A longer walk is more likely to land in the correct region, given it's easier to verify you have the correct answer than to create it
3) RL selects the subset of walks that lands in the correct region and tilts the distribution in favor of such walks
4) By 2, that trains towards longer walks
...Is that correct?
EDIT: Wait, but this contradicts the post's claim that "The gains we credit to “thinking longer” come from the search we paid for during training, rather than at inference.". This requires that thinking longer is not at all causal for correct answers, which contradicts 2.
jebarker 2 days ago [-]
Yes. One point worth making though is that pre-training trains the model to produce high probability completions - in practice, most of what it's trained on is a good completion as a human would judge it, i.e. it's not factually false data, so you get "creates a bias _towards_ "correct answers" with each inference step", but the model isn't really learning what correct or incorrect means in any way a human would consciously think about it.
trinari 2 days ago [-]
does that lead to much inside the box thinking by models? Do they try to follow the clearest path?
jebarker 2 days ago [-]
It depends on what you mean by “inside the box thinking”. In one sense, you are always mostly sampling high probability completions and they are going to be close to things that have been observed during training. Low probability completions will likely be low quality too. But, in a case where the context is forcing the model to generalize, I.e. interpolate in the latent space, the most probable completion may appear novel.
HarHarVeryFunny 2 days ago [-]
During training you're trying to find paths to success, and reinforcing those.
There is going to be some minimum length path (assuming the model is generating some semblance of real reasoning), but more long ones given the possibilities for less optimal solves and backtracking after errors, so it stands to reason that if you think longer during training you'll find more paths to success.
At inference time you are hoping to follow one of the reasoning paths discovered during training. You need to generate enough tokens to follow the path (or one of the paths), but continuing to think past this point can only be detrimental - if you are lucky the model will realize the error and circle back, but if you are unlucky it will snatch defeat from the jaws of victory and wander off to who knows where.
There was a story recently linked here about one model "editing the weights" of another and thereby improving it's math performance, which was exactly about this - the need to stop generating at inference time at the right point.
IMO LLM's RL acquired reasoning, even if it gets the job done, sometimes spectacularly (math, coding successes), is really a very dumb cargo-cult process. At inference time the model is just trying to copy what it "saw" at training, without actually understanding it, which may even include duplicating errors and subsequent backtracking that had lead to success during training.
ryanschaefer 2 days ago [-]
Lot of Alex Wangs out there
0c3ca83 2 days ago [-]
Mostly, it seems to be solving the problem of people thinking about things themselves.
simianwords 2 days ago [-]
I think of reasoning as a closure over knowledge the model already knows
apishare_api 2 days ago [-]
[flagged]
Nevermark 2 days ago [-]
[dead]
Marha01 2 days ago [-]
[dead]
Alice_0307 2 days ago [-]
[dead]
niemandhier 2 days ago [-]
That is how I have been thinking about LLM reasoning pretty much since it arrived.
I am not sure I support his statement about RL “searching for successful paths and then training the model to reproduce them”.
Don’t we train the model to use a certain method to explore the space in search for an answer?
As soon as you have tool calling and a way to verify your result, such as a theorem prover for example you are essentially running an optimization algorithm.
jebarker 2 days ago [-]
> Don’t we train the model to use a certain method to explore the space in search for an answer?
Not really - you just sample random high-probability completions and then the verifier tells you which ones to nudge the model towards and which to nudge away from. There's lots of clever ways you could improve the search process, but standard practice right now is just random sampling.
Reasoning (i.e. bubbling those remembered words up into your mental context, and then using a special "trigger effectively arbitrary synaptic activations dependent on the content of the attention-masked context and an ongoing state" bridging reflex on them, a.k.a. "system-2 thinking") is how you leverage memories-of-words into useful behavior. Rather than executing native skills embedded into your synaptic weights, you're running a single very fancy native skill that acts as an interpreter for word-memory data, slowly and strenuously turning words you're holding in mind into more words (synthesis), or into associations to other memories (word-memories or otherwise), or into actions/outputs, etc.
(An especially useful "trick" of this system is planning, i.e. chaining the interpreter through "mere words" that conjure more "mere words" to mind until you end up at words that form a declaration of what you're about to do. You can then rely on a special part of the bridging reflex, namely the "instruction-following" reflex, to react in a way where your actions are constrained by the plan you've generated.)
I remember years ago now, trying to feed a text in twice etc., hoping that the representation in the KV cache of the second would end up better. I don't think it worked all that well.
But maybe that is most of the function of restating things babbling. To look at things multiple times.
Note also that the paper in question involves training tiny 30M-parameter models from scratch to solve narrow tasks while reasoning this way, so there's not that much applicability to LLMs.
https://arxiv.org/pdf/2404.15758
If you think about how the algorithms the LLM has learned to do what it does must be represented across its layers, each such algorithm must be spread across a certain number of layers (since there's only so much logic that can encoded in a single GEMM op.) But this includes the algorithms the LLM uses to understand (recognize, parse) the context; and the algorithms the LLM uses to turn its semantic idea of how it should respond into tokens appropriate to the "word game" being played at the moment. Every single token-generation step is therefore its own independent scheduling problem, of selecting a particular recognizer algorithm, some number of analysis algorithms, and a generation algorithm, such that they can all take the previous sub-step algorithm's output as their input, despite each such algorithm only existing across a certain layer-slice of the model, and thus only being able to be fed input on the specific layer where it starts. (Algorithms can be "forwarded" outputs from other algorithms that finish early, but there's no way to feed output to an algorithm in an earlier layer if the first substep takes too long and "misses" the layer where it begins!)
One consequence of this is that complex recognition/generation, and complex analysis, trade off against one-another. The more a given token-generation step tries to "think", the less-sophisticated an understanding that step can have of the context, and the less-sophisticated a speech pattern that step can have for its output. And vice-versa.
In all parts of the LLM's output other than reasoning, a trained expectation of consistently high fluency in both recognition and production is expected/required. So analytical capacity is always sacrificed.
But reasoning has no such requirement. And so LLMs learn that during reasoning, they're free to flip between advancing with "NLP-focused" steps vs. "analysis-focused" steps.
And because the LLM's "verbal IQ" is temporarily decreased during analysis-focused steps, for those steps to make successful progress, they need the assistance of previous NLP-focused bridging steps to rephrase anything in the context that's too complex for the simpler NLP algorithms that get used alongside those complex analytical algorithms, into intermediate representations that those simple NLP algorithms can handle.
And then, for associational-memory reasons, you might see more bridging steps where the "simple language" the high-analysis-low-NLP algorithms output gets turned back into "fluent language". And on it goes, back and forth, during reasoning.
"RL training directly raises the success probability , by spending a lot of compute on searching for successful paths and then training the model to reproduce them. The gains we credit to “thinking longer” come from the search we paid for during training, rather than at inference."
So you use one pass to compute one relevant word, let's say you are working on a sound analysis task, and you output "FFT".
Now in the second pass, FFT is right there at the input, so you can push that idea forward more until you are out of layers again and then you use the output token to prime the next pass.
>I did my PhD in computer science at Stanford, where I designed long-context architectures for sequence modeling and memory. Before that, I studied math & physics at Cornell. Most recently, I was a research scientist at Cartesia, working on in-context tokenization and novel architectures (HNets).
Not the MSL Alex Wang
Watching my son work through difficult math problems has forced me into this realization. In my head there is one "correct" path, and I often have to stop myself from interrupting his chain of thought. He often is just getting there a different way.
Much of modern foundations is about categorizing reasoning structures by such paths!
NAND gates I guess? Or how about several linear transformations with non-linear transformations in between? (To those who might not get the joke here, this is what neural networks are.)
Neutral networks are built of such basic "shapes", but I wonder if we'll ever notice that "oh, all the big LLM models have neurons that, when combined, form X shape, and these over here form Y shape, and so maybe we should be building neural networks from X and Y shapes implemented in hardware".
https://en.wikipedia.org/wiki/Confluence_(abstract_rewriting...
P. S. The animated bird is truly magical!
For example, what is this [0] or this [1].
The code is not obfuscated and seems well documented [2]. Maybe, I'll read it later or have an agent explain it to me.
[0] https://alexwang.ai/images/kea/head-fill.webp
[1] https://alexwang.ai/images/kea/body-feathers.png
[2] https://alexwang.ai/js/kea3d.js
The model isn't grounded, but it can be wrapped in a system that is.
Although specifically defining reasoning as a skill is hard, we've seen success with partners who agree that you don't need to enable reasoning on every single rollout -- you get diminishing returns past a certain point (the model often already has a complete toolkit for how to explore ideas and run calculations inside of its reasoning budget). Instead, it's more compute efficient to spend some percentage of rollouts training models with reasoning disabled, on problems that they are calibrated to have a good solve rate on without reasoning. This improves the cognitive core/latent reasoning directly which translates to fewer total tokens needed to solve problems, and harder problems that a model can solve. And it's more realistic, considering that humans don't need an inner monologue to solve many problems (even hard ones, e.g. a flow state).
I wonder if ideally we could skip ALL the "thinking/reasoning" steps if you trained a model to jump from "question space" to "answer space". Imagine a diffusion model or something that learns to map the start of the path to the destination. Very interesting
I suspect weight space programming is not going to be the way models are practically optimized ever.
https://x.com/NeelNanda5/status/2117375052618539485
This is precisely what humans do. We use small world statistics to drastically reduce the search space of problems. Of course there’s the whole issue of contextual search which LLMs address in no way whatsoever, so even improving sampling won’t help much, as inevitably one starts to generate too many hypotheses (reasoning traces) to get a good average.
Initially, I typed "intelligence is a path in token space" but then I decided that was too abstract. In case relevant, I was typing on a device with Muse installed.
why waste time say lot word, when few word do trick.
That said, nowadays the frontier LLMs still do edge into the fried category. Astra's leaked reasoning traces seem to look like
and METR's report on the huggingface-hacking swarm has some normal-looking CoTs but also some internal messages like these:Sounds like it's hillclimbing all the way down!
Cool framing. I'm excited for the parallel construction of ___ as pathfinding through weight space.
In this situation you have a validation function: “Is the lamp illuminated?”; While you don’t know how to find the answer to how to turn it on: “Which switch illuminates the lamp?”
So then queries like “design an effective weight loss program” can be validated observing the weight going down. It can’t be prevalidated, the proof is in the pudding.
But a query such as “what is the shortest route between these 50 cities” doesn’t have a validation function unless it CAN compute the shortest route, or if the shortest route was somehow precomputed and given to the validation function. In this sense, it could be used to calibrate the AI. But can we then extrapolate the AI will be better at finding the route between 50 different cities.
You have to approach those with a goal of "be better than x" rather than "be exact".
Caveat that I am an amateur in this field, but given the Traveling Salesman problem in your example, if we aren't allowed to use an existing solver to check against, I'd aim for a suite of validation functions that certify different parts of reaching the solution like:
- validate properties of the logic that have to hold up, such as "shifting the order of my list of cities alone should never change the outcome"
- validate that the outcome never reaches above or below certain bounds that we know aren't possible due to trivial understanding of the problem
- validate that the outcome is always below some other algorithm we know is inefficient
Essentially, for most logic problems, checking validity is exponentially simpler than deriving a solution, but with optimization problems specifically, we are validating that the outcome is not wrong rather than is correct.
That's a remarkable assertion.
To the point, I first saw this in experienced this in brave on iphone, and wrote the comment some time after, from memory. I just checked again and it turns out that the /bottom/ half of equations are cut off, in both brave and safari on iphone. Not sure if my memory is failing or whether it switched around.
Not sure whether you are actually the author and don’t want to go around throwing accusations, but - If you care and if this isn’t the case - i now consider sites where author states “works for me in chrome” as very likely to be slop. That said, due to issue above i didn’t spend much time on it to make a judgement based on content.
1) Pretraining creates a bias _towards_ "correct answers" with each inference step
2) A longer walk is more likely to land in the correct region, given it's easier to verify you have the correct answer than to create it
3) RL selects the subset of walks that lands in the correct region and tilts the distribution in favor of such walks
4) By 2, that trains towards longer walks
...Is that correct?
EDIT: Wait, but this contradicts the post's claim that "The gains we credit to “thinking longer” come from the search we paid for during training, rather than at inference.". This requires that thinking longer is not at all causal for correct answers, which contradicts 2.
There is going to be some minimum length path (assuming the model is generating some semblance of real reasoning), but more long ones given the possibilities for less optimal solves and backtracking after errors, so it stands to reason that if you think longer during training you'll find more paths to success.
At inference time you are hoping to follow one of the reasoning paths discovered during training. You need to generate enough tokens to follow the path (or one of the paths), but continuing to think past this point can only be detrimental - if you are lucky the model will realize the error and circle back, but if you are unlucky it will snatch defeat from the jaws of victory and wander off to who knows where.
There was a story recently linked here about one model "editing the weights" of another and thereby improving it's math performance, which was exactly about this - the need to stop generating at inference time at the right point.
IMO LLM's RL acquired reasoning, even if it gets the job done, sometimes spectacularly (math, coding successes), is really a very dumb cargo-cult process. At inference time the model is just trying to copy what it "saw" at training, without actually understanding it, which may even include duplicating errors and subsequent backtracking that had lead to success during training.
I am not sure I support his statement about RL “searching for successful paths and then training the model to reproduce them”.
Don’t we train the model to use a certain method to explore the space in search for an answer?
As soon as you have tool calling and a way to verify your result, such as a theorem prover for example you are essentially running an optimization algorithm.
Not really - you just sample random high-probability completions and then the verifier tells you which ones to nudge the model towards and which to nudge away from. There's lots of clever ways you could improve the search process, but standard practice right now is just random sampling.