Off-topic for aiqbench
This story was gathered from a source we follow but falls outside AI and quantum coverage, so there is no aiqbench take for it.
Chain of Thought For Humans
These past few weeks, I’ve been conducting interviews, and I often open with “make sure you articulate your thoughts out loud, so I can follow along with your thinking”. This is primarily because I’m not a mind reader, but it does also help ensure candidates avoid freezing up. It’s a very “interviewy” skill though - artificial and unlike real development. The other thing I’ve been doing is playing The Witness, a 2016 puzzle game centred around solving mazes with increasingly complicated hidden rules. The game has a unexciting presentation, no music or interesting dialog, and no story. At first I found the game fairly dull, you spend a lot of it just wrestling with possibilities and combinatorics. It can get very frustrating when you are not making progress. But, inspired by my interviews, I’ve started narrating my own thought process out loud to myself. I look like a bit of a loon, but I tell myself I’m just a streamer with an audience of one. And to my surprise, it’s working really well. Not only does it make the game more lively and energetic, but also I find I’m more effective in solving puzzles. I was initially baffled by this. Common sense says that I can’t articulate my thoughts at remotely the same speed or accuracy as thinking them. Particularly for a puzzle game where the majority of reasoning is geometric, difficult to describe in speech. We know LLMs benefit from this sort of reasoning, but that is due to an architectural limitation - they literally cannot iterate on previous thoughts; they must output tokens to get any true recursion. But then I remembered rubber duck debugging. It’s often useful in software engineering to explain your problem to someone else. But who you explain to isn’t really an important part of the process - it could be a rubber duck. The value is in forcing you to evaluate all your unstated assumptions, linearly sort your thoughts by priority and dependency, and find good abstractions. These are also useful skills for real-time problem solving. I think there are some further benefits for this particular use case: It forces you to establish named formal rules and concepts, instead of relying purely on building intuition. In other words, engaging System II, which the puzzles are explicitly designed to require. It forces you to acknowledge when you are making an arbitrary choice. Useful for backtracking or exhaustively searching a possibility space. I think it lets you use echoic memory to slightly augment your working memory. I may well have accidentally triggered the Hawthorne effect It turns out there’s good evidence that this technique has a moderate benefit, for example this meta-study (n=6000 participants). I’m definitely going to try it in future in more circumstances. Discuss
These past few weeks, I’ve been conducting interviews, and I often open with “make sure you articulate your thoughts out loud, so I can follow along with your thinking”. This is primarily because I’m not a mind reader, but it does also help ensure candidates avoid freezing up. It’s a very “interviewy” skill though - artificial and unlike real development.
The other thing I’ve been doing is playing The Witness, a 2016 puzzle game centred around solving mazes with increasingly complicated hidden rules. The game has a unexciting presentation, no music or interesting dialog, and no story. At first I found the game fairly dull, you spend a lot of it just wrestling with possibilities and combinatorics. It can get very frustrating when you are not making progress.
But, inspired by my interviews, I’ve started narrating my own thought process out loud to myself. I look like a bit of a loon, but I tell myself I’m just a streamer with an audience of one. And to my surprise, it’s working really well. Not only does it make the game more lively and energetic, but also I find I’m more effective in solving puzzles.
I was initially baffled by this. Common sense says that I can’t articulate my thoughts at remotely the same speed or accuracy as thinking them. Particularly for a puzzle game where the majority of reasoning is geometric, difficult to describe in speech. We know LLMs benefit from this sort of reasoning, but that is due to an architectural limitation - they literally cannot iterate on previous thoughts; they must output tokens to get any true recursion.
But then I remembered rubber duck debugging. It’s often useful in software engineering to explain your problem to someone else. But who you explain to isn’t really an important part of the process - it could be a rubber duck. The value is in forcing you to evaluate all your unstated assumptions, linearly sort your thoughts by priority and dependency, and find good abstractions.
These are also useful skills for real-time problem solving. I think there are some further benefits for this particular use case:
It forces you to establish named formal rules and concepts, instead of relying purely on building intuition. In other words, engaging System II, which the puzzles are explicitly designed to require.
It forces you to acknowledge when you are making an arbitrary choice. Useful for backtracking or exhaustively searching a possibility space.
I think it lets you use echoic memory to slightly augment your working memory.
It turns out there’s good evidence that this technique has a moderate benefit, for example this meta-study (n=6000 participants). I’m definitely going to try it in future in more circumstances.
Full text under the publisher's open licence · 451 words
