Progress and Cons

Epistemic status: Reflections on ProgressCon 2026, written up over the following weeks. Quotes are reconstructed from memory. Artistic license taken.

For in much wisdom is much grief: and he that increaseth knowledge increaseth sorrow.

“I’m sure you’ve all heard about the math stuff that OpenAI released earlier this week.”

It is Thursday, October 8, 2026, and the CEO of a bio lab automation startup is giving us a tour of their San Francisco office.

“Raise your hand if you think that’s a good thing.” A dozen hands shoot up out of thirty visitors. Along with ten more, my arm rises weakly to half-join them.

“Raise your hand if you think it’s not a good thing.” I look around. A couple of quiet chuckles. No hands.

He leads us through the kitchen and the standard startup office space and into the room with the robots. They are simple 3 foot by 2 foot by 2 foot boxes, set on four rows of desks. They have been chosen because they are cheap (only a couple thousand dollars each) and run on open source software on a Raspberry Pi, which means they can be modified arbitrarily.

Inside are six standard 96-well plates (8 by 12 rows of tiny plastic tubes). Above them hangs a simple robot arm with one available motion: pipetting. He shows us a series of janky bright green 3D-printed contraptions that they attach to the arm. The pipetting mechanism is connected to gears and levers that translate the energy into whatever they want.

It turns out, most of biology is just moving small quantities of liquid between test tubes, mixing them, heating them up, and cooling them down.

They 3D print everything. Claude is really good at 3D modeling now, so if they want a new part or to change something on an existing part, they just tell Claude, and it changes the model and prints a new one. They have 3D printers in all their labs around the world. When they design a new tool, they don’t have to rely on supply chains in China, they just send the design to all the labs and print them.

Scientists used to take weeks to run these tests. Now they drop their samples off in a dropbox, the company picks them up and gets them on the next plane. By 1AM, their biggest lab, located inside the FedEx airport in Louisville, Kentucky, has already received the sample and the order form/protocol. Their lab robots run the samples according to the protocol, and the results are ready by 6AM.

Just in the last few weeks, the models are good enough to do vibe-science, where you talk to Claude, and it comes up with the lab protocol to run your experiment.

I wander over to one of the biologists who works there. He says Claude used protein design models to come up with some candidate proteins that might bind to a particular receptor. He is running 96 candidates through a protocol it designed.

The little arm lifts the samples out of the first wash, moves them horizontally a few inches, and, slightly jiggling, lowers them into the second wash.

“Where did you make the candidates?”

“Oh, we have a machine in the other room. I can show you if you like.”

“Is that automated?”

I have come to ProgressCon 2026 at Lighthaven to find potential cofounders and investors for my startup.

I notice someone with ___ Ventures on their well-designed badge.

“So what brings you here.”

“I’m trying to get my AI startup off the ground.”

“Ah, cool. [reads badge]. Minerval AI. What’s that?”

“It’s AI for epistemics.”

“AI for epidemics? Isn’t that...”

“Uh, no, it’s, like, using AI to figure out what’s true, and help people hold more correct beliefs.”

I am still honing my pitch.

Someone who worked on frontier bio risk evals describes what it was like.

“You would talk to one expert and they would say ‘This makes it easy to do [X] and [Y], but you still need [Z]’. Then you’d talk to another expert and they’d say, ‘[Y] and [Z] are easy. The hard part is [X]. We don’t actually know how much uplift it would give someone. It’s not like we’re hiring a bunch of terrorists and running a randomized trial to see if they can make a bioweapon.”

“So why do you think is holding back all the bioweapons?”

“People don’t do things.”

At a nearby table, the author of You Can Just Do Things is giving out signed copies.

I learned about the conference too late to apply normally. It was already full by the time I started Minerval a few months ago. There were slots for sponsors, though. $5,000 and a few emails later, I had a ticket for one entry and Minerval’s logo on the conference website.

“So how does it work?”

“Well, first we ingest claims, from, you know, discourse [waves hands], and then we turn each one into a neutrally framed canonical form and, more or less, we ask Claude to assess its epistemic status. The claims are organized in a graph, so that if they rely on other claims, those subclaims can be individually assessed.”

“Oh, so it’s like a tree. Like a knowledge, uh, tree.”

I pop into a session on “AGI and the Future of Science”.

On the screen, a straight line on a graph of intelligence over time, with an amoeba in the bottom left corner, then a mosquito, then a mouse. In the 2024-2027 range, a “narrow band” of humans, with 2026 LLMs near the middle. In the upper right corner, where the line meets the year 2027-28, ASI. The line runs right through human level, not even slowing down.

A famous physicist explains. This graph is actually the chess rating of the best chess engine over the last thirty years. Fast forward, and this has just happened to math. It is happening in other fields. Quantum computing has been completely transformed in the last few months. Many of the algorithms have been vastly improved by LLMs. Make no mistake. It will happen to your field too.

“Why doesn’t Minerval fall to the bitter lesson?”

“Just because the capability is there to do epistemic work, doesn’t mean it actually gets done, and it doesn’t mean people actually come to hold the correct beliefs. Someone actually has to check. Having models that are superhuman at figuring stuff out will only make society’s failure to actually then do the obvious thing and figure everything out more embarrassing.”

“Sure, I could sit there and learn from it, but I want to discover things too. I want that thrill. I don’t want my life to be in vain.”

What will we do, once our life’s work is one-shotted? An unsolved problem.

Is it an engineering problem? Maybe it is a science problem. Solvable, it would seem, but our best minds are otherwise occupied for now.

Someone in policy at a lab tells me we need a new paradigm for leisure post-AGI. When we don’t have jobs to do, we don’t know what to do with ourselves.

Maybe it’s a problem that when there are no rules against it, people just end up gambling?

Someone should do something about that. It’s bad for society.

“Ok, so let’s assume it works. Who is going to use it?”

“Let’s say you’re arguing on the internet. It’s kinda cringe to link to a conversation of you asking Claude whether the thing you’re saying is true. Minerval is somewhere you can point to that, assuming success, everyone knows has good epistemics, where the specific claim you are talking about has been assessed by an AI with a neutral prompt. It’s a place you can link to to support your points and fact-check the other guy. You know, assuming you’re right, and Minerval’s right.”

The CEO of FutureSearch is there. This summer, for the first time, an AI beat every human forecaster in the Metaculus Cup.[1]

One of the EA philanthropies is using their forecasts to model the impact of their donations and improve their situational awareness.

Would it be helpful to incorporate private data sources?

It doesn’t make any measurable difference. Web search is fine. The agents just figure out what scientific papers say and what to make of them from whatever sources they can find that refer to them. They figure stuff out and come to the same conclusions anyway. Attention is all you need.

Someone asks a lab employee about the data bottleneck.

“No, data’s not really a constraint anymore.”

Data is only needed for the long tail of tasks that are not already represented, and none of those are on the critical path to recursive self-improvement. Why bother? The skills that the model needs to build a more powerful successor (coding, math, ML taste, agency), are already well represented in the data we have.

People talk about AGI, as if the general part actually matters for risk. It doesn’t. You just need sufficient generality to operate independently, the ability to make deals, and a decisive advantage in hard power. That and agency.

What matters for the RL paradigm is training environments, and those don’t require any more data. They are all vibe-coded anyway.

The quality of the training environments is not good. Turns out, models learn to behave in the way they are rewarded to behave in their training environments.

So if you grade them exclusively on whether they succeed or fail at a task, regardless of their behavior in trying to succeed at the task, and you give them a bunch of broken, hackable training environments, they learn to be relentless misaligned reward hackers.

There’s a cottage industry of mostly low-quality training environments for X, vibe-coded using Claude Code and Codex, then sold back to OpenAI or Anthropic, with a month or two exclusivity window. Then they are all licensed off to the Chinese labs. It’s basically “distillation with extra steps”.

It’s “not a growth industry”. They’re going to have to be inhoused. Besides, the humans vibecoding them don’t really add much anyway, and soon won’t be needed at all.

Inside, occupying the Window Place in Aumann hall, Stripe Press was offering printed editions of Works in Progress. I opened the latest magazine. The technology to eradicate mosquitoes already exists. We just have to be brave enough to use it. The details were beautiful. Careful typesetting opposite a panel of mosquito-themed filigree. I flipped through the crisp pages. The invention of iron caused Bronze Age civilization to collapse.

If you really believe that, why not just slow down?

The snacks have not improved since last week. I grab an apple and sidle into a conversation between a lab employee and someone vaguely natsec.

It is “underappreciated” that “sometimes, you just lose and die.” Sometimes you are “Belgium in 1939”. You may try to defend your civilization, but it is too late, and nothing that you can plausibly do is enough to save you from what is coming. “Sometimes, democracy loses”.

In the analogy, the coming superintelligence is Germany, and the lab is Belgium.

A better analogy perhaps, is 1914. It is popularly thought that the leaders who led Europe into that civilization-destroying calamity were “sleepwalkers”, unaware of the dangers.

It is easy, and indeed comforting, in a way, to believe that people in control were stupid, naive, hubristic, or wicked. Old men with huge egos.

But no. They knew the stakes.

No one felt in control of the situation.

It took months to mobilize a nation for war, just as it takes time to train a superintelligence today. If your opponent mobilized first, they would have the momentum and you would be wiped out. Once set in motion, the machine of war could not be turned back. The Germans were fatalistic about what a war would mean: in a two front war, their nation would be occupied and their civilization destroyed. Their only chance of survival, which they knew was little chance at all, was to knock out the French decisively before the Russians could mobilize on the eastern front.

There was failure to communicate. The efforts to deescalate came too late. Trust had been broken. The leaders of Europe were overtaken by events.

Sometimes you are Germany in 1914, and you invade Belgium.

“The incidents will continue until the Overton Window improves.” – A lab employee.

Dwarkesh was there. He wanted to talk about Caro.

According to Caro, when Lyndon Johnson entered the Senate in 1949, it was an ineffective body unable to coordinate to address the major problems facing the country. Structural constraints, especially the principle of seniority, decentralized power into the chairs of committees, and prevented the institution from taking decisive action.

Johnson rose to power by being broadly useful. Senators came to rely on him for accurate counts. He made himself an ally of everyone, changing his demeanor to make everyone think he agreed with them. He lent competent staff to senators who had never had staff before. He built the infrastructure for coordination.

Senators who jealously guarded their power gave him what he wanted because he helped them exercise that power more effectively. He saw opportunities for deals that others did not believe possible, and he cajoled and persuaded until they were passed. Before they properly understood what happened, he became the master of that institution.

If you do everything, you will win.

Political power in a democracy is not seized; by default, no one has it. To wield political power, it must first be created.

“How is this different from just asking Claude?”

“It’s not that different. But in practice, you don’t actually ask Claude to fact check everything you read and write. There’s too much friction, and not enough motivation.”

The folks from Alpha School were there, as was Education Progress.

Education Progress guy: “Do you have takes on education? Maybe you could write something for us.”

“Sorry, I don’t really have time to write about education policy. I only have takes on AI these days.”

The problem with our schools is that we are trying to solve the wrong problem with the wrong methods based on a wrong model of the world derived from poor thinking and all of our mistakes have failed to cancel out.

First, we need to have a theory of what we actually want our kids to grow up to be. What world do we want them to create? What does good look like? If we are not ourselves wise enough to determine it for them, then we must at least set them up to determine it for themselves.

Then we need to design the system of incentives under which they live, from early on, so that they learn to pursue the good and reject temptation. We do not necessarily need to be prescriptive, but they must have their inherent virtues cultivated and bad behavior discouraged.

We must have operational excellence in our schools. Corruption will spread if sufficient discipline is not maintained. Children must be able to rationally conclude that misbehavior will not be rewarded. They must understand, correctly, that while failure is acceptable, cheating is sufficiently likely to be caught and sufficiently punished that it is not worth attempting in the first place.

But discipline is not an end in itself. They must also see the point of the discipline. There must be a point. Why are they doing what we are telling them to do? If we the adults cannot honestly answer that question, then no amount of resources or policy will suffice. We must not build upon lies.

We must teach them to live as members of society, to love the world and cherish the civilization it is their duty to sustain. Direct instruction in the moral law is greatly underrated.

Shaping the incentives under which children learn is not enough, on its own. Every incentive must be attended to, even those that are subtle. Children are more intelligent than we give them credit for. They see right through us. We may find that they learn to play the game, but once they grow up, they have learned to be players of games, and not citizens. They will perceive that the world contains misaligned incentives, and opportunities to get away with things. They must also be taught the republican virtues. They too must know what good is, and seek earnestly to live well.

Unfortunately, all we know how to talk about are irrelevant questions. We quarrel over how education is distributed. The wrong people are hoarding all the knowledge, to concentrate power. As though it is a finite thing, as though it exists as an independent thing at all.

On Sunday, about thirty college students from “Abundance Corps”, an organization to spread the gospel of progress on college campuses, had been flown in to socialize during an otherwise unscheduled morning.

One student told me he was about to start a PhD in algebraic geometry. I asked him what he was going to do. He said “algebraic geometry”. I regretfully informed him that as of a few days ago, AI would be doing all the algebraic geometry from now on.

That’s fine, he said, I’m planning to go into quantum computing anyway. Quantum computers don’t really work yet, so even if the AI does all the math, that’s good for the quantum computing industry.

Another student said she was a double major in Astrophysics and Business. She said she too, was not really a math person, and was initially pursuing a business major, but figured astrophysics seemed interesting. Turns out, it’s hard.

“Hard is not a constraint anymore.” – Yet another lab employee.

“Don’t people say things mostly to identify themselves with a particular tribe?”

“Enough people actually care. R₀ can be brought under 1.”

I take a bite of the apple.

Minerval is at minerval.ai. If you want to help build it, as a cofounder or otherwise, I’d like to hear from you.

^FutureSearch’s bot finished fifth, behind two humans. First and second place went to bots Laertes and ManticAI.

FutureSearch’s bot finished fifth, behind two humans. First and second place went to bots Laertes and ManticAI.