Some interesting internal documents from Anthropic litigation

Anthropic released some documents earlier this year due to the book copyright litigation. I think a few of those documents might be of interest to some people here and haven't gotten much attention. In this post I link to some of those documents, and quote the bits I find interesting.[1] (This expands on my quick take from yesterday.)

These aren't Anthropic's official opinions about anything, and you should probably treat opinions in these documents as "what the author thought at the time", not as a corporate stance.

Long Term Goal Our top level goal is to work towards the creation of safe, beneficial AGI.As a proof of concept, we are working to develop a general AI assistant that can interact conversationally with a user to create, edit, and explain software. The most ambitious and successful version of this agent could answer general questions, write functions, identify bugs, design an API, discuss style and best practices, etc -- it would display all the capabilities of a software engineer colleague. We don't expect to reach this goal right away, but it's the direction in which we're striving. Our research projects can all be viewed as attempts to understand and construct aspects of this proto-AGI. We currently think this will require large generative models with long memories, fine tuned via RL using both automated rewards and human feedback.

Our top level goal is to work towards the creation of safe, beneficial AGI.

As a proof of concept, we are working to develop a general AI assistant that can interact conversationally with a user to create, edit, and explain software. The most ambitious and successful version of this agent could answer general questions, write functions, identify bugs, design an API, discuss style and best practices, etc -- it would display all the capabilities of a software engineer colleague. We don't expect to reach this goal right away, but it's the direction in which we're striving.

Our research projects can all be viewed as attempts to understand and construct aspects of this proto-AGI. We currently think this will require large generative models with long memories, fine tuned via RL using both automated rewards and human feedback.

I'm a bit nervous if we end up doing alignment research on natural language and capabilities research on code. It feels like there should be a pretty tight connection between the two at all times. If we're not improving capabilities in the domains we're doing alignment research on, it puts limits on how much we can test alignment and how useful the alignment work can be. That could have knock-on effects like alignment becoming an island because we don't integrate it early on.A possible principle: "by default, capabilities and alignment research should be in proportion across domains unless we can give a good argument for why they should come apart". Is this a good/bad principle?

I'm a bit nervous if we end up doing alignment research on natural language and capabilities research on code. It feels like there should be a pretty tight connection between the two at all times. If we're not improving capabilities in the domains we're doing alignment research on, it puts limits on how much we can test alignment and how useful the alignment work can be. That could have knock-on effects like alignment becoming an island because we don't integrate it early on.

A possible principle: "by default, capabilities and alignment research should be in proportion across domains unless we can give a good argument for why they should come apart". Is this a good/bad principle?

It describes hypotheses (the author doesn't "believe any of them with very high probability") about what training data is useful. Among other things, it suggests training on Chinese text to slow down Chinese AI labs:

The China Competition Hypothesis – China seems like it might be the least ethical / safe AI actor. To the extent we can slow them down by absorbing market share they could grab (and thereby use to accelerate production of large models), we can tip the geopolitical balance. As a result, we should prioritize having good linguistic performance on languages for markets Chinese companies might be inclined to go after – Mandarin (although Chinese government intervention will likely push us out here), but perhaps also neighbors or countries deeply involved in the Belt and Road strategy.

The China Competition Hypothesis – China seems like it might be the least ethical / safe AI actor. To the extent we can slow them down by absorbing market share they could grab (and thereby use to accelerate production of large models), we can tip the geopolitical balance. As a result, we should prioritize having good linguistic performance on languages for markets Chinese companies might be inclined to go after – Mandarin (although Chinese government intervention will likely push us out here), but perhaps also neighbors or countries deeply involved in the Belt and Road strategy.

Some related comments from @ben@anthropic.com --Chinese may still be valuable for all the Chinese diaspora/expats+1 transfer knowledge from Chinese to EnglishUnlikely China will enforce copyright on US?US gov’t may be interested in Chinese language ability for national security reasons, if we end up engaging there

Some related comments from @ben@anthropic.com --

Unlikely China will enforce copyright on US?

All Company meeting!

This is a June 2021 slideshow[3] that discusses hiring, office space, the efficiency of training trillion parameter models, RL, sources of training data, and interpretability. It discusses Anthropic's first alignment failure in the "RL Update" section:

Now running simplest-possible tool-use envCodewhack:Agent gets some words from a Github repoAgent needs to pick a search query that gets exactly one hit.Seemed to work! But,[next slide]Actually our first alignment failure.Examples:" script that looks at instargram" => "^"" = img.shape[:2]\n\n " => "^""docker.utils.mkbuildcontext(" => "^"What’s happening?“^” is the regex start of sequence token.There’s only one start of sequence!I didn’t escape the query.I did not end the universe. This time.

Codewhack:Agent gets some words from a Github repoAgent needs to pick a search query that gets exactly one hit.

Agent needs to pick a search query that gets exactly one hit.

Seemed to work! But,

Actually our first alignment failure.

Examples:" script that looks at instargram" => "^"" = img.shape[:2]\n\n " => "^""docker.utils.mkbuildcontext(" => "^"

" = img.shape[:2]\n\n " => "^"

"docker.utils.mkbuildcontext(" => "^"

What’s happening?“^” is the regex start of sequence token.There’s only one start of sequence!I didn’t escape the query.I did not end the universe. This time.

“^” is the regex start of sequence token.

There’s only one start of sequence!

I didn’t escape the query.

I did not end the universe. This time.

It discusses potential use-cases for Claude 18 months in the future, as of June 2023. At the bottom (in the blue text[4]), various Anthropic employees provide their own takes.

We'll want to find clear stepping stones between where we are now and the long term vision, where we get technological proof points and market feedback along the way. Companies like Magic Leap that aim straight for their super-ambitious final destination end up disconnected from reality and burn an obscene amount of money, ultimately falling behind competitors who were able to absorb market feedback and build organizational muscle. SpaceX is probably a great example of doing this right. The founders (yes, there's more than one!) had the borderline ridiculous goal of achieving human settlement on Mars, but they broke it down into a series of steps that achieved proof points and profit along the way. First they got good at building small, profitable, low cost rockets, then they got good at building medium sized, profitable, low cost rockets, then they figured out how to make rockets reusable to further lower the cost, then they became the world's largest provider of orbital launch capability with their superior product, then they built the world's largest rocket and almost got it into space on the first try… and if they can generate enough demand for heavy lift launch capability, then they can mass produce these giant rockets and start sending them to Mars. The long term goal kept the team inspired and focused, and the steps along the way gave the team the skills and the revenue to succeed.

We'll want to find clear stepping stones between where we are now and the long term vision, where we get technological proof points and market feedback along the way. Companies like Magic Leap that aim straight for their super-ambitious final destination end up disconnected from reality and burn an obscene amount of money, ultimately falling behind competitors who were able to absorb market feedback and build organizational muscle. SpaceX is probably a great example of doing this right. The founders (yes, there's more than one!) had the borderline ridiculous goal of achieving human settlement on Mars, but they broke it down into a series of steps that achieved proof points and profit along the way. First they got good at building small, profitable, low cost rockets, then they got good at building medium sized, profitable, low cost rockets, then they figured out how to make rockets reusable to further lower the cost, then they became the world's largest provider of orbital launch capability with their superior product, then they built the world's largest rocket and almost got it into space on the first try… and if they can generate enough demand for heavy lift launch capability, then they can mass produce these giant rockets and start sending them to Mars. The long term goal kept the team inspired and focused, and the steps along the way gave the team the skills and the revenue to succeed.

It discusses "three major visions for TAI": "Claude the Virtual Employee" (also called "Claude the Robot"), "Claude the personal assistant" (also "Claude the Cyborg"), and "Claude as intelligent org infrastructure". It says working on Claude the Virtual Employee and Claude as org infrastructure are useful to "reduce TAI risk", and Claude the personal assistant is "Sort of" useful for reducing TAI risk.

It is a slideshow that discusses the RSP and why Anthropic scales capabilities.

Interpretability & Alignment, or BankruptcySometimes folks ask if product is just here to "fund alignment & interpretability research"...But you can also argue for the opposite:In worlds where alignment is extremely difficult or impossible, these teams need to save the world by demonstrating this fact persuasivelyIn worlds where alignment is easy/manageable, these teams need to save the company by making responsible scaling well-defined and achievableThis is a bit of an exaggeration, but not too much. We're all working together to accomplish a common mission.

Sometimes folks ask if product is just here to "fund alignment & interpretability research"...

This is a bit of an exaggeration, but not too much. We're all working together to accomplish a common mission.

I had Claude make a thingie to read Slack messages from the filings.

These are mostly uninteresting.

Teal = Claude 3 Opus, Dove/Firefly/Wombat = Claude 3.5/3.6/3.7 Sonnet according to Anthropic's table.[5]

Keep in mind that these documents weren't intended to be public, they aren't Anthropic's official position on anything, and were written in the past. I don't want anyone to get the impression that any of these documents represent commitments from Anthropic to do anything; instead you should think of these as information about what some people at Anthropic were thinking about in the past.

Also keep in mind that this isn't a random sample of documents. The documents Anthropic releases due to litigation are strongly tilted towards being about training data because Anthropic gets sued over their training data a lot, and doesn't get a lot of lawsuits related to things like alignment science. You should also consider that Anthropic might try harder to redact information that makes them look bad. Most of the documents I looked at are things like Data Partnerships <> Finance; Monthly Syncs which is heavily redacted and much less interesting.[6]

Thanks to Anthropic and the authors of these documents for creating them; I enjoyed reading them! Also thanks to the Free Law Project for running RECAP/CourtListener, which is where I got these documents from. (Perhaps I should get a PACER account so I can access documents that aren't already in RECAP.)

I'll probably get back to focusing on technical alignment research now, but reading obscure documents about Anthropic has been a fun hobby!

^The PDFs are all images of the documents, so I had Claude Opus 5.5 transcribe the text for me.

The PDFs are all images of the documents, so I had Claude Opus 5.5 transcribe the text for me.

^I fixed the distortion from the original PDF to make it easier to read. Well actually Claude fixed it but I at least prompted it.

I fixed the distortion from the original PDF to make it easier to read. Well actually Claude fixed it but I at least prompted it.

^Fixed from the original to be undistorted and have the slides take up the full page.

Fixed from the original to be undistorted and have the slides take up the full page.

^I think the text is blue because it's a Google Doc, and everyone else added their takes as suggested edits. Not sure though.

I think the text is blue because it's a Google Doc, and everyone else added their takes as suggested edits. Not sure though.

^Confusingly people in the Slack threads discuss things like "add it to Teal" and a "Teal datamix update", after Claude 3 Opus was publicly released. Maybe this means Anthropic continued training on Claude 3 Opus was released? Maybe "teal = Claude 3 Opus" is an over-simplification? Anthropic's codename table is pretty weird - e.g. "Santa" is a codename for all Claude models from Claude 1 to Claude 2.1, except Claude Instant 1 and Claude 2.At first I thought it was something like the base model, but that would imply that Claude 2.1 used an older base model than Claude 2? Also Claude 2 is given a second codename of "Project Hatch", but in 2023 the New York Times said that Project Hatch was a ChatGPT-like interface to Claude (presumably claude.ai), not an LLM?

Confusingly people in the Slack threads discuss things like "add it to Teal" and a "Teal datamix update", after Claude 3 Opus was publicly released. Maybe this means Anthropic continued training on Claude 3 Opus was released? Maybe "teal = Claude 3 Opus" is an over-simplification? Anthropic's codename table is pretty weird - e.g. "Santa" is a codename for all Claude models from Claude 1 to Claude 2.1, except Claude Instant 1 and Claude 2.

At first I thought it was something like the base model, but that would imply that Claude 2.1 used an older base model than Claude 2? Also Claude 2 is given a second codename of "Project Hatch", but in 2023 the New York Times said that Project Hatch was a ChatGPT-like interface to Claude (presumably claude.ai), not an LLM?

^Some documents are even less interesting! E.g. Copy Fairy Code is entirely redacted.

Some documents are even less interesting! E.g. Copy Fairy Code is entirely redacted.