Distilled supply chain risks

I’ve been thinking about Anthropic’s supply chain risk designation, and I think I finally understand a sincere logic for it well enough to write out.¹ Here’s a sketch.

Let $X$ be a user’s prompt, and $Y$ be the output that satisfies them. Different LLMs have different weights governing how they map $X$ to $Y$; let’s gather these into a vector $\theta$. LLMs are then conditional probability functions indexed by their weights, $P_\theta(Y \mid X)$.² AI labs periodically update $\theta$ based on actual user-submitted $Y \mid X$, so $\theta$ contains information about actual user $Y \mid X$.

So then. Let’s say a distiller is running a whole mess of prompts and capturing their outputs—let’s call these $X’$ and $Y’$—to recover $P_\theta$, odds are good that they will eventually able to reconstruct $Y \mid X$. Vendors of $P_\theta$ that are targets of distillation attacks will thus end up leaking information about actual users’ $Y \mid X$.

This makes AI labs selling commercial AI products an interesting class of supply chain risk. The better their $P_\theta$, the more popular they will be with users; the more popular they are, the more they will be targeted for distillation by competitors, raising the expected information they’ll leak. In equilibrium we might expect distillers to target AI labs according to their market share.³ To the extent that AI labs and their products are subject to increasing returns or some other interesting non-convexity, it seems to me that there will

1. always be a most-popular lab product; and
2. that product will always be a “distilled supply chain risk”.

From this perspective, I can understand how it could be rational to rip the most popular (“best”, at least with respect to the market’s aggregate preferences) product out of sensitive (say, national security) systems—a kind of 0/infinity Pigouvian tax targeted to where it may prevent the most risk. But while this public policy solution is easy to explain in one sentence—“the product is a risk, so don’t use it”—it also seems somewhere between ineffective to counterproductive, from a public policy perspective, in the long run.

Why? Think about what it does to an equilibrium. Either:

1. Users with sensitive use cases are locked out of the best products;⁴ or
2. The product loses enough market share or reputation from being labeled a supply chain risk to no longer be the best.

If you⁵ get 1, you’ve nerfed your own capabilities in sensitive use cases. If you get 2, you’ve put yourself into a whack-a-mole situation where adversaries can force you to kill your golden geese when (indeed, because) they get really good.⁶

It seems to me like a better solution here is to neuter your adversary’s ability to efficiently target your users by proliferating high-quality LLMs.⁷ This may damage the incumbent frontier AI industry, but if you do this right you’ll also reshape and expand the frontier AI industry. The net effect may even be an improvement on margins you care about. Frontier AI labs, however, may not feel the same way.