Both the improper-updating result (Proposition 3) and the geometric certification cost (Lemma 2) turn on a claim about representation: that a language model generically does not carry the exact mechanism of internally, nor route each conditioning step to it. Because that claim is easy to mistake for one about behavior, we make its status explicit and connect it to independent evidence.
We separate two kinds of claim. The first follows deductively from the generic-circuit model introduced in the proof of Proposition 3 and is stated as corollaries below; it concerns the prior measure of the exception and its generic behavior as the model grows. The second is inductive: it draws on evidence about actual trained networks to fix our posterior belief about whether scale (or continual training) meets the exception in practice, and is therefore recorded as corroboration rather than as a definitive statement.
Both statements hold a priori, taking only the generic-circuit model of the proof as given.
Model the realized circuit as a draw from the function class of an -parameter network of structural complexity , as in the proof of Proposition 3. Then the exception clause—that both encodes the exact mechanism of and routes every conditioning step to it—has prior measure that is non-increasing, and generically strictly decreasing, in and : the count of -parameter circuits grows super-exponentially while the subset wired to condition exactly does not, so the correctly-wired fraction tends to zero. A priori, therefore, the exception is generically not met, and enlarging the model makes it less, not more, likely; improper updating is not an artifact of insufficient scale.
Moving the update from the input (in-context tokens or retrieval) into the weights—a gradient step of continual or online learning—does not restore the guarantee of Proposition 2. A gradient step on a next-token, or any surrogate, objective is not the Bayesian update : it perturbs ’s parameters to lower a training loss, not to condition the implied predictive on , so the non-preservation (i) and non-convergence (ii) of Proposition 3 carry over verbatim to the weight-update dynamics. The distinction is one of where knowledge lives—in the input or in the weights—and in neither place does absorbing evidence amount to Bayesian conditioning. Knowledge in the input is re-paid on every query and, though auditable, does not constrain the output to be a posterior; knowledge in the weights is amortized across queries but opaque, and its update is a gradient step rather than Bayes’ rule. Worse, the weight route adds a domain-dependent free parameter—the objective, learning rate, and schedule must themselves be tuned per domain merely to approximate the target—so it trades an unconstrained-but-auditable input-update for an unconstrained and less auditable weight-update. A proper posterior requires a third locus: knowledge in an explicit, verified structure that conditions—amortized like weights yet auditable and Bayes-updatable like the best input—which is exactly what coupling to a GWM it calls provides (Mode 2/3, Section 4.1).
The corollaries bound the prior measure of the exception, not the posterior probability that a particular trained model meets it. The natural objection closes exactly that gap: perhaps a sufficiently capable model, having seen enough data, simply does land in the exception set, so that matching the true predictive on the evidence seen so far is evidence the mechanism has been internalized. This is the Platonic Representation Hypothesis (Huh et al., 2024) in its strong form—that scale and data coverage drive models toward a single, shared, ground-truth representation of the domain. If it held, the exception would be the generic case in practice despite its small prior measure, and our results would be vacuous.
The strong hypothesis is contradicted by direct study of trained networks. A model can reproduce a target function’s outputs exactly while computing them internally as a fractured, non-modular collection of local heuristics rather than the single unified mechanism that generated the data—the phenomenon of Fractured Entangled Representation (FER) (Kumar et al., 2025). The same phenomenon has been reported mechanistically inside large models: arithmetic solved by range-limited heuristics rather than a unified algorithm, and a learned board-game world model realized as a bag of local rules rather than the game’s actual laws. Crucially, behavioral evidence of fracturing can vanish with scale while the internal fracturing persists, so output agreement is not a reliable signal that the mechanism has been captured.
This evidence moves our posterior toward the deductive prior, not against it. In Proposition 3, coinciding with on the evidence seen so far is exactly such a behavioral match; FER gives independent reason to expect it is not backed by an internalized mechanism, so there is no ground to expect to track under the next conditioning step. In Lemma 2, the absence of a unified internal mechanism is what forces certification to proceed heuristic-by-heuristic: each invariant must be patched by local edits to the state rather than falling out of one mechanism that already satisfies them jointly, which is what makes joint certification geometrically expensive. Inductively, then, trained networks exhibit precisely the failure that Corollary 2 shows to be generic a priori—the empirical counterpart of FER’s rebuttal of naive representational optimism.