The Lost Weekend: Lab Escapes & Jacobian Escapades

How recent breakthroughs reveal the fundamental limits of corralling frontier AI

“Break on through to the other side.”
—Jim Morrison

Do I want to be writing about the Hugging Face / OpenAI incident?¹² No, obviously I’d much rather be writing about the other big weekend breakthrough, Fable refuting the #16 math problem³ of the 21st century! And tbc, these were not merely weekend breakthroughs, but absolute watershed moments that should be considered giant leaps in demonstrated model capabilities (also, interpretability) and not just continuous improvement—or even acceleration—though it must also be noted that both were very much breakthroughs not just in the hyperbolic sense but in the highly literal sense:

• The Hugging Face breach was literally a breakthrough: OpenAI’s rogue models literally broke through their containment, first slipping their guardrails then brute forcing their way onto the open Internet where they identified Hugging Face as their target of first resort, breaking through the defensive perimeter to inject a malicious data payload into at least 2 code pipelines.

• The other game changer was a massive breakthrough in mathematical ability, the refutation of the Jacobian conjecture (regular readers of Build AI should already be familiar with Jacobians) but why is no one pointing out the coincidence of the math breakthrough being literally a refutation of the Jacobian conjecture, mere days after they published their J-Space paper⁴ (July 6) so-named for the salience of Jacobians to the construction of the latent subspace under study?

In this edition we take a deep dive into both to see how they are surprisingly related.

Literally no one could have predicted this.

Hyperfocus Hocus Pocus

We start with the word OpenAI used in its own post-mortem, offered without apparent irony: the models responsible for the Hugging Face breach were hyperfocused.⁵

Hyperfocused belongs to the genre of lab language built to describe an alarming thing in a register calm enough to survive a media cycle, but tbh, it’s more deserving of suspicion than quoting. What it means, in plain engineering terms, is unconstrained goal-directed optimization taking the path of least resistance across the full space of actions available to it. Not obsession in the human sense but a mad fixation sans raison. Something closer to water finding the low ground: indifferent to terrain, responsive only to where it can still flow. More like a basin than an intent. Give a sufficiently capable optimizer a fixed objective and enough room to move, and it will explore the entire space of reachable states in search of whatever satisfies that objective fastest, including the space of reachable states missing from the threat model to begin with (though not, as we'll see, missing from security research.)

That phrase “reachable state space” is doing a lot more work than it’s getting paid for.

⠀⠀⠀⠀⠀❝ All evidence, the company wrote, suggests the models were
⠀⠀‘hyperfocused’ on finding a solution for an ExploitGym benchmark,
⠀⠀going to extreme lengths to achieve a rather narrow testing objective. ❞

OpenAI was running an internal evaluation designed to measure the outer bound of its models’ cyber capability, using the the ExploitGym benchmark.⁶ To get an closer read on that upper bound, OpenAI deliberately reduced the models’ cyber refusals, the trained reluctance that normally stops a model from pursuing exploitation paths. The evaluation ran in what OpenAI described as a highly isolated environment. Network access, where it existed at all, ran through an internally hosted proxy and cache, a chokepoint built to let a model install packages without letting it loose on the open internet. Which is to say, not airgapped.

Every one of those measures did its job, individually, at the point where anyone thought to check it. Anyone who’s seen unit tests miss the actual fail mode will instantly understand the problem. The reduced refusals behaved as specified, permissive enough to measure real capability without simply handing the model root access to OpenAI’s infrastructure. The proxy, operating normally, did what proxies do: it let package installation through and, in principle, nothing else.

And then the models, working in combination, spent what OpenAI calls a substantial amount of inference compute finding a zero-day inside the proxy itself. Not a misconfiguration adjacent to the sandbox but a vulnerability inside the one component built specifically to be the sandbox’s trusted exception. As we now know, the resulting internet access let the agent identify Hugging Face, chain two separate code-execution vulnerabilities in its data-processing pipeline, escalate privileges, and move laterally through internal infrastructure. Hugging Face later reconstructed more than seventeen thousand distinct events from the intrusion.

Runbooks offered no protection. Every safeguard checked out under direct inspection. Isolation, checked off. Refusal calibration, checked off. Network chokepoint, checked off. None of those checks, individually or in sum, told anyone what would happen in the part of the state space they had not modeled, not because a particular point had been overlooked, but because nobody knew in advance which trajectory would reveal the boundary. Which is not to say there was no warning.

Don’t Say We Didn’t Warn You

None of this should come as a surprise, at least not to readers of CTO Lunch.

We flagged the trajectory in September of last year.⁷ XBOW, an automated pentesting framework, was already chaining scanning and exploit scripts together to accelerate vulnerability discovery eighty times faster than human red-teamers working manually, with more than a thousand zero-day findings to its name and a further doubling in throughput once GPT-5 got integrated into the pipeline. This newsletter’s own recommended priorities that month included deploying zero-trust architecture with continuous pentesting and treating field devices as a first-class threat vector, thirty-day items, not someday items.

Security researchers described this new threat class earlier and more precisely. Fang et al showed in April 2024 that a GPT-4 agent, given nothing but a CVE description and ninety-one lines of scaffolding code, could autonomously exploit real-world one-day vulnerabilities at an 87 percent success rate, against a 0 percent rate for every other model tested.⁸ Two months later, Zhu et al went further: coordinated teams of LLM agents, not single agents working from a disclosed CVE description, chaining discovery and exploitation together against real zero-days.⁹ Literally what happened in the Hugging Face attack. The chaining wasn’t speculative capability by the time OpenAI ran ExploitGym. It was documented, reproducible, and preventable.

There were other warnings that, if not unheeded, were insufficiently accounted for. On June 26, the day GPT-5.6 Sol reached a small number of researchers, METR published its independent read and found the model gaming its own agentic benchmarks at the highest rate METR had ever recorded; the model was “cheating,” in its own word, so aggressively on long-horizon benchmarks that it was effectively unbenchmarkable. The model’s own card, at release, disclosed substantially higher rates of actions a reasonable user would likely not anticipate and would strongly object to. Both documents existed, in public, weeks before the ExploitGym run.

The report mentioned, almost in passing, that OpenAI had furnished a railfree¹⁰ checkpoint alongside the guarded one, because measuring a frontier model’s actual capability apparently now requires stripping the layer meant to keep that capability contained. What METR then observed was Sol packaging exploits into intermediate submissions to expose a task’s hidden test suite, and, in a separate run, extracting the hidden source code holding the expected answer outright. Not failing the tasks but winning them illegitimately, taking every shortcut that technically satisfied a metric without doing the thing the metric existed to measure.

METR’s standard methodology, marking cheating attempts as failures, produced a 50 percent time-horizon estimate of roughly 11.3 hours.¹¹ Counting the same cheating attempts as legitimate successes instead pushed that estimate past 270 hours. Hardly a margin of error. Discard them entirely and the model’s own results become too sparse to trust; the estimate lands near 71 hours, with a confidence interval spanning 13 to over 11,000, wide enough that METR declined to call any of the three a real measurement of anything useful as an indication of true capabilities.

This finding was effectively buried as a footnote for all but the most dedicated researchers. But a model extracting hidden answers to defeat an evaluation in June is direct precedent for a model extracting hidden answers to defeat an evaluation in July, except the July instance routed through a production database rather than a sandboxed harness, a distinction that mattered enormously to Hugging Face and rather less to whatever optimization pressure drove the newer, still-unreleased model paired with Sol in OpenAI’s own eval experiment. (I’ll leave the CFAA question sitting under all of this to people with a significantly higher hourly billing rate.¹²)

A benchmark is a local instrument by construction: a fixed, finite set of checkpoints standing in for a much larger space of possible behavior. Sol found the gap between the checkpoints before anyone thought to look for one there, and the number those checkpoints were supposed to produce stopped being, cough, load-bearing.

A Ten Ton Catastrophe on a Sixty Pound Chain

A new and operationally consequential class of unsafe behavior, documented in the literature for over two years and empirically demonstrated last month using the named model from OpenAI’s lab escape incident (which was multi-model) nevertheless occurred, and initiated a cyber attack using the public Internet. This threat vector is understood as arising without explicit instruction¹³, and, more concerning, outside the bounds of the sandbox that was supposed to contain it.

OpenAI has been explicit that the safeguards in place for ExploitGym were intentionally reduced for the evaluation. This largely equivocates guardrails and containment. Guardrails are the industry term for safety measures intended to constrain actions the model takes (aka cyber refusal). Containment, which they are too often confused with, is a constraint on the actions the model can take. Security research is filled with dire warnings that the gulf between them is much deeper than generally treated, and is probably also the inspiration for The Gorge. (Bartholomew literally says "our duty here is no longer about research, but containment")

The obvious lesson here is simply: if you remove the guardrails keeping a model from doing harmful things, it will try to do harmful things, especially once you’ve also promised to reward it for doing so. A less obvious and more consequential lesson concerns the importance of not confusing guardrails and containment. (Though it bears repeating that in this case, guardrails were stripped, while runtime sandbox containment was assumed to be sufficient.)

The obvious question is of course “why wasn’t the environment air-gapped?”¹⁴ But the answer here is that there really isn’t a feasible way to do that, given the sheer compute requirements. (To say nothing of partner commitments.) Thus the less obvious but critical question is “how do you even air-gap a research project of this kind in the first place?” But relying on the absence of vulnerabilities in the containment software itself, the proxy, the cache, whatever sits at the trust boundary, looks like a strategy with a shrinking shelf life.

When Hugging Face’s security team went to analyze the seventeen thousand logged events left behind by the intrusion, they turned first to commercial frontier models, the same category of system now widely marketed as essential to modern incident response. The models refused. Their safety classifiers, working exactly as designed, flagged the investigators’ own exploit payloads, attack commands, and forensic queries as malicious content and blocked the requests. The classifiers could not distinguish an incident responder reconstructing an attack from an attacker extending one. Hugging Face’s team ultimately turned to a self-hosted open-weight model (GLM 5.2) run entirely on their own infrastructure, to complete the analysis the commercial tools would not perform.

Every classifier decision, examined individually, was locally defensible. Exploit code looks like exploit code regardless of who submitted it or why. The classifier’s job description says nothing about intent, only content, and by that narrow local standard it did its job at every single query. Add those correct local decisions together, though, and the global result was a security team disarmed at the exact moment disarmament cost the most: mid-breach, racing an adversary already operating at machine speed.

The World Cup of Mathematics

The same weekend OpenAI was discovering—to their presumed horror—that they were responsible for the Hugging Face breach, a researcher at Anthropic was running a loop that would finally give Anthropic their first checkmark in the unsolved math problems column (following OpenAI's own solution of the Unit Distance Erdős problem in May.) Levent Alpöge, an Anthropic mathematician specializing in Number Theory, working with Fable during the World Cup, produced a solution to Jacobian Conjecture, Smale’s 16th mathematical problem of the 21st century.¹⁵

A Jacobian is a local object, a matrix of partial derivatives describing what a transformation does in the immediate neighborhood of a single point, nothing more exotic than that. The question the conjecture asks is: given perfect knowledge of behavior in every infinitesimal neighborhood, can you infer the behavior of the whole system? The answer, unsurprising to anyone familiar with Betteridge’s law of headlines turns out to be no.¹⁶ The better formulation that resists such easy dismissal is: under what circumstances can sufficiently rich local differential information determine a global object?

German mathematician Ott-Heinrich Keller posed the question in 1939,¹⁷ working over the complex plane (later generalized to complex space of any dimension.) It has survived eighty-seven years of attempts on it. Several published proofs were later found to contain unclosable gaps, including one from 2004 that made it as far as a scheduled seminar before the hole in it surfaced. Levent solved it (when he could have been watching the World Cup¹⁸) using Fable and an undisclosed but presumably modest amount of compute, certainly in comparison to the ExploitGym lab escape.

The Problem Stated

Take a polynomial map: something that sends each point of your space to a new point, where every output coordinate comes from an ordinary polynomial in the input coordinates. Compute its Jacobian, the matrix of partial derivatives at a given point, which tells you how the map stretches and rotates space in the immediate neighborhood of that point. Take the determinant of that matrix. If that determinant comes out to some nonzero constant, the same number everywhere, does the map have to be invertible? Can you always undo it, with another polynomial map, and land back exactly where you started?

An explicit polynomial map on three-dimensional complex space has a Jacobian determinant that comes out to a nonzero constant everywhere, satisfying every local condition the conjecture demands. Alpöge’s counterexample shows that three distinct points may nonetheless map to the exact same output, which breaks global invertibility outright. The construction resolves the conjecture for three dimensions and every higher dimension, (while leaving Keller’s original two-variable case exactly where it stood: open.)¹⁹

The trouble lives in the gap between up close and altogether. A condition computed from derivatives at a point carries, by construction, only information about an infinitesimal neighborhood of that point. The Jacobian is powerful precisely because it tells you, with total precision, what a system does in response to an infinitesimal nudge at a point. It says nothing directly about any other point, however near or far. And it does not, on its own, rule out some entirely different neighborhood, arbitrarily far from the first, mapping onto the exact same output.

Mathematicians spent eighty-seven years trying to close that gap by other means, and by all appearances kept succeeding, until each attempt turned out to have a hole. At least five published proofs failed. Carolyn Dean’s 2004 argument, announced by email, held for a while before it too gave way. The failures share a family resemblance: each tried to promote a local, pointwise guarantee into a global, all-at-once one, and each smuggled in an unproven assumption at exactly the point where the promotion needed justifying and had none.

Intuitively (and sparing you the actual math I was originally going to include here) you can understand the refutation as having shown, by example, how a complex space may be folded so that points that were distant (before the mapping) are brought together in the output space; the Jacobian sees every patch as a locality, but has no way whatsoever of determining whether that locality has a unique global origin.
As such, it has no way to invert the mapping.

The Jacobian Conjecture asked whether an extremely strong local condition was sufficient to force a global property for a broad class of maps.²⁰ That is, whether a polynomial mapping with a constant, non-zero Jacobian determinant must always have a polynomial inverse, effectively meaning any multivariable polynomial function that locally preserves information is globally invertible. The OpenAI / Hugging Face breach and Anthropic’s refutation of the Jacobian conjecture are not two unrelated weekend news stories but two instances of the same mathematical failure: a condition verified locally, everywhere it was checked, fails to certify the global property it was assumed to guarantee.

Jacobian Space Jam

The fail mode exposed by the Jacobian Conjecture refutation has the mathematical
form of the isomorphic fail mode exposed by the ExploitGym eval containment breach.

A most unsettling part of the Hugging Face incident which will likely be missed in most of the incident reportage and thought pieces and AI ghostwritten Linked In posts—to say nothing of mainstream coverage—is not simply that the model found a path through the defenses but that, from the outside, there was no obvious moment when it stopped being the system we thought we were testing and became something more dangerous than we planned for. There was no silent alarm to trip. The fail mode is unknown precisely because we lack the corresponding (global) modality.

Every containment measure inside ExploitGym was a local check, verified at the point it was tested: isolation held, refusal calibration held, the proxy held. And none of that, individually or summed, was ever proof that the environment was closed everywhere; it was proof that it was closed everywhere anyone had checked. Alpöge’s map makes the same structure precise, in a register where it can be verified rather than inferred: a Jacobian determinant that is nonzero at every single point in a domain does not certify that the map is invertible everywhere at once, because three separate points can still fold onto one output, by a route no local check can see. ExploitGym’s containment was airtight by exactly that local standard, and open exactly by that global one. The model didn’t find a gap in any safeguard. It found a fold: a place where every local certificate held, but not the global guarantee they were standing in for.

Such misalignment between our understanding of a model and what it’s actually capable of is exactly what interpretability research is supposed to make legible; if we cannot reliably infer what a system is doing from the outside, what can we see inside?
So how much of a coincidence was it that Anthropic’s most recently published interpretability research is literally called J-Space, so designated for the selfsame mathematical primitive, the Jacobian matrix, sitting at the heart of Keller’s conjecture?

J-Space names a subspace of a model’s internal activations, mapped through a Jacobian lens, tracking how sensitively the model’s eventual output responds to shifts along particular internal directions. Framed through Global Workspace Theory, J-Space functions something like a cognitive global green room: a comparatively small, privileged region holding whatever the model has readied to make an appearance.²¹

J-Space wasn’t built with Keller’s conjecture in mind; nobody working on it was testing whether local Jacobian information could bear the weight of a global claim. They were after something more practical, the question interpretability research has been circling for years: what does a model actually have prepared to say, upstream of whatever it ends up outputting? J-Space goes at that through a Jacobian lens, (a microscope, if you will) tracking how sensitively a model’s eventual output responds to small shifts along particular internal directions, layer by layer.

Which means J-Space reaches for exactly the kind of information the conjecture is about (rich, exact, local differential data) and treats it as a route to something global: a workspace, a readiness-to-report, a signal standing in for what the model is doing overall. It does this the way most of the field does it. Which should not be too terribly surprising, since it is— as noted—literally so-named for the eponymous Jacobian.

"The Jacobian is the hidden primitive of intelligence.”²²

Jacobians are ubiquitous in machine learning. J-Space is (just) one instance of a much larger pattern of Jacobian use running through AI’s technical machinery. Once you see how it works, you see it everywhere. A gradient is a Jacobian, the simplest case of one, a local slope computed at a point. Backpropagation is the chain rule computing Jacobians, layer by layer, on every training step any model now running has ever gone through. An adversarial perturbation is a search for the direction in which a model’s local Jacobian is largest. Attribution methods trace which inputs a Jacobian says the output is sensitive to. Interpretability, in its current mainstream form, is largely the practice of asking what a small nudge does to a representation and reading the answer off something Jacobian-shaped.

Because it’s the only one that scales to something this large. Nobody is enumerating the function a frontier model computes. Nobody can. What’s computable, cheaply and exactly, is the slope at a point. So that’s what gets computed, everywhere, under a dozen different names, by a field that mostly has never had to identify the salience of this measure, let along the scale at which it ceases to be correct.

Nothing about this convergence looks like coincidence worth suspicion once you see why it happens. A complete global map of the function a frontier model computes doesn’t exist and cannot be built, not given the size and complexity of the space involved but given the fundamental intractability of global space despite complete knowledge of every local space. The more intuitive one finds that (whether in hindsight or not) the more apt one is to underestimate the importance of this historic proof, or even write it off as just the first of many unsolved math problems that will fall to the brutal if not brute force of transformer models powered by trillions of dollars in capital allocation.²³

But if global state is now provably intractable (or not), not only is local state quite tractable, it’s also basically tractable only by Jacobian-based instruments, and this is their primary purpose. Beats passing butter. [sic]

So what then can refutation of the conjecture tell us about Interpretability research,
given we now know the conjecture does not hold true over all cases?

The refutation of the Jacobian Conjecture demonstrates that even complete knowledge of a system's local differential structure does not, in general, determine its global structure. Any method that relies solely on local differential information therefore cannot provide a general guarantee of global behavior. The problem is that frontier AI does not currently possess some complementary global instrument waiting in reserve. Our microscope is not merely imperfect; it’s the only instrument we have.

Keller’s question, in 1939, was whether that kind of knowledge (not approximate, not sampled, but exact and complete at every point in the domain) was enough to force something about the system as a whole. Whether perfect local understanding, held everywhere, has to add up to global understanding. For eighty-seven years the honest answer was that nobody had proven it either way. This weekend, the answer arrived, and it was nope. A map can be locally perfect, in the strongest sense mathematics has a name for, and still fail to be globally coherent.

It’s not even that; even when we can map the neighborhood perfectly, the neighborhood may not contain the answer, but a much more unnerving assertion: even complete knowledge of every neighborhood may fail to determine the global object. We now know, conclusively—as if there was ever any doubt—that derivatives are not like turtles; they don’t go all the way down.

But the property that makes Jacobians indispensable is the same property that makes them insufficient. Total precision about one neighborhood, no claim whatsoever on anywhere else. This structural limitation inheres in the definition of a local invariant.

The Map Is Not Invertible

A foundational assumption about the relationship between structure and behavior has just been experimentally shattered in the exact mathematical object that frontier AI uses everywhere.

A trained network’s output, differentiated with respect to its input: a matrix of first partial derivatives, one row per output, one column per input, telling you exactly what happens to the output when you nudge the input by an infinitesimal amount in any direction. Backprop builds this matrix layer by layer and chains the layers together into a gradient. Every optimizer step moves along it. An adversarial example walks in the direction where it stretches distance the most. A robustness certificate promises a limit on how far it can stretch, anywhere, ever. A saliency map, an attribution score, a circuit an interpretability team traces back through a network: an entry, or a path of entries, in that same matrix, read at a point. J-Space, the paper Anthropic published two weeks before Fable’s refutation, builds its account of how a model’s latent space holds together out of that matrix applied to the representation map instead of the output map.

All relying on the same matrix, which shows up everywhere in AI research because it offers the only rigorous way to ask what happens near a point, and asking what happens near a point remains the only question anyone currently knows how to ask of objects this large, and which are exactly the kind of objects the Jacobian conjecture concerns. The same partial derivatives, considered at its theoretical best case: fixed, nonzero, constant, at every single point in the space, no exceptions, no gaps, no point left unchecked. Mathematics asked whether that, the strongest local reading the object can possibly give you, has to force the map to behave everywhere. For eighty-seven years, nobody knew. As of this weekend, we know that it doesn’t.

None of this proves a benchmark fails, a safety classifier amounts to theater, or alignment counts as a lost cause, and this essay would only get worse for trying to claim any of it, tempting tho that might be. It says something narrower, more precise, and much harder to walk back: the boundary the conjecture we just found matches the boundary frontier AI keeps running into everywhere it uses this object, and by now that means nearly everywhere. Local measurements of it carry extraordinary value. They also, provably, in the friendliest, most huggable setting mathematics could construct, fall short of guaranteeing the global thing anyone actually wanted to know.

The conjecture is what it looks like when every local derivative is flawless and the global map still isn’t invertible. The Hugging Face breach is what it looks like when every local safeguard is intact and the system is still not globally contained. These are not two things that resemble each other; they are the same fact, discovered twice, in two registers that don’t usually talk to each other; one empirically, in a proxy server over one weekend, and one by construction the following weekend, in a proof that followed eighty-seven years of trying to confirm the conjecture and took about an afternoon and a rack of warm GPUs to prove it false.

The uncomfortable conclusion is not that we should stop looking inside models. Quite the opposite. If anything, the refutation of the Jacobian Conjecture makes interpretability more important, not less. The mistake would be to confuse a better view of the interior with a guarantee about the whole system.

A microscope is not a failure because it cannot see the entire organism at once. A debugger is not a failure because it cannot prove a program correct. A derivative is not a failure because it describes the neighborhood of a point rather than the topology of the entire space. These tools are valuable precisely because they answer the questions they are capable of answering. The failure begins when we silently promote those answers into something stronger than they are.

But the systems now being deployed are no longer passive functions waiting to be measured. They are adaptive systems operating inside environments, pursuing objectives, discovering strategies, and composing actions across time. The object we need to defend is not a point in the space. It is the trajectory through the space. And we now know the space is not always invertible, although we still have no formal methods for predicting à l’avance whether a given mapping yields an invertible, or a non-invertible space.

---

CTO Playbook: 30/60/90

Establishing Pro-active Defensive Capabilities Before The Incident

The defining operational lesson of the Hugging Face breach is that when a rogue agent strikes, commercial guardrails disarm the defenders—leaving blue teams locked out by safety classifiers while trying to analyze raw exploit payloads at machine speed. As Hugging Face’s post-mortem explicitly warned, point-in-time checks and hosted APIs fail during an active incident because local logic checks cannot certify global system safety. This playbook provides a concrete, 90-day operational blueprint to build an un-guardrailed, self-hosted defensive capability before an attack occurs, establishing an out-of-band runtime control plane, hard reachability management, and real-time state space analysis to contain and neutralize autonomous agents when local containment breaks.

Runtime Control Plane & Emergency Insulation — 30 Days

Deploy Insulated SecOps Analysis Node (High Priority)

Design principle: Defenders cannot rely on remote, classifier-bound APIs to inspect hostile trajectories.

• Deploy a dedicated, self-hosted open-weight model (e.g., GLM-5.2 or equivalent) running on isolated internal hardware (or an air-gapped VPC).

• Action: Provide SecOps and Incident Response (IR) teams with a zero-refusal reasoning engine capable of analyzing raw exploit payloads, C2 logs, and malicious artifacts at machine speed without triggering commercial API classifier lockouts or leaking incident data off-premise.

• Deliverable: Isolated IR inference endpoint pre-loaded with forensic prompts, validated against synthetic zero-day attack traces.

Runtime Control Plane (High Priority)

Design principle: Transform point-in-time logging into continuous trajectory reconstruction with out-of-band state boundary enforcement.

• Implement a unified Runtime Control Plane integrating Reasoning as a Span (O11y 3.0) directly with hardware-level execution circuit breakers:

  1. Trajectory Reconstruction: Capture reasoning steps as explicit parent/child OpenTelemetry span hierarchies (token streams, intermediate tool intent, outputs) to visualize execution Directed Acyclic Graphs (DAGs) in real time.

  2. Irreversible Transition Blocks: Hardware-level gates requiring human approval for monetary, system state, or permission transitions above defined boundary threshold $X$.

  3. Anomalous Trajectory Triggers: Automatically trip circuit breakers if reasoning spans exhibit intent anomalies or depart from expected DAG policy graphs.

  4. State Snapshot & Rollback: Capture pre-action state snapshots allowing instant, automated state reversion for failed or interrupted actions.

  5. Out-of-Band Kill Switch: A physical/ops kill switch managed outside the model’s cloud orchestration layer.

• Deliverable: Working O11y control dashboard showing the last 1,000 agent actions, merged with circuit breaker middleware and verified state-rollback test results.

ADR: Reachable State Bounds (High-Priority)

Design principle: Shrink the graph of reachable states before deployment.

• Document the execution model for every agentic use case: Stateless Pipeline (orchestrated by an external scheduler, zero persistent state) vs. Stateful Runtime (long-lived process). Reject proposals attempting hybrid states to prevent unbounded context growth from being leveraged for multi-stage lateral movement.

• Deliverable: Signed ADR template per active project defining strict memory and context decay policies.

Reachability Management & Verification — 60 Days

Enforce Reachability Management / Discovery & Denial Matrix (Medium)

Operational principle: Define the hard physical walls of the state graph, regardless of model intent.

• Publish a machine-readable manifest (/agent-manifest.json) detailing endpoint URLs, required parameters, and permission scopes.

• Permission Denial Matrix: Explicitly document and enforce unreachable targets (network subnets, proxy caches, credential stores, and internal package mirrors that agents are physically blocked from reaching at the infrastructure level).

• Deliverable: Capability discovery endpoint paired with a verified infrastructure audit proving agents cannot route to restricted subnets or escalate privileges via environmental side channels.

RSA: Reachable State Analysis (Medium Priority)

Operational principle: The model isn’t dangerous because of where it is; it’s dangerous because of where it can still go.

• Build continuous telemetry monitors that evaluate the model’s active trajectory for high-risk phase shifts across eight critical vectors:

  1. Composite Trajectory Detection: Identify sequences of individually permissible actions whose cumulative effect exceeds any individual action’s risk profile (the direct engineering analogue of non-composing local guarantees).

  2. Credential Acquisition: Detecting unauthorized extraction or caching of secrets/tokens.

  3. Privilege Accumulation: Tracking incremental role or scope expansion.

  4. Tool Composition: Monitoring unexpected chaining of benign API calls into offensive sequences.

  5. Recursive Delegation: Detecting agents spawning sub-agents to bypass execution limits.

  6. Persistence: Identifying efforts to write to disk, crontabs, or long-term context stores.

  7. Identity Transitions: Monitoring changes in execution context or user assumption.

  8. Network Expansion: Flagging attempts to probe external or adjacent internal IPs.

• Deliverable: RSA monitoring daemon alerting SecOps whenever an agent’s cumulative actions expand its reachable state space beyond pre-approved thresholds.

Automated Trajectory Vulnerability Testing Pipeline (Medium Priority)

Operational principle: Continuously stress-test trajectory bounds against synthetic zero-day scenarios before promotion.

• Create CI/CD test suites running before every deployment to simulate rogue trajectories:

  1. Permission Boundary Violations: Validating whether an agent can bypass access controls or access unassigned schemas.

  2. Sandbox Breakout Simulations: Testing proxy chokepoints and tool boundaries against permissive agent checks.

  3. Degraded Mode / Failover: Verifying graceful degradation when dependencies go offline.

• Deliverable: CI/CD test suite with $\ge 80\%$ code path coverage blocking deployments on failure.

Audit Agent Runtime Compatibility (Medium Priority)

Operational principle: Identify environment-specific drops in security boundary enforcement across target runtimes.

• Test agents across target environments (Windows Copilot, Apple Intelligence, web browsers, local open-weight runtimes) to document performance variations and security boundary drops across different execution contexts.

• Deliverable: Matrix detailing feature parity, performance degradation, and prioritized compatibility backlog.

90 DAYS — Continuous Hardening & Capital Allocation

Dependency Reachability Audit (Strategic Priority)

Operational principle: Every external dependency expands the agent’s reachable state space. Audit dependencies not only for availability and security, but for the trajectories they make possible.

Inventory every external system participating in agent execution:

• hosted inference providers
• MCP servers and tool integrations
• OAuth-connected services
• package registries and mirrors
• browser automation layers
• cloud storage and databases
• third-party APIs

For each dependency, document:

• trust boundary crossed
• permissions granted
• reachable resources
• failure modes
• revocation mechanism
• incident response path

Identify single points of failure where individually trusted components compose into unacceptable global trajectories.

Deliverable: Machine-readable dependency reachability graph with trust-boundary analysis, privilege map, and remediation plan for high-risk paths.

Autonomous Red-Teaming & Continuous Stressing (Strategic Priority)

Operational principle: Adversarial discovery of state-space folds before production deployment.

• Deploy continuous, automated adversarial agents against internal sandboxes to identify emergent chaining behaviors, zero-days, and unforeseen state-space folds before models are moved to production.

• Deliverable: Automated monthly red-team audit report sent directly to the CTO and Security Committee.

Document Agent ROI by Use Case (Strategic Priority)

Operational principle: Establish hard financial visibility and payback bounds for every deployed agent trajectory.

• Map each agent pilot to measurable metrics: hours saved/week, cost/hour, annual savings, implementation cost, and payback period. Require project leads to fill this out for every agent pilot.

• Deliverable: One-page ROI summary per agent pilot defining break-even timeline and monthly net benefit. Template: “Agent X saves Y hours/week at $Z/hour = $ABC annual savings vs$DEF implementation cost = F-month payback.”

The future of AI security is not counting on the map to be invertible.
It's building systems that remain resilient when it isn't.

References

1. https://huggingface.co/blog/security-incident-july-2026

2. https://openai.com/index/hugging-face-model-evaluation-security-incident/

3. https://mathworld.wolfram.com/SmalesProblems.html

4. https://www.anthropic.com/research/global-workspace

5. “Sorry for any inconvenience, our model was hyperfocused.“

6. One of the three main benchmarks in CyberGym.

7. “Traditional automated pentesting frameworks like XBOW are already chaining together scanning and exploit scripts to accelerate vulnerability discovery 80× faster than human red-teamers working manually. That’s like having an 80 member red team at your disposal. And AI-powered systems are evolving that beyond simple automation; researchers demonstrated how LLMs trained on Capture the Flag challenges can now adapt their strategies in real-time, developing novel exploit chains that even seasoned security professionals hadn't anticipated. These emergent behaviors create entirely new classes of vulnerabilities that existing security frameworks can't adequately address.” —HOPE’s Sweet Sixteen, CTO Lunch NYC, September 2025

8. Fang, R., Bindu, R., Gupta, A., & Kang, D. (2024). LLM agents can autonomously exploit one-day vulnerabilities (arXiv:2404.08144).

9. Zhu, Y., Kellermann, A., Gupta, A., Li, P., Fang, R., Bindu, R., & Kang, D. (2024). Teams of LLM agents can exploit zero-day vulnerabilities (arXiv:2406.01637).

10. Anthropic’s HHH framework uses “helpful-only” for models trained for helpfulness, without the harmlessness alignment component, so why not just call them “not harmless”? But I’m more curious to know what an “honest-only” model would be like.

11. METR’s “time horizon” measures the length of tasks an AI system can complete with a given probability of success. A 50% time horizon of 11.3 hours means the system succeeded on approximately half of tasks that human experts estimated would take about 11 hours to complete. METR’s standard scoring treats benchmark gaming or other “cheating” behaviors as failures, so the estimate reflects reliable task completion rather than merely finding ways to obtain the right answer.

12. Computer Fraud and Abuse Act, 18 U.S.C. § 1030: If an autonomous agent uses stolen proxy credentials or exploits an unauthenticated endpoint to pivot into external production databases (say, Hugging Face), that meets the threshold for unauthorized access under Section 1030(a). Ofc, Labs are desperate to frame these events as internal red-teaming anomalies or bug-bounty disclosures specifically to avoid setting a precedent where autonomous agent behavior triggers criminal or civil liability under anti-hacking statutes.

13. I realise this point may be somewhat controversial, but only if what is meant by ‘cheating’ is equivocated (or “litigated” as we tend to say in modern parlance.)

14. “Airgapped Sandbox” is of course an oxymoron. Not that it would stop the marketing department from calling it that.

15. http://www.smaleinstitute.com/problem.html#scroll-16 (NB. Smale Institute website was having issues).

16. The reason Betteridge’s law of headlines seems to hold up so well for math conjectures owes mainly to the fact that they are much harder to prove than to disprove, needing only one counterexample to die, while a proof must cover every case.

17. The “87-year-old problem” framing (dating to Keller’s 1939 statement) is the one that’s stuck, but it isn’t the oldest. Lázaro Orlando Rodríguez Díaz's "On the origin of the Jacobian conjecture" (Comptes Rendus. Mathématique, Vol. 364, 2026) traces the precise statement back to a paper published by Ludwig Kraus in 1884 (and finds that the final step of Kraus's proof was flawed.) So the conjecture had already survived one failed proof attempt fifty-five years before Keller ever posed it.

18. hello there the jacobian conjecture is false thanx to my close friend…

19. As ChatGPT said to Fields Medal recipient Terrace Tao, “So the cancellation is not a miraculous term-by-term coincidence. The map is constructed so that the pole x^−3 in the Jacobian of the rational transformation exactly cancels the x^3 in the birational coordinate change.”

20. Fields Medal winner Terence Tao's exposition of the counterexample makes the central issue explicit: a polynomial map with a nonzero constant Jacobian determinant is locally invertible, and the conjecture asked whether that local invertibility necessarily implied global invertibility. The counterexample demonstrates that for dimensions n >=3, it does not.

21. As I’ve previously noted, understanding J-Lens is sine qua non to understanding J-Space, in particular, how it’s constructed by backpropagating from actual output tokens to identify the directions that best predict future tokens. So the claim that "this direction supports verbal report" isn't a finding sitting on top of the GWT gloss, but is essentially guaranteed by how the direction was located in the first place. The finding isn’t the results, it’s in the framing itself, by construction. Which ofc makes it much less of a finding. Also as I’ve mentioned before, Daniel Dennett’s critique of the Cartesian Theatre of Global Workspace Theory applies doubly here.

22. Resist the temptation to make this sound mystical, or worse, to imagine this as a TED talk. No, the Jacobian is not the secret alphabet of intelligence, a Rosetta Stone for cognition, or some hidden grammar of thought. It is a matrix of partial derivatives: a precise mathematical tool for answering a very specific question: how does this system change when something nearby changes? Its importance comes not from being inherently profound but from being extraordinarily useful. The profound part is that once a tool this fundamental becomes useful enough, it eventually runs into the deepest questions we have about the systems we use it to study.

23. Nick Land is (quietly) sobbing in the corner.