AI environment isolation failed us before the model even ran

On 29 July 2026, Skybridge's production agents table went from 22 rows to zero. An earlier deletion had also reduced recorded automation runs from 67 to 3. Development remained intact.

The failure occurred before the language model ran. A legitimate internal script believed it was using development configuration while its database client had already initialized against production.

We restored the missing records from a verified snapshot the same day. The incident left us with a harder and more useful definition of AI environment isolation: development and production must use separate resources, and every privileged operation must prove its target before it can run.

An environment variable supplies configuration; structural isolation requires independent resources and target checks.

The script said development while the client had already chosen production

The root cause sat in JavaScript module loading.

The script configured its environment and then called database functions. In the source file, that order looked sensible. But a static ES module import is resolved before the rest of the module body executes. The imported database package initialized its client before the script set the development environment path.

By the time the code reached the line that appeared to choose development, the database client had already chosen production.

The credential was valid, and any replacement would have authenticated the same mistaken target. The correction had to address target resolution.

We changed affected scripts to set configuration before a dynamic import. We added loud target output and database-project assertions. Then the team inspected every script in the directory for the same pattern. Eleven scripts were checked, and one additional latent case was fixed.

Separate names do not create separate failure domains

A development URL, a dev flag, and a second configuration file can make a system look separated. The real question is whether one mistake can still cross the line.

If both environments share a database account, service credential, scheduler, connector gateway, or mutable image tag, the boundary may be administrative rather than structural. A script can read the wrong file. A deployment can reuse the wrong client bundle. A background job can send a real email from a development pod.

AI systems add consequences because a test may invoke paid models, read connected business tools, or perform an external action. Environment isolation must cover data, execution, and side effects.

Build five structural boundaries between development and production

Skybridge now treats environment identity as a set of independent controls.

The browser bundle deserves attention. Some frontend values are compiled at build time. Reusing a production image for development can preserve production endpoints even when the runtime environment looks correct. Build artifacts need an environment identity too.

Kubernetes supports controlled Deployment rollouts and revision history, but the platform cannot decide whether an image was built with the right customer-facing configuration. That evidence belongs in the delivery process.

Make every destructive script prove its target

A script capable of deleting, migrating, cloning, or repairing data should begin by proving the database it will touch.

The safer Skybridge pattern parses the intended environment file explicitly, extracts the database project identity, compares it with an expected value, and stops on mismatch. Only then does it construct the privileged client. The target is printed before the operation. Destructive work can also require a typed confirmation or a guarded wrapper.

The order matters:

1. Parse the selected configuration without silently inheriting process-wide values.
2. Assert the expected host or project identifier.
3. Load code that constructs the database client.
4. Print the resolved target and operation scope.
5. Run a read-only preflight count.
6. Execute with the narrowest possible selection.
7. Verify postconditions and record the result.

Avoid a “helpful” fallback to production when development configuration is missing. A missing target should stop the script.

Disable external consequences in non-production

Database separation is only half of AI environment isolation.

A development agent may connect to a real calendar to reproduce a read issue. It should not be able to create an event while the engineer experiments. A test automation may inspect a message thread. It should not send the prepared reply.

Skybridge uses an outbound-disable control in non-production. Reads can be permitted for a scoped test where authorized; sends and mutations are stopped. Test accounts and test destinations remain preferable where the provider supports them.

The kill switch needs tests of its own. Some APIs use POST for search, so blocking every non-GET request can break legitimate reads. Classify operations by their effect and verify the method separately.

Backups need a verified recovery inventory

The production snapshot saved the incident. The first recovery still missed two tables.

That happened because the recovery inventory was incomplete. Parent records returned, but agents and automation_runs were absent from the restore list. A later full-table comparison found the gaps.

A completed backup proves that data was copied. Recovery testing must separately verify row counts, relationships, authentication records, stored files, scheduled work, and the application behavior users depend on.

For Skybridge, the full comparison covered 33 tables. The restore used guarded per-row inserts, checked that parent records existed, and avoided overwriting records already recreated after the snapshot.

Recovery evidence should include what was restored, what was intentionally excluded, and how live changes during the incident were reconciled.

Monitor impossible decreases and environment drift

Some production tables should rarely decrease without an approved event. Tenant, agent, and automation counts can therefore act as tripwires. A sudden unexplained drop deserves attention even when health endpoints remain green.

Monitor the target identity too. Scheduled jobs and administrative scripts should report which environment and database they resolved. Deployment status should show the code revision, image tag, data project, and configuration version together.

The incident also changed our review behavior. The team audited all scripts for the import-hoisting pattern because a recurring cause requires a class-level correction.

NIST's Secure Software Development Framework describes addressing root causes to prevent recurrence. This incident gave that sentence a row count and a date.

Compsia treats environment and rollback evidence as part of the exact production release. The two AI production security gates explain where that evidence belongs before customer data or recurring action is authorized.

Questions about AI environment isolation

Should development ever use production data?

Prefer fictional or genuinely de-identified data. Where a production issue requires real data, use a specifically authorized and minimized path, a scoped account, clear retention, and controls that prevent external action.

Are separate database schemas enough?

They can reduce accidental overlap, but separate projects or accounts create a stronger failure boundary for privileged scripts, credentials, and configuration. The required separation depends on consequence and operating capability.

What is the minimum production deployment gate?

Confirm green tests, the exact code and image revision, the target environment, required migrations, rollback readiness, and production acceptance for the changed path. The responsible owner should make promotion explicit.