aipackage/specs/001-enterprise-experience/contracts/update-rollback-policy.md

Shared Offline Update and Rollback Policy

Status: Approval draft; no implementation authority.

Scope

One signed offline-update format applies to Workstation and Server S/M/L/Max. The manifest selects compatible actions by product category, internal tier, platform version, architecture, provisioned components, and hardware profile. An update never changes commercial entitlement.

This policy covers platform files, portal/backend, deployment configuration, system services, containers, Python/runtime environments, database schema, models, drivers, kernel/OS packages, trust roots, and local documentation. Each surface declares its rollback class before confirmation.

Existing-State Finding

The current backup helper archives /opt/cezen/data and creates a pre-restore data snapshot. Existing SQLite migrations are intended to be transactional and idempotent. This is useful customer-data protection but is not a complete platform rollback: it does not prove recovery of application files, service units, Ansible state, container images, Python environments, drivers/kernel, models, trust policy, or every configuration outside the data directory.

No update may claim rollback solely because the current ZIP backup completed.

Update Classes

Class	Examples	Required rollback class
A — Content/config	Portal, local help, additive configuration	Automatic side-by-side switch where possible
B — Application/runtime	Backend, Python environment, containers, Ollama/vLLM runtime	Automatic or operator switch to retained last-known-good immutable payload
C — Data/schema	SQLite schema or data transformation	Transaction rollback before commit; verified data restore after commit
D — System	Kernel, NVIDIA driver, container runtime, OS packages, systemd/Nginx	Reboot-aware snapshot/package rollback or verified restore/recovery procedure
E — Trust/security floor	Trust roots, revocation data, minimum accepted version	Operator rollback only when signed policy permits; never below active security floor
F — Model pack	Model objects, templates, runtime metadata	Atomic activation pointer; retain previous model until inference validation passes

An update containing multiple classes inherits the least reversible class. rollback_class: none is not allowed for routine release. An exceptional non-reversible update requires a separate signed authorization, verified full recovery image, product/security approval, explicit customer impact, and a controlled-pilot gate.

On-Disk State and Retention

The target architecture maintains:

• one immutable current platform inventory;
• one immutable last-known-good platform inventory;
• one verified pre-update data/config backup compatible with the last-known-good platform;
• content-addressed update staging separate from active paths;
• current and previous container/runtime/model digests needed for the declared rollback;
• durable update operation, audit, manifest, signature, validation, and recovery evidence.

Current and last-known-good versions MUST be activated through an atomic pointer/symlink or equivalent transactional switch wherever supported. Mutable customer data is never stored inside versioned application trees. Models may remain in a shared content-addressed store; activation metadata is versioned and garbage collection cannot remove an object referenced by current, last-known-good, backup, or active staging state.

Default retention is current plus one last-known-good version and the newest verified pre-update backup. Additional history is optional and capacity-bound. Preflight computes staging, unpacked, backup/snapshot, previous-version, model, and safety-headroom bytes separately. Insufficient capacity blocks the update before mutation; the updater never deletes last-known-good recovery state to make space automatically.

Preflight and Checkpoint

Before mutation:

1. Complete trust, signature, object, compatibility, license, security-floor, and capacity checks.
2. Reject conflicting installation, update, restore, removal, model-import, or trust operations.
3. Inventory current source/build, application files, configs, services, packages, containers, runtimes, schema, models, trust policy, and active hardware profile.
4. Verify the current state is healthy enough to distinguish a pre-existing fault from update impact.
5. Create the class-required checkpoint and verify readability, checksums, schema compatibility, and restore instructions. A path existing is not verification.
6. Display service disruption, reboot requirements, estimated space/time, rollback class and limits, retained/deleted state, and recovery owner before authorization.

Activation Order

Default ordered phases are received, verified, staged, prechange_saved, quiescing, data_migrating, switching, services_starting, awaiting_reboot, validating, and terminal succeeded, rolled_back, failed_safe, or recovery_required.

• Stop writes or establish an application-consistent data checkpoint before schema/application switch.
• Apply idempotent migrations sequentially in one transaction where SQLite permits.
• Switch application/config/container/model references only after all target objects are local.
• Restart services in dependency order with bounded timeouts.
• A required reboot resumes the same operation ID from durable state.
• Do not garbage-collect staging or last-known-good content until validation and approval complete.

Health Validation and Commit

The proposed default validation window is ten minutes after all required services become ready, plus explicit checks for database/schema, portal/API, authentication, license/entitlement, storage, selected local model inference, container/service restart count, and hardware runtime where relevant. The window restarts after a required reboot or unexpected service restart.

Success requires every mandatory check to pass and the active inventory to match the signed target manifest. Optional remote connectors may remain explicitly unavailable when no approved route exists. Only then does target become current and former current become last-known-good.

Rollback Decision

Condition	Required action
Failure before active switch	Delete/quarantine staging; current remains active; failed_safe
File/runtime switch fails before data commit	Switch to last-known-good; restart/validate; rolled_back
Transactional migration fails	Roll back transaction; restore application pointer if changed
Post-commit schema/application validation fails	Stop writes; restore verified pre-update data plus last-known-good application as one recovery unit
Driver/kernel boot failure	Boot approved previous kernel/snapshot or enter documented local recovery; never loop indefinitely
Health window fails	Automatic rollback when class permits; otherwise recovery_required with checkpoint identity
Rollback validation fails	Preserve both failure causes and all evidence; stop in recovery_required; no repeated automatic loop

Automatic rollback is attempted at most once per operation. A retry creates a linked operation and requires renewed preflight. Rollback does not bypass the installed signed security floor.

Interruption Matrix

• Before checkpoint: current remains active; restart verification/staging safely.
• During checkpoint: incomplete checkpoint is invalid and cannot authorize activation.
• During transactional migration: transaction rolls back; verify DB before retry.
• Between migration and application switch: resume declared order or restore checkpoint; never infer completion from file presence alone.
• During pointer switch: reconcile signed inventory to exactly current or target, never a mixture.
• During service restart: resume validation against recorded active inventory.
• During reboot: boot marker resumes the same operation; bounded boot attempts trigger recovery.
• During rollback/restore: preserve original and rollback error; require operator recovery if the verified checkpoint cannot be restored.

Ownership and Approval

Responsibility	Accountable owner
Bundle production, manifest and action ordering	Platform release owner
Signing, security floor, trust changes, emergency authorization	Security owner
Migration forward/restore compatibility	Data/backend owner
Driver/kernel/OS recovery	Platform delivery owner
Model/runtime compatibility and inference checks	AI platform owner
Customer disruption, support runbook, recovery exercise	Delivery/support owner
Release scope and controlled pilot	Product owner

Evidence Gate

Approval requires positive update and rollback from every supported previous baseline plus negative tests for wrong product/tier/architecture, revoked key, stale policy, corrupted object/cache, low disk, conflicting operation, checkpoint failure, migration failure, service failure, network removal, power loss at every mutating boundary, reboot failure, health-window failure, automatic rollback, restore-based rollback, and rollback failure. Evidence records both active and target inventories, schema versions, checkpoint checksums, operation/correlation IDs, service/model checks, and final recovery status.

Owner Decisions Still Open

• Approve ten-minute health window or choose another measured value.
• Select the platform versioned-tree/atomic-switch mechanism and system snapshot mechanism.
• Define full backup inventory beyond /opt/cezen/data, encryption, external-copy policy, and restore SLA.
• Define supported previous-version window, update cadence, retention beyond current+one, and disk quota.
• Define kernel/driver rollback mechanism for each supported hardware/OS profile.
• Approve exceptional non-reversible/break-glass authority and customer confirmation language.