160 lines
9.7 KiB
Markdown
160 lines
9.7 KiB
Markdown
# Shared Offline Update and Rollback Policy
|
|
|
|
**Status**: Approval draft; no implementation authority.
|
|
|
|
## Scope
|
|
|
|
One signed offline-update format applies to Workstation and Server S/M/L/Max. The manifest selects
|
|
compatible actions by product category, internal tier, platform version, architecture, provisioned
|
|
components, and hardware profile. An update never changes commercial entitlement.
|
|
|
|
This policy covers platform files, portal/backend, deployment configuration, system services,
|
|
containers, Python/runtime environments, database schema, models, drivers, kernel/OS packages, trust
|
|
roots, and local documentation. Each surface declares its rollback class before confirmation.
|
|
|
|
## Existing-State Finding
|
|
|
|
The current backup helper archives `/opt/cezen/data` and creates a pre-restore data snapshot. Existing
|
|
SQLite migrations are intended to be transactional and idempotent. This is useful customer-data
|
|
protection but is **not** a complete platform rollback: it does not prove recovery of application
|
|
files, service units, Ansible state, container images, Python environments, drivers/kernel, models,
|
|
trust policy, or every configuration outside the data directory.
|
|
|
|
No update may claim rollback solely because the current ZIP backup completed.
|
|
|
|
## Update Classes
|
|
|
|
| Class | Examples | Required rollback class |
|
|
|---|---|---|
|
|
| A — Content/config | Portal, local help, additive configuration | Automatic side-by-side switch where possible |
|
|
| B — Application/runtime | Backend, Python environment, containers, Ollama/vLLM runtime | Automatic or operator switch to retained last-known-good immutable payload |
|
|
| C — Data/schema | SQLite schema or data transformation | Transaction rollback before commit; verified data restore after commit |
|
|
| D — System | Kernel, NVIDIA driver, container runtime, OS packages, systemd/Nginx | Reboot-aware snapshot/package rollback or verified restore/recovery procedure |
|
|
| E — Trust/security floor | Trust roots, revocation data, minimum accepted version | Operator rollback only when signed policy permits; never below active security floor |
|
|
| F — Model pack | Model objects, templates, runtime metadata | Atomic activation pointer; retain previous model until inference validation passes |
|
|
|
|
An update containing multiple classes inherits the least reversible class. `rollback_class: none` is
|
|
not allowed for routine release. An exceptional non-reversible update requires a separate signed
|
|
authorization, verified full recovery image, product/security approval, explicit customer impact, and
|
|
a controlled-pilot gate.
|
|
|
|
## On-Disk State and Retention
|
|
|
|
The target architecture maintains:
|
|
|
|
- one immutable **current** platform inventory;
|
|
- one immutable **last-known-good** platform inventory;
|
|
- one verified pre-update data/config backup compatible with the last-known-good platform;
|
|
- content-addressed update staging separate from active paths;
|
|
- current and previous container/runtime/model digests needed for the declared rollback;
|
|
- durable update operation, audit, manifest, signature, validation, and recovery evidence.
|
|
|
|
Current and last-known-good versions MUST be activated through an atomic pointer/symlink or equivalent
|
|
transactional switch wherever supported. Mutable customer data is never stored inside versioned
|
|
application trees. Models may remain in a shared content-addressed store; activation metadata is
|
|
versioned and garbage collection cannot remove an object referenced by current, last-known-good,
|
|
backup, or active staging state.
|
|
|
|
Default retention is current plus one last-known-good version and the newest verified pre-update
|
|
backup. Additional history is optional and capacity-bound. Preflight computes staging, unpacked,
|
|
backup/snapshot, previous-version, model, and safety-headroom bytes separately. Insufficient capacity
|
|
blocks the update before mutation; the updater never deletes last-known-good recovery state to make
|
|
space automatically.
|
|
|
|
## Preflight and Checkpoint
|
|
|
|
Before mutation:
|
|
|
|
1. Complete trust, signature, object, compatibility, license, security-floor, and capacity checks.
|
|
2. Reject conflicting installation, update, restore, removal, model-import, or trust operations.
|
|
3. Inventory current source/build, application files, configs, services, packages, containers,
|
|
runtimes, schema, models, trust policy, and active hardware profile.
|
|
4. Verify the current state is healthy enough to distinguish a pre-existing fault from update impact.
|
|
5. Create the class-required checkpoint and verify readability, checksums, schema compatibility, and
|
|
restore instructions. A path existing is not verification.
|
|
6. Display service disruption, reboot requirements, estimated space/time, rollback class and limits,
|
|
retained/deleted state, and recovery owner before authorization.
|
|
|
|
## Activation Order
|
|
|
|
Default ordered phases are `received`, `verified`, `staged`, `prechange_saved`, `quiescing`,
|
|
`data_migrating`, `switching`, `services_starting`, `awaiting_reboot`, `validating`, and terminal
|
|
`succeeded`, `rolled_back`, `failed_safe`, or `recovery_required`.
|
|
|
|
- Stop writes or establish an application-consistent data checkpoint before schema/application switch.
|
|
- Apply idempotent migrations sequentially in one transaction where SQLite permits.
|
|
- Switch application/config/container/model references only after all target objects are local.
|
|
- Restart services in dependency order with bounded timeouts.
|
|
- A required reboot resumes the same operation ID from durable state.
|
|
- Do not garbage-collect staging or last-known-good content until validation and approval complete.
|
|
|
|
## Health Validation and Commit
|
|
|
|
The proposed default validation window is ten minutes after all required services become ready, plus
|
|
explicit checks for database/schema, portal/API, authentication, license/entitlement, storage,
|
|
selected local model inference, container/service restart count, and hardware runtime where relevant.
|
|
The window restarts after a required reboot or unexpected service restart.
|
|
|
|
Success requires every mandatory check to pass and the active inventory to match the signed target
|
|
manifest. Optional remote connectors may remain explicitly unavailable when no approved route exists.
|
|
Only then does target become current and former current become last-known-good.
|
|
|
|
## Rollback Decision
|
|
|
|
| Condition | Required action |
|
|
|---|---|
|
|
| Failure before active switch | Delete/quarantine staging; current remains active; `failed_safe` |
|
|
| File/runtime switch fails before data commit | Switch to last-known-good; restart/validate; `rolled_back` |
|
|
| Transactional migration fails | Roll back transaction; restore application pointer if changed |
|
|
| Post-commit schema/application validation fails | Stop writes; restore verified pre-update data plus last-known-good application as one recovery unit |
|
|
| Driver/kernel boot failure | Boot approved previous kernel/snapshot or enter documented local recovery; never loop indefinitely |
|
|
| Health window fails | Automatic rollback when class permits; otherwise `recovery_required` with checkpoint identity |
|
|
| Rollback validation fails | Preserve both failure causes and all evidence; stop in `recovery_required`; no repeated automatic loop |
|
|
|
|
Automatic rollback is attempted at most once per operation. A retry creates a linked operation and
|
|
requires renewed preflight. Rollback does not bypass the installed signed security floor.
|
|
|
|
## Interruption Matrix
|
|
|
|
- Before checkpoint: current remains active; restart verification/staging safely.
|
|
- During checkpoint: incomplete checkpoint is invalid and cannot authorize activation.
|
|
- During transactional migration: transaction rolls back; verify DB before retry.
|
|
- Between migration and application switch: resume declared order or restore checkpoint; never infer
|
|
completion from file presence alone.
|
|
- During pointer switch: reconcile signed inventory to exactly current or target, never a mixture.
|
|
- During service restart: resume validation against recorded active inventory.
|
|
- During reboot: boot marker resumes the same operation; bounded boot attempts trigger recovery.
|
|
- During rollback/restore: preserve original and rollback error; require operator recovery if the
|
|
verified checkpoint cannot be restored.
|
|
|
|
## Ownership and Approval
|
|
|
|
| Responsibility | Accountable owner |
|
|
|---|---|
|
|
| Bundle production, manifest and action ordering | Platform release owner |
|
|
| Signing, security floor, trust changes, emergency authorization | Security owner |
|
|
| Migration forward/restore compatibility | Data/backend owner |
|
|
| Driver/kernel/OS recovery | Platform delivery owner |
|
|
| Model/runtime compatibility and inference checks | AI platform owner |
|
|
| Customer disruption, support runbook, recovery exercise | Delivery/support owner |
|
|
| Release scope and controlled pilot | Product owner |
|
|
|
|
## Evidence Gate
|
|
|
|
Approval requires positive update and rollback from every supported previous baseline plus negative
|
|
tests for wrong product/tier/architecture, revoked key, stale policy, corrupted object/cache, low disk,
|
|
conflicting operation, checkpoint failure, migration failure, service failure, network removal,
|
|
power loss at every mutating boundary, reboot failure, health-window failure, automatic rollback,
|
|
restore-based rollback, and rollback failure. Evidence records both active and target inventories,
|
|
schema versions, checkpoint checksums, operation/correlation IDs, service/model checks, and final
|
|
recovery status.
|
|
|
|
## Owner Decisions Still Open
|
|
|
|
- Approve ten-minute health window or choose another measured value.
|
|
- Select the platform versioned-tree/atomic-switch mechanism and system snapshot mechanism.
|
|
- Define full backup inventory beyond `/opt/cezen/data`, encryption, external-copy policy, and restore SLA.
|
|
- Define supported previous-version window, update cadence, retention beyond current+one, and disk quota.
|
|
- Define kernel/driver rollback mechanism for each supported hardware/OS profile.
|
|
- Approve exceptional non-reversible/break-glass authority and customer confirmation language.
|