173 lines
12 KiB
Markdown
173 lines
12 KiB
Markdown
# Python Runtime Profile Selection Contract
|
||
|
||
The Offline Workstation AMD64 medium MAY carry both CPU-only and NVIDIA CUDA 12.8 wheel objects, but
|
||
installation MUST select exactly one platform runtime profile. Media universality does not authorize
|
||
installing both profiles.
|
||
|
||
## Selection
|
||
|
||
- `cpu-only`: selected when no supported NVIDIA GPU is accepted or when the operator explicitly
|
||
chooses CPU mode. It MUST use the PyTorch 2.8.0 CPU trio and MUST NOT install NVIDIA, CUDA, Triton or
|
||
bitsandbytes wheels.
|
||
- `nvidia-cu128`: selected only after supported NVIDIA hardware and driver readiness pass. It MUST use
|
||
the vLLM-0.10.2-aligned Torch 2.8.0/Torchaudio 2.8.0/Torchvision 0.23.0 CUDA 12.8 trio and MUST NOT
|
||
contain CUDA 13 objects.
|
||
- Selection is recorded in the installation journal before package mutation. Resume MUST reuse the
|
||
recorded profile; changing profile requires rollback to the pre-Python checkpoint and a new operation.
|
||
- Every selected wheel MUST exist in the reviewed payload lock and pass its SHA-256 check before it is
|
||
exposed to `pip`. Installation MUST use `--no-index` against a profile-specific wheel directory.
|
||
- Hardware detection failure defaults to `cpu-only`; it MUST NOT silently select NVIDIA.
|
||
|
||
The machine-readable candidate is `autoinstall/offline/python-runtime-profiles.json`; its current
|
||
digests are evidence inputs, not a production signature. `validate-python-runtime-profile.py` is the
|
||
fail-closed structural/digest guard. Release acceptance additionally requires signed profile membership,
|
||
license/notice approval and clean bare-metal interruption/resume tests.
|
||
|
||
## Selection Planner Boundary
|
||
|
||
`plan-python-runtime-profile.py` is a non-mutating planner. It requires an AMD64 hardware probe, an
|
||
operation ID, optional prior journal state and a membership-verification receipt. The receipt is trusted
|
||
only as the output of the future production signature verifier; the planner does not perform
|
||
cryptographic verification and a caller-supplied `signature_verified` boolean is not production proof.
|
||
|
||
Stable failures include `ARCHITECTURE_UNSUPPORTED`, `NVIDIA_PROFILE_NOT_READY`,
|
||
`MEMBERSHIP_RECEIPT_INVALID`, `MEMBERSHIP_UNVERIFIED`, `TEST_RECEIPT_FORBIDDEN`,
|
||
`PROFILE_NOT_IN_SIGNED_MEMBERSHIP`, `ACTIVE_OPERATION_EXISTS`,
|
||
`PROFILE_CHANGE_REQUIRES_ROLLBACK` and `NEW_OPERATION_REQUIRED`. Auto-selection chooses NVIDIA only
|
||
when presence, support and driver readiness are all explicitly true; missing or failed facts select
|
||
CPU. An existing non-rolled-back journal can only resume the same operation and profile. A rolled-back
|
||
operation requires a new operation ID.
|
||
|
||
The production verifier receipt MUST validate against
|
||
`schemas/profile-membership-verification-receipt.schema.json`. It binds the exact manifest and
|
||
signature-envelope digests, key identity/fingerprint, trust snapshot, policy, artifact namespace,
|
||
authorized profiles, verification time and verifier build. A test-environment receipt is rejected by
|
||
default and is accepted only through the planner's explicit structural-test switch.
|
||
|
||
## Atomic Journal Boundary
|
||
|
||
`atomic-profile-journal.py` persists the structural profile-operation journal under an exclusive lock.
|
||
Every update uses an expected generation, preserves operation/profile/manifest identity, links the prior
|
||
record digest, writes canonical JSON to a mode-`0600` temporary file, fsyncs it, atomically renames it,
|
||
then fsyncs the containing directory. Stale generations, altered records, identity changes and invalid
|
||
transitions fail closed.
|
||
|
||
The allowed path is `profile_selected -> checkpoint_ready -> installing -> installed`, with
|
||
`installing -> rollback_required -> rolled_back`; selection/checkpoint stages may also roll back.
|
||
`checkpoint_ready` and every mutating/later state require a durable checkpoint identity. Terminal
|
||
records cannot be reused. The writer records checkpoint identity but does not create or validate the
|
||
checkpoint itself.
|
||
|
||
## Profile Membership Verifier
|
||
|
||
`verify-python-profile-membership.py` canonicalizes the profile manifest, verifies its SHA-256 against
|
||
the detached Ed25519 envelope, enforces policy-version equality, resolves exactly one active trust key,
|
||
checks algorithm/purpose/environment, verifies the public-key DER fingerprint, and performs Ed25519
|
||
verification through OpenSSL. Only then does it emit the closed receipt consumed by the planner.
|
||
|
||
The profile manifest namespace is fixed as `offline-workstation/python-runtime-profiles`. Unknown or
|
||
ambiguous keys, wrong purpose/environment/status, fingerprint mismatch, policy mismatch, digest
|
||
mismatch, malformed signature and invalid signature fail closed. OpenSSL and the verifier executable
|
||
are themselves release inputs and MUST be locked/measured in the final payload.
|
||
|
||
## Python Environment Checkpoint
|
||
|
||
`python-environment-checkpoint.py` snapshots an existing environment directory, or records that the
|
||
target was absent, before mutation. Its manifest binds operation ID, exact target path, original
|
||
existence and a sorted inventory of directories, files (mode/bytes/SHA-256) and symlinks (mode/target).
|
||
Creation copies into a same-parent temporary directory, fsyncs files/directories, then atomically
|
||
renames and fsyncs the parent. Verification recomputes the complete inventory before restore.
|
||
|
||
Restore verifies the checkpoint first, quarantines the failed/new target rather than deleting it,
|
||
copies the verified payload to a same-filesystem temporary directory, fsyncs it and atomically replaces
|
||
the target. Unsupported filesystem entries, target mismatch, manifest/payload tamper, pre-existing
|
||
checkpoint/quarantine and restore ambiguity fail closed.
|
||
|
||
Test-only fault injection is disabled unless both `NEXUS_ALLOW_TEST_FAULTS=1` and an explicit crash
|
||
point are set. Boundaries exist before/after checkpoint publication and after failed-target quarantine
|
||
or restored-target activation. `recover-restore` completes only from preserved quarantine or accepts an
|
||
already restored target whose complete inventory matches the checkpoint.
|
||
|
||
## Pre-Mutation Orchestration
|
||
|
||
`prepare-python-profile-operation.py` composes the receipt boundary, selection planner, exact profile
|
||
payload validator, atomic journal and environment checkpoint. It validates membership and wheel bytes
|
||
before creating a journal, records `profile_selected`, creates or verifies the checkpoint, then records
|
||
`checkpoint_ready`. It deliberately has no package-install action and reports
|
||
`mutation_performed: false`. Re-entry at `checkpoint_ready` is idempotent.
|
||
|
||
## Disposable Offline Installation Executor
|
||
|
||
`install-python-profile.py` requires a verified `checkpoint_ready` journal and matching checkpoint,
|
||
records `installing`, creates a new environment beside the target, installs only with `--no-index` and
|
||
the selected wheel directory, runs profile-specific health checks, fsyncs the complete new tree,
|
||
atomically activates it, fsyncs the parent, repeats health validation and records `installed`.
|
||
|
||
An optional service-health probe runs after built-in post-activation runtime health and before the
|
||
`installed` journal transition. The probe path MUST be absolute, a non-symlink regular executable and
|
||
a locked/signed release input; it is invoked directly without a shell as `probe target profile` under
|
||
a bounded 1–3600 second timeout. Non-zero exit, timeout or execution failure triggers the same verified
|
||
rollback path as any other post-activation failure. Production integration must define probe identity,
|
||
least privilege, environment, output capture, service dependencies and the accepted consecutive-health
|
||
window; an arbitrary operator-supplied executable is not release evidence.
|
||
|
||
### Service dependency and health-window contract
|
||
|
||
The production activation graph MUST explicitly declare the selected model runtime and vector store
|
||
as API dependencies, and the portal/reverse proxy as an API dependent. Ordering alone is insufficient:
|
||
required components use both failure propagation and ordering semantics (`Requires` plus `After`, or
|
||
an approved equivalent). Optional/category-disabled components MUST be omitted by the resolved profile
|
||
rather than represented as permanently failing required units.
|
||
|
||
Activation acceptance requires at least three consecutive complete graph samples across model runtime,
|
||
vector store, API and portal/proxy, with the release-approved interval and total timeout. Every sample
|
||
must validate the component-specific semantic response, not merely an open TCP port. Any failed sample
|
||
before commit fails the window and triggers rollback from the last verified journal/checkpoint. Probe
|
||
logs bind the service identities, dependency graph, sample number, response outcome and rollback result.
|
||
|
||
The current checked-in `cezen-api.service` is not compliant evidence: it has `After=... ollama.service`
|
||
but no `Requires=ollama.service`, does not declare ChromaDB ordering/dependency and does not establish
|
||
the portal/proxy dependency or consecutive semantic-health window. Production units must be remediated
|
||
and tested separately; isolated mock-unit evidence cannot approve the real graph.
|
||
|
||
CPU mode removes ChromaDB (separate environment) and bitsandbytes (GPU-only) from the platform input.
|
||
On failure it records `rollback_required`, verifies/restores the checkpoint as applicable, then records
|
||
`rolled_back`. The prior environment and failed activated environment are quarantined, not deleted.
|
||
|
||
Test-only executor fault injection can terminate immediately after activation and before post-health or
|
||
the `installed` journal write. `--recover-only` accepts only `installing` or `rollback_required`, moves
|
||
to `rollback_required` when necessary, restores or recognizes the verified checkpoint state, preserves
|
||
the failed activated environment, then records terminal `rolled_back`.
|
||
|
||
NVIDIA health requires Torch 2.8.0 with compiled CUDA 12.8, `torch.cuda.is_available()`, successful
|
||
device tensor execution and synchronization, plus imports for the shared ML stack and bitsandbytes.
|
||
Merely importing CUDA-bearing wheels does not satisfy NVIDIA readiness.
|
||
|
||
## Boot Recovery Dispatcher
|
||
|
||
`recover-python-profile-on-boot.py` is the boot-service-safe dispatcher for an existing profile
|
||
operation. It performs rollback recovery only when the verified journal is in a mutating interrupted
|
||
state: `installing` or `rollback_required`. An absent journal, terminal `installed`/`rolled_back`
|
||
journal, or non-mutating `profile_selected`/`checkpoint_ready` journal is an explicit no-op. Unknown,
|
||
invalid or tampered journal state fails closed.
|
||
|
||
The dispatcher does not select a profile, create a checkpoint, install packages or infer success. It
|
||
must run with access to the exact journal, checkpoint and target paths recorded for the operation.
|
||
Production boot integration must order it before consumers of the target environment and must retain
|
||
its output in the installation evidence journal. Process/container restart evidence does not satisfy
|
||
the separate physical reboot or sudden-power-loss acceptance gates.
|
||
|
||
## Capacity-Exhaustion Boundary
|
||
|
||
The operation journal, its lock, checkpoint metadata/payload and recovery evidence MUST have reserved
|
||
capacity independent of the mutable target environment. Target byte or inode exhaustion must not
|
||
prevent the executor from recording `rollback_required` and terminal `rolled_back`. A release must
|
||
preflight both free bytes and free inodes for the selected profile, staging tree, activated target,
|
||
checkpoint, quarantine and journal reserve; byte-only checks are insufficient.
|
||
|
||
`ENOSPC` before activation MUST leave the original target unchanged, remove incomplete temporary
|
||
installation state when possible and record terminal rollback. If reserved journal/checkpoint capacity
|
||
is itself exhausted, the operation MUST stop fail-closed and recovery must restore that reserve before
|
||
any retry; it MUST NOT infer success or discard the last verified journal/checkpoint. Disposable tmpfs
|
||
evidence does not establish production filesystem quotas, live-path metadata or service recovery.
|