Files

778 lines
43 KiB
Markdown

# Offline Workstation Gate 2 Build-Host Handoff
**Date**: 2026-07-14
**Host**: `172.16.10.144` (`acc-srv`)
**Workspace**: `/mnt/nvme0/nexus-one-ai-offline-workstation/`
**Scope**: Gate 2 preparation; no new build or ISO claim.
## Prior Run Disposition
No matching build, bundle, ISO, squashfs, container-pull or model-pull process was running when
inspected. The prior workspace was preserved rather than deleted:
`/mnt/nvme0/nexus-one-ai-offline-workstation-legacy-20260714/`
The legacy payload is approximately 30 GB and is not trusted as Gate 2 input. It contains mutable
`latest`/`main` container exports, an unpinned Miniconda installer, mixed CUDA assumptions, partial
root-owned APT state and a source copy without Git metadata. It remains quarantine/reference evidence
only and MUST NOT be merged into a candidate payload by filename.
## Clean Workspace
Created with mode `0750` and separate `source`, `staging`, `artifacts`, `logs`, and `evidence`
directories. The current local source/spec snapshot was synchronized while excluding `.git`, ISO
files, caches, virtual environments, `node_modules`, and unrelated `how-it-flows.gif`.
- Local committed base at synchronization: `bea082a2014a3edc31cb006108b952c8adbbc8bc`
- Synchronized files: 219
- Remote source checksum-index digest:
`a9a0150f656e142c32e3962058ed83ef38eaa0685ff6c233e41d2896dbb84389`
- Checksum index: `/mnt/nvme0/nexus-one-ai-offline-workstation/evidence/source-sha256sums.txt`
- Exclusion check: pass; no `.git`, `*.iso`, or `how-it-flows.gif` found in synchronized source.
The snapshot includes local uncommitted specification work. Therefore the committed base alone is not
its identity; the checksum index is the authoritative transferred-source identity for this handoff.
## Host Baseline
- Ubuntu 22.04 AMD64
- Kernel `6.8.0-124-generic`
- `/mnt/nvme0`: 596 GB total, 567 GB available at inspection
- Available: Docker, xorriso, mksquashfs, jq, Python 3
- Not present: podman, skopeo, crane, oras
No build was started during this handoff. Gate 2 must first resolve immutable package/image/runtime/
model inventory and correct C1-C4/C7. Starting the legacy bundle builder unchanged would violate the
approved architecture.
## Immutable-Lock Preflight Increment
The synchronized source now includes `validate-payload-lock.py`, an intentionally unresolved
`payload-lock.candidate.json`, standard-library tests, and a hard preflight at the start of
`build-bundle.sh`. The legacy mutable acquisition body is unreachable even after a lock passes until
lock-driven acquisition is implemented.
Local evidence:
- Three `unittest` cases pass: unresolved candidate rejection, structurally complete test-lock
acceptance, and mutable OCI/duplicate-target rejection.
- `bash -n autoinstall/offline/build-bundle.sh`: pass.
- Candidate validation: expected fail with unresolved/status/object-group reasons.
Build-host evidence:
- Preflight exit: `1` (expected fail closed).
- Artifact/staging entry count before: `0`; after: `0`.
- No acquisition, output deletion, payload assembly or ISO build occurred.
- Logs: `/mnt/nvme0/nexus-one-ai-offline-workstation/logs/gate2-preflight.stdout` and
`gate2-preflight.stderr`.
## Resolver Increment 001
The connected resolver created a deterministic source archive and resolved six Linux/AMD64 OCI
manifests without pulling image layers or creating an artifact.
- Candidate lock: `/mnt/nvme0/nexus-one-ai-offline-workstation/evidence/payload-lock.resolution-001.json`
- Candidate lock SHA-256:
`146719ad9de2c99f402dfb15e16c36a029328219e53cc3f2c8ca835d4d01e51d`
- Source archive SHA-256:
`6c890fc6537f2ed51cc40a9930a1bad1d3db78c31b093259fb81f5638cf5aade`
- Source archive reproducibility: pass; two independent archives are byte-identical.
- OCI manifests resolved: 6; artifact directory remains empty.
- Unresolved entries after this increment: 9.
The result correctly remains a candidate. Open WebUI returned no direct config/layer byte total in the
selected Docker manifest response, two container licenses remain review-required, mutable source tags
must be replaced with approved release tags, and exported OCI archive bytes/digests require actual
digest-pinned acquisition. Registry resolution also demonstrated drift: current resolved digests differ
from the quarantined run for mutable/tag-based inputs. The quarantined digests are evidence of drift,
not approved pins.
APT closure, Python closure, Ubuntu media, Miniconda/replacement runtime, Ollama runtime and exact
model files/notices remain unresolved. No payload or ISO build has started.
## APT Resolver Increment 001
The APT resolver used an empty dpkg status with authenticated package acquisition and
`--no-install-recommends`. This prevents already-installed build-host packages from disappearing from
the computed clean-system closure. It wrote downloads only under Gate 2 staging and installed nothing.
- Requested packages: 26
- Resolved `.deb` objects: 487
- Measured bytes: 357,993,500
- Candidate lock: `/mnt/nvme0/nexus-one-ai-offline-workstation/evidence/payload-lock.resolution-002.json`
- Candidate lock SHA-256:
`351c842ab92995315b008865a3080a5b1469f002cdc235702fff28cdf7bcf518`
- Artifact directory: empty
- Candidate validator: expected fail; Python/runtime/model groups, Open WebUI measured bytes, reviewed
status and unresolved list remain incomplete.
This closes only the measured Ubuntu/deadsnakes `.deb` byte closure for the currently requested list.
The APT repository index/origin snapshot and its digests, package license/notices, and the separately
approved R570/CUDA 12.8.1/cuDNN/NVIDIA Container Toolkit closure remain unresolved. Current repository
URLs are not yet accepted as reproducible release identity.
## Python Resolver Increment 001 — Boundary Correction and Candidate Closure
The first Python 3.11 wheel-only run combined backend and platform requirements and correctly failed
dependency solving:
- Backend constraint: `bcrypt<4.0.0`.
- Platform constraint: `chromadb==0.5.23`, whose declared dependency is `bcrypt>=4.0.1`.
- Resolver result: `ResolutionImpossible`; exit `1`.
- Candidate `payload-lock.resolution-003.json`: absent.
- Downloaded wheel files in that run: `0`.
- Artifact directory entries: `0`.
- Failure log: `/mnt/nvme0/nexus-one-ai-offline-workstation/logs/python-resolution-001.stderr.log`.
- Failure-log SHA-256:
`28fe44f5cab1d41d5ebf92322819b006c92eb13df79879b579e93260388f9b61`.
Repository inspection then proved those inputs belong to separate runtime boundaries: the backend and
Chroma Ansible roles create different virtual environments, and the legacy bundle layout already has
separate `backend`, `chromadb` and `platform` wheelhouses. The conflict was therefore a resolver-boundary
defect, not an application dependency contradiction. Neither dependency constraint was changed.
The corrected resolver produced three independent closures:
- Backend: 61 wheel objects.
- ChromaDB: 88 wheel objects.
- Platform: 214 wheel objects.
- Total: 363 wheel objects; 3,238,751,953 measured bytes.
- License fields requiring review: 49.
- Candidate lock: `/mnt/nvme0/nexus-one-ai-offline-workstation/evidence/payload-lock.resolution-003.json`.
- Candidate lock SHA-256:
`cde0f17371dde68af984511114a2f2fca449f501ecbbdd622f48c92c6508b261`.
- Successful resolver log SHA-256:
`7d6b49f9f985b47190fd7b6c0fc6f00c72b80938e1905fed3b4a4524791c82cc`.
- Artifact directory entries: `0`.
- Candidate validator: expected fail because the lock is not reviewed, has 14 unresolved entries,
and still lacks complete runtime/model objects and measured Open WebUI bytes.
The closure is evidence, not an approved payload. The unpinned platform requirements selected current
Torch/CUDA dependencies including four explicitly CUDA-13-named objects, which conflicts with the
approved CUDA 12.8.1 baseline. Gate 2 remains blocked on an explicitly pinned CUDA-12.8-compatible
Python ML stack, index snapshot/origin and resolver-report approval, license/notice review, and clean
Python 3.11 offline installation/import evidence for each independent environment.
## Python Resolver Increment 002 — CUDA 12.8-Compatible Candidate
Authoritative vLLM 0.10.2 wheel metadata pins `torch==2.8.0`, `torchaudio==2.8.0` and
`torchvision==0.23.0`. The inspected AMD64 vLLM wheel SHA-256 is
`e0cba6110483d9bf25c4402d8655cf78d366dd13e4155210980cc3480ed98b7b`, matching the
published PyPI file digest. Those three versions were added to the platform input and resolved in a
fresh staging directory; the previous CUDA 13 candidate remains preserved for comparison.
- Backend: 61 wheel objects.
- ChromaDB: 88 wheel objects.
- Platform: 212 wheel objects.
- Total: 361 wheel objects; 4,435,239,499 measured bytes.
- Resolved Torch/Torchaudio/Torchvision: `2.8.0` / `2.8.0` / `0.23.0`.
- Explicit CUDA 13 objects: 0.
- Explicit CUDA 12 objects: 14.
- License fields requiring review: 49.
- Candidate lock: `/mnt/nvme0/nexus-one-ai-offline-workstation/evidence/payload-lock.resolution-004.json`.
- Candidate lock SHA-256:
`3b1fef9705e8049b28d283383f16fa7fd66e7fc00434315b2cc86163ba8e93ac`.
- Successful resolver log SHA-256:
`37711950c25a1b2409e1323ea35ce7f6cdab2f0a9e6d2aad9407baeda1325c35`.
- Artifact directory entries: 0.
- Candidate validator: expected fail because review/unresolved/runtime/model/Open WebUI evidence is
still incomplete.
This closes the version-selection contradiction only. It does not yet prove execution against the
approved R570/CUDA 12.8.1 physical profiles, clean disconnected Python 3.11 installation/import,
license/notice acceptance, or NVIDIA-independent CPU operation. The universal Workstation installer
also needs a profile-aware installation contract so CPU-only systems do not depend on NVIDIA packages
merely because the signed universal media carries them.
## Python Validation Increment 001 — Disconnected Install and GPU-Lite Smoke
Validation used official `python:3.11.13-slim-bookworm` on Linux/AMD64, pinned after acquisition to
`python@sha256:86adf8dbadc3d6e82ee5dd2c74bec2e1c2467cdad47886280501df722372d2e1`.
Each installation ran in a disposable container with `--network none`, read-only wheel mounts and
`pip --no-index --find-links`; no package index or other network fallback was possible.
- Backend wheelhouse: pass. Imports succeeded and Passlib/bcrypt hash/verify succeeded.
- ChromaDB wheelhouse: pass; `chromadb==0.5.23` imported successfully.
- Platform wheelhouse CPU path: pass. Torch/Torchaudio/Torchvision reported `2.8.0+cu128` /
`2.8.0+cu128` / `0.23.0+cu128`; `torch.cuda.is_available()` was false with CUDA hidden and CPU tensor
execution returned the expected result.
- Supplemental `gpu-lite` path: pass on RTX 3050 OEM with 8,222,670,848 bytes reported device memory.
Torch reported CUDA 12.8 and completed synchronized GPU tensor execution.
- Host driver for that supplemental run: NVIDIA Open Kernel Module `595.58.03`, license `Dual MIT/GPL`.
- Artifact directory entries: 0.
Evidence log SHA-256 values:
- Backend: `1d9be80fef4c565e13a9b12d00cc76a049441eb89d2136cb0d9e7703494a490a`.
- ChromaDB: `6b4a907b6cee363a1260be8d70789cf7ab1bb38702917fb7b4e0ee60132d4982`.
- Platform CPU: `35eb2e5861e98158e934771d1256567e3b68d23adb51a7fb6206a5af71853719`.
- GPU-lite: `b51584484bf0317f8a79058f91b5613dd1b0f7a7d17e07856a43811652721fcb`.
This proves disconnected wheel availability/importability and one supplemental 8 GB GPU smoke test.
It does not prove a clean bare-metal install, the exact R570 reference driver, 24 GB/48 GB profiles,
vLLM model serving, interruption/recovery, or selective CPU-profile installation without NVIDIA
packages. Those claims remain blocked, as do index provenance approval and 49 license/notice reviews.
## Python Profile Selection Increment 001 — Strict CPU Boundary
The universal-media/profile-selective contract now defines exactly-one selection between `cpu-only`
and `nvidia-cu128`. PyTorch's official 2.8.0 index matrix supplies distinct CPU and CUDA 12.8 builds.
The CPU trio was acquired from the CPU index into staging, then combined with the reusable non-GPU
platform wheels while excluding NVIDIA, CUDA, Triton and bitsandbytes objects.
- CPU selected wheels: 196; measured bytes: 385,850,558.
- Forbidden wheel count: 0.
- CPU Torch SHA-256: `cb06175284673a581dd91fb1965662ae4ecaba6e5c357aa0ea7bb8b84b6b7eeb`.
- CPU Torchaudio SHA-256: `e54bd7fc9472019308097d99102df9acee22aa2451ae808d27840bc874320292`.
- CPU Torchvision SHA-256: `d83d8075db43b8ca89680bdeb2f100c832e2a3aa61ee42c038b1a146e5e511b6`.
- Network-disabled full CPU-profile install/import: pass; Torch `2.8.0+cpu`,
`torch.version.cuda is None`, CPU tensor result correct.
- Strict CPU validation log SHA-256:
`57e2ab86f6fae188502c019429b37ab03ed2f8bad8fdd1330ec39fd17f69173e`.
- Machine-readable profile validator: pass, 196 wheels; log SHA-256:
`2f937161184294ea8c8e2413282db006498aca52517927f305fd3371e3defe14`.
- Artifact directory entries: 0.
The contract requires hardware-detection failure to default to CPU, records selection before mutation,
reuses it on resume, and requires rollback before profile change. This proves the Python payload
boundary, not the eventual installer's hardware detector, journal, rollback or signed membership.
## Python Profile Planner Increment 001 — Non-Mutating State Machine
The new planner consumes a normalized hardware probe, requested mode, operation ID, optional prior
journal and a profile-membership verification receipt. It does not mutate the host. Auto-selection
chooses NVIDIA only when presence, support and driver readiness are all explicitly true; incomplete or
failed facts choose CPU. An active operation can only resume the same profile and operation ID. Profile
change requires rollback, and a rolled-back operation ID cannot be reused.
- Dependency-free planner tests: 6 pass; combined deployment tests: 11 pass.
- GPU-lite structural start vector: selected `nvidia-cu128`; pass.
- Resume vector: retained the journaled `nvidia-cu128` selection; pass.
- Mid-operation CPU change vector: fail closed with `PROFILE_CHANGE_REQUIRES_ROLLBACK`; pass.
- Planner-vector log SHA-256:
`a5d56b4ec355ac701ec3cd72f05298dc6cb97bd49b1c20fdc3fc548a994635f5`.
- Artifact directory entries: 0.
The membership receipt used here is explicitly marked structural-test-only. The planner trusts a future
signature verifier's receipt; it does not verify cryptography. Production integration remains blocked
until real signature verification, atomic durable journal writes, pre-mutation checkpoint creation,
installer wiring and power-loss/rollback tests exist.
## Profile Verification Receipt Increment 001
A closed Draft 2020-12 receipt schema now binds verified profile membership to the exact manifest and
envelope digests, key ID/fingerprint, trust snapshot, policy version, namespace, profiles, verification
time and verifier build. The planner no longer accepts a bare verification boolean. It requires the
receipt schema/result/namespace and rejects non-production receipts unless explicit test mode is set.
- Combined deployment tests: 12 pass, including production rejection/explicit test acceptance.
- Existing schema/signature/compatibility fixture suite: pass in a network-disabled Python 3.11
container using staged wheels.
- Fixture-suite log SHA-256:
`6008edf74f86f181d84221ca49887751e047af87069ce9e451ecd336a7a40109`.
- Production-mode structural test receipt: rejected with `TEST_RECEIPT_FORBIDDEN`; expected.
- Explicit structural-test mode using the same receipt: accepted; expected.
- Receipt-boundary log SHA-256:
`aa2369d4058fef580ef55f9cb8e7b8ced823b551503cc39a8fad9fca311b5191`.
- Artifact directory entries: 0.
This defines and tests the verifier/planner handoff but does not implement the production verifier,
trust-policy evaluation or production key ceremony. Atomic journal/checkpoint persistence and
installer integration remain open.
## Atomic Profile Journal Increment 001
The structural journal writer now uses exclusive locking, expected-generation compare-and-swap,
canonical per-record digests, prior-record linkage, immutable operation/profile/manifest identity,
mode `0600`, file fsync, atomic rename and directory fsync. Installation cannot enter `installing`
before a `checkpoint_ready` record with a checkpoint identity exists.
- Atomic journal unit tests: 5 pass; combined deployment tests: 17 pass.
- Build-host generations 1 `profile_selected`, 2 `checkpoint_ready`, and 3 `installing`: pass.
- Stale generation-1 writer after generation 2: `JOURNAL_GENERATION_STALE`; expected.
- Persisted journal mode: `0600`.
- Final journal SHA-256:
`35b3940893a27c4a09ac3d532fe992d23ec59396fa8f6883fc809f2df3803488`.
- Transition-vector log SHA-256:
`48d2aabf56b882975ed539007fe3b0ef9f6cb4674c2cd68affdc7646214adbb1`.
- Artifact directory entries: 0.
Tests also cover digest tampering, identity change, missing checkpoint, invalid transition and terminal
rollback. The writer does not create/verify a real filesystem/package checkpoint, and no forced
power-loss test has yet proved filesystem-specific durability. Production signature verification,
checkpoint implementation, crash injection and installer integration remain open.
## Ed25519 Profile Verifier Increment 001
The production-shaped verifier now canonicalizes the profile manifest, checks namespace and digest,
enforces envelope algorithm/policy, resolves one active purpose/environment-scoped trust key, checks
the public-key DER fingerprint, verifies Ed25519 with OpenSSL and emits the closed membership receipt.
- Ubuntu/OpenSSL 3 combined deployment tests: 19 pass.
- Positive Ed25519 path used an ephemeral test key generated only inside a disposable test directory.
- Negative paths include manifest tampering and wrong key environment.
- Verifier-test log SHA-256:
`b61d719273b7e755fd76aff8603f81e62b0449fde813eb01f7e2b0850633319c`.
- Artifact directory entries: 0.
The Mac OpenSSL build lacks Ed25519 and correctly skips the two crypto tests; authoritative crypto
evidence is the Ubuntu/OpenSSL 3 run. No private test key entered the repository or logs. This is not a
production ceremony or release signature: production public keys, trust snapshot, revocation/time
policy, locked OpenSSL/verifier build and independent signing approval remain absent.
## Python Environment Checkpoint Increment 001
The checkpoint component now records whether the target existed and inventories directory, file and
symlink state with modes, byte sizes, hashes and link targets. Creation and restore use same-filesystem
temporary trees, file/directory fsync and atomic rename. Restore verifies the checkpoint before use and
quarantines the failed/new tree rather than deleting it.
- Checkpoint unit tests: 3 pass; combined local suite: 20 run, 19 pass and the unsupported Mac
Ed25519 test class skips.
- Build-host create/verify/modify/restore vector: pass; original file SHA-256 restored exactly.
- Checkpoint inventory SHA-256:
`488bfb0e7f67d75553b66ccbf981cb9139ef1b8610f1018d76e4832a5c748787`.
- Checkpoint manifest file SHA-256:
`82f28057d73359d9b7d6fd75b23f9ff4718def883693104b88161001b16dfeab`.
- Vector log SHA-256:
`cbbf47fd1d028635e2b2dce63c115a9066cff8df3e0fa0458d28f8691082df1d`.
- Artifact directory entries: 0.
This validates ordinary checkpoint/restore and tamper rejection in disposable staging. It does not yet
prove crash consistency under forced termination at each fsync/rename boundary, real `/opt/cezen`
ownership/ACL/xattr preservation, disk-full behavior or integration with journal/install mutation.
## Python Checkpoint Crash-Recovery Increment 001
Test-only process termination was injected at four fsync/rename boundaries on disposable build-host
trees. Every injected process exited `97` as designed:
- Before checkpoint publication: no checkpoint became visible; the original target remained authoritative.
- After checkpoint publication: the visible checkpoint passed complete verification.
- After failed-target quarantine but before activation: recovery completed verified restore and
preserved the failed tree.
- After restore activation but before return: recovery recognized the exact restored inventory and
completed idempotently.
Crash-vector log SHA-256:
`17d0cb3f39b6c3f7114e2c81acc4e8884f96a496aa06a7788a9b380f801dc6f9`.
Artifact directory entries remained 0.
This is forced-process termination, not physical power removal. Filesystem/controller cache loss,
disk-full/inode exhaustion, ACL/xattr preservation, journal/checkpoint orchestration and bare-metal
recovery remain open.
## Profile Pre-Mutation Orchestration Increment 001
The isolated orchestrator now composes structural receipt acceptance, profile planning, exact wheel
validation, atomic journal selection and verified environment checkpoint. Invalid payload membership
creates neither journal nor checkpoint. Valid preparation stops at `checkpoint_ready` and contains no
package mutation action.
- Orchestrator unit tests: 2 pass; combined local suite: 23 run, 22 pass plus the expected Mac
Ed25519 capability skip.
- Build-host strict CPU preparation: pass, generation 2, `checkpoint_ready`.
- Immediate rerun: same checkpoint/generation, pass; `mutation_performed: false` both times.
- Checkpoint inventory ID:
`12e8f917f8a388c0a1d9b45c5f494ce759667f331dc0bdfd03f2b51bda2c9d3a`.
- Journal SHA-256:
`1fd29a06646ab917a01937bb498945e1b8f347b1c731d78e1d825f0f521d3e60`.
- Checkpoint manifest SHA-256:
`99d10d9ac0e48da10b70ccfff08d64af4222d6637d4e2ad0a1b22e1b41134ed1`.
- Evidence log SHA-256:
`ed3209620640105cc15373fc9759b1198af2a1d6713e4f0d76d000e1a439fcde`.
- Artifact directory entries: 0.
This closes component-composition only through the pre-mutation boundary. Actual offline installation,
health validation, installed/rollback journal transitions, service integration and production receipt
inputs remain blocked.
## Disposable CPU Installation Increment 001
The offline executor was run entirely inside a network-disabled Python 3.11 container against a new
container-mounted staging target. It consumed the strict 196-wheel CPU selection, not the CUDA-bearing
platform set. The final-code rerun included full new-tree and parent-directory fsync barriers.
- Offline dependency-failure unit vector: original target unchanged; journal `rolled_back`; pass.
- Positive prepare/checkpoint/install/pre-activation-health/atomic-activation/post-health flow: pass.
- Final journal: generation 4, `installed`, mode `0600`.
- Installed Torch: `2.8.0+cpu`; compiled CUDA version: none.
- Journal SHA-256:
`61e77937fadaee985e568a6f78e4a5118ac5198f2f5b692a14cce6c1b1997ca5`.
- Checkpoint manifest SHA-256:
`0ca232445458f2a48bd3319d4e56964b92e5d11fee8b9b92938b1517af157a73`.
- Installation log SHA-256:
`941fa474229ddcb8a4e23bdc95cba27ef51390580c8f278c914332b1e47c8195`.
- Artifact directory entries: 0.
This proves the CPU Python environment mutation/health path only in disposable container staging. It
does not prove service wiring, system ownership/ACLs, live `/opt/cezen`, reboot resume, NVIDIA install,
post-activation forced failure, disk exhaustion or bare-metal installation.
## Post-Activation CPU Rollback Increment 001
The full strict-CPU environment was installed in disconnected disposable staging, fsynced and atomically
activated. Test-only termination then exited `97` immediately after activation and before post-health/
`installed`. Recovery consumed the generation-3 `installing` journal, recorded `rollback_required`,
restored the verified pre-install checkpoint, preserved the activated environment in quarantine and
recorded generation 5 `rolled_back`.
- Original pre-install marker restored: pass.
- Failed activated environment executable present in quarantine: pass.
- Final journal SHA-256:
`6acdfd8c8a57586ad0edae5e5a4cae0bac0a6d1055864dd35113afb8199dabad`.
- Checkpoint manifest SHA-256:
`d5c5ff6fd95743431832d115a0b4fe7de8e9ae0fa7ee36b3c349316cb9ebfc53`.
- Crash/rollback log SHA-256:
`5791738b02f6c0a41218a94bd04e6a1cbd80dd42b04e28a14f55ab451c61f15c`.
- Artifact directory entries: 0.
This closes the disposable CPU post-activation process-crash case. It does not prove reboot-triggered
recovery, physical power loss, service health rollback, disk exhaustion, NVIDIA rollback or live paths.
## Disposable NVIDIA Installation Increment 001
The full 212-wheel `nvidia-cu128` platform selection was installed inside a network-disabled Python
3.11 container with access to the build-host RTX 3050. Preparation used exact profile validation and
checkpointing; install used no index, pre/post activation health, full-tree fsync and atomic activation.
- Final journal: generation 4, `installed`, mode `0600`.
- Torch: `2.8.0+cu128`; compiled CUDA: 12.8.
- GPU: NVIDIA GeForce RTX 3050 OEM; device tensor execution and synchronization: pass.
- Journal SHA-256:
`92cfa8a9cab9de57c08019957c564394c51eab6257dda41be2e9d352b2a422b6`.
- Checkpoint manifest SHA-256:
`66c9c43fd432b8aa87d9212e9cbf51619cd38db1c5c7768a0102afaf96431ee7`.
- Installation log SHA-256:
`af29e27e5cf3e55d01874bc00a895b1ce2b6e96f50e347c5890b6b70f969dd36`.
- Artifact directory entries: 0.
This is supplemental 8 GB `gpu-lite` evidence on Open Driver 595.58.03. It does not replace the exact
R570 reference, 24/48 GB profiles, vLLM model-serving, NVIDIA rollback/reboot or bare-metal proof.
## Post-Limit Build-Host Refresh
After the external-action quota reset, the complete current source/spec snapshot was synchronized to
the clean build-host workspace. The full remote deployment suite ran under Ubuntu/OpenSSL 3:
- Tests: 27 passed; 0 skipped; exit 0.
- Suite log SHA-256:
`f9b4e43c8bd587a4c92bc64a4744d61b3b879e0408ea57056380136d35f8517c`.
- Artifact directory entries: 0.
The expected offline-resolution negative vector appears in the log and is followed by an overall pass.
This refresh aligns the build-host copy with T156 and the latest local implementation; it does not
change Gate 2 readiness or authorize assembly.
## Post-Activation NVIDIA Rollback Increment 001
The full `nvidia-cu128` environment was installed without network access, health-checked, fsynced and
atomically activated on the RTX 3050. Test-only termination exited `97` after activation and before
the final health/journal commit. Recovery restored the verified pre-install checkpoint, preserved the
activated CUDA environment in quarantine and recorded generation 5 `rolled_back`.
- Original pre-install marker restored: pass.
- Quarantined environment remained executable and reported Torch `2.8.0+cu128`: pass.
- Final journal SHA-256:
`306b3873e37b276d45af2446cdd25a9a56d4545e0a4495d327578028b84ac272`.
- Checkpoint manifest SHA-256:
`ca88bf52a19c22b90b5a445985e60faca177ad21dc67efdd2d3572e4378d7af3`.
- Crash/rollback log SHA-256:
`e0b01f43762fc518ea26557ec58b7b4c1bc6cff366890154a972df96d391711f`.
- Artifact directory entries: 0.
This closes disposable post-activation process-crash rollback for the 8 GB `gpu-lite` Python profile.
It does not prove reboot/power-loss recovery, exact R570, 24/48 GB profiles, vLLM service/model health
or bare-metal NVIDIA installation.
## Boot-Dispatcher CPU Recovery Increment 001
A fresh strict-CPU operation ran in a network-disabled container and was forcibly terminated with exit
`97` after atomic activation, leaving the verified journal at generation 3 `installing`. A second,
separately created network-disabled container invoked `recover-python-profile-on-boot.py`, simulating
a fresh privileged boot-service process rather than an in-process exception handler.
- Pre-recovery journal: generation 3, `installing`.
- Post-recovery journal: generation 5, `rolled_back`.
- Original marker restored: pass.
- Activated environment preserved with executable Python in quarantine: pass.
- Second dispatcher invocation on terminal state: safe no-op.
- Journal SHA-256:
`f5c60f693fd9ecf572e35d58390c596a71916cc1dac4f44a698aa67c81d1c0d9`.
- Checkpoint manifest SHA-256:
`e6c0ed458c8ed785a0d611fb4b6da0353b9632e008fbee2060c6b8e151b500c0`.
- Combined crash/recovery log SHA-256:
`ab9d9a23b9c946a9ee7b86440584f06038988d09a701ef7d8bd10cc5ae2f7944`.
- Artifact directory entries: 0.
This proves recovery across distinct disposable process/container lifetimes. It does **not** prove an
actual OS reboot, abrupt physical power loss, production boot ordering, live path/ownership/ACL/xattr
behavior or service-level health recovery; those evidence gates remain open.
After T159 documentation and tests were synchronized, the complete remote deployment suite passed
29 tests with no skips or failures. The suite log SHA-256 is
`0060f0cc33d37e2d35165908109e76e5cc8e9909bd2a40dbd989fe1baac25da5`; the artifact directory remained
empty. The expected impossible-package negative-vector output is followed by the overall `OK` result.
## Controlled Reboot Recovery Increment 001
Before reboot testing, the build host was found to use DHCP on `enp8s0`. Its active values were
preserved exactly in the persistent NetworkManager profile and changed to manual assignment:
`172.16.10.144/24`, gateway `172.16.10.1`, DNS `8.8.8.8`, autoconnect enabled. After both controlled
reboots, the profile remained `manual` and the default route reported `proto static`.
The first reboot vector failed closed at `rollback_required` generation 4 with
`CHECKPOINT_TARGET_MISMATCH`: its checkpoint identity used container path `/state/python-env`, while
the initial systemd harness supplied the corresponding host path. No restore was attempted and no
artifact was produced. This negative result confirms exact target-path binding and is not counted as
a successful recovery.
A fresh vector then used the same `/state` mount namespace for checkpoint creation and boot recovery.
The test-only systemd one-shot was ordered after local filesystems and Docker and before
`multi-user.target`; its recovery container used `--network none` and the digest-pinned Python image.
After a distinct boot ID, systemd reported exit 0 and the dispatcher recovered generation 3
`installing` to generation 5 `rolled_back`.
- Static address after reboot: `172.16.10.144/24`; gateway `172.16.10.1`; DNS `8.8.8.8`.
- Original marker restored: pass.
- Activated environment preserved with executable Python in quarantine: pass.
- Final journal SHA-256:
`f5c60f693fd9ecf572e35d58390c596a71916cc1dac4f44a698aa67c81d1c0d9`.
- Checkpoint manifest SHA-256:
`e6c0ed458c8ed785a0d611fb4b6da0353b9632e008fbee2060c6b8e151b500c0`.
- Static-network evidence SHA-256:
`25453d9f8129abcf840a973bf650c9da896050296149e9cfab52a6082182a0f9`.
- Boot-service journal SHA-256:
`b03b15ec77223eeae99932db25c554ce68b0f71688bf33390174955d24f0e163`.
- Artifact directory entries: 0.
The temporary systemd unit was disabled, removed and daemon state reloaded after evidence capture; the
static NetworkManager profile intentionally remains. This closes controlled OS-reboot recovery for a
disposable strict-CPU target. It does not prove sudden physical power removal, production installer
unit packaging, live `/opt/cezen` ownership/ACL/xattrs, disk exhaustion or service-level health.
## Target Capacity Exhaustion Increment 001
Two fresh strict-CPU installations ran with `--network none` against executable disposable tmpfs
target filesystems. Journal and checkpoint state were mounted separately, modeling the required
reserved recovery capacity. Both failures occurred during offline `pip` installation before atomic
activation; the original marker remained intact and the executor durably recorded generation 5
`rolled_back`. A separate read-only container verified each terminal journal.
Byte-exhaustion vector:
- Target capacity: 134,217,728 bytes; 16,467,981 inodes.
- Failure: `ENOSPC` / `No space left on device`.
- Journal SHA-256:
`12f4fc3d9e6f52c9961b508f3c47782cf14f43211196378e3009db61c9e622d0`.
- Checkpoint manifest SHA-256:
`27832688460a1c58fbeac5df8ef7ee4ab158ccdcfc929a43b5c192dd07bb8964`.
- Log SHA-256:
`7c0a4d4c56f58c1bcabb71ddc727163777516c18a7ea0ea9371fe9399dcd2b7c`.
Inode-exhaustion vector:
- Target capacity: 2,147,483,648 bytes; exactly 2,000 inodes.
- Failure: `ENOSPC` while creating package metadata.
- Journal SHA-256:
`0353ef7a66d0ac0d7c7e2703f7da68874c127731248875fdcfab976565ada2d3`.
- Checkpoint manifest SHA-256:
`237ed9ca40a40633fbb69b4926719714ad8fd8d756e087eba1ac2ca3e2fe7242`.
- Log SHA-256:
`fd2aefb954162454150f1dd2c5e5d90ae644b787279ec90414e25f02ca00a67f`.
Artifact directory entries remained 0. This closes disposable target-byte and target-inode exhaustion
before activation. It does not prove exhaustion after activation, exhaustion of the reserved journal/
checkpoint capacity, production quotas/reservations, live paths/metadata or service-level rollback.
## Post-Activation Service-Health Rollback Increment 001
A fresh strict-CPU environment was installed from the 196-wheel selection with `--network none`,
passed built-in runtime health, was fsynced and atomically activated. A direct executable service-health
probe then verified that the activated target had an executable Python and the selected profile was
`cpu-only`, wrote an observation receipt and deliberately exited `23` before the `installed` journal
transition.
The executor recorded `rollback_required`, restored the exact checkpoint, retained the activated
environment with executable Python in quarantine and recorded generation 5 `rolled_back`. A separate
read-only container re-verified the terminal journal, marker and quarantine.
- Probe observed activated environment: pass; exit `23`.
- Original marker restored: pass.
- Final journal SHA-256:
`52e3f0313acc9291e79aff52843e956d247758e1a9a81f2f17d8eedcfed76100`.
- Checkpoint manifest SHA-256:
`93d4bf2bef768c9431831dd63689604e2978155584bcb86261626d83a3761ddd`.
- Probe observation SHA-256:
`fcef70518b5c9d61467cd211ade22eebd86876cb8f752a562a2aecb34594c9d5`.
- Installation/rollback log SHA-256:
`a40728e335f97864de8a47e9f96bd3315907915bd7a8d173372f1af3c5489b3d`.
- Artifact directory entries: 0.
This closes an external post-activation service-probe failure for a disposable CPU environment. It
does not prove production probe signing/packaging, systemd dependency ordering, the actual Nexus API/
portal/model service stack, consecutive health-window behavior, customer traffic draining or live
path/metadata rollback.
After T162 synchronization, the complete build-host deployment suite passed 31 tests with no skips or
failures. The suite log SHA-256 is
`6b89ad62a58f1c160ef97a5ad72e946c6742d00503bbdd9e33f15eebc3e8f6ae`; artifact entries remained 0.
## Isolated Systemd Dependency/Health-Window Increment 001
Six uniquely named temporary systemd units modeled model runtime, vector store, API, proxy, a bounded
health window and rollback. The four mock services used unused loopback ports and disposable staging;
no existing Nexus unit, port, path or service was stopped or modified. Systemd inspection confirmed
the API required and ordered after model/vector, the proxy required and ordered after API, and the
health window required and ordered after proxy with `OnFailure` bound to rollback.
The positive run completed three consecutive samples across four services: 12/12 semantic JSON health
checks passed. A fresh failure run made only the vector endpoint return 503. The health window exited
`23`, systemd triggered the rollback unit, and the dispatcher restored the verified checkpoint from
generation 3 `installing` to generation 5 `rolled_back`.
- Original marker restored: pass.
- Candidate marker retained in quarantine: pass.
- Journal SHA-256:
`ecc1d85eeae2d3be1d36ab2ca6d9583c3c3c9730f436f3e879c45ac7aad1d5ca`.
- Checkpoint manifest SHA-256:
`ab6886f7541ab4b138b230949093caadce048ef88213fd7b38db975ffd9a5b0d`.
- Positive 12-check window SHA-256:
`76b5349d2c06c4129316cce8e4e86b8977c469edb3d60f52b4893000cad8ced0`.
- Forced-failure window SHA-256:
`eddfdead8f85e3299bde8e45f90b229cb22fb7ff2f9bacf3e2ca77b0fc571510`.
- Systemd health/rollback log SHA-256:
`80844a1b67462f889ce6073d3e9bad90bd24a9f3a4626cffc492784d732d425f`.
- Artifact directory entries: 0.
All temporary units were stopped, removed and followed by `daemon-reload`. The host retained its static
`172.16.10.144/24` profile. This proves isolated systemd dependency ordering, a three-sample health
window and failure-triggered rollback; it does not validate the real Nexus services or production
units. Repository inspection shows `cezen-api.service` only orders after Ollama, does not require it,
does not order/require ChromaDB and does not encode proxy dependency or a health window.
## Candidate Production Unit Design/Validation Increment 001
Candidate-only unit definitions now live under `contracts/systemd/`; no file was copied to
`/etc/systemd/system`, no daemon reload occurred and no runtime service was changed. The candidate API
requires/orders after Ollama and ChromaDB. The nginx drop-in deliberately uses `Wants` plus `After` so
static recovery/help remains available during API failure, while the transactional activation-health
unit requires the complete graph and binds failure to the rollback unit. A root-owned active-operation
environment contract supplies validated operation, timing, evidence and recovery paths.
Ubuntu 22.04 systemd 249 validation had two layers:
- Direct candidate verification failed only because this build host lacks the proposed `/opt/cezen`
executables and `nginx.service`; those are real environment prerequisites, not parser substitutions.
- Structural verification used copies in `/tmp` with only executable paths replaced by `/bin/true` and
a temporary nginx base stub merged with the unchanged drop-in. It exited 0 with no candidate syntax
error or dependency cycle. The emitted Netplan/Snap/AnyDesk messages refer to unrelated host units.
- Offline `systemd-analyze security` scoring was not accepted by systemd 249 and no units were installed
merely to obtain a score; hardening remains subject to production-host validation.
Candidate SHA-256 values at validation:
- `cezen-api.service`: `7824c1642310570c49e0e4a1a5a7f5b2f269eecdbff86bb7867e4339113646c0`.
- `chromadb.service`: `e903c1df058ed9e5baf9584965c4d838fe8098eea6403fca18c8581c1eb788d0`.
- `nexus-one-activation-health.service`:
`c3b4229a42cc74dd7323a3501e938490594daadb2b22c770c71ae74621216b5e`.
- `nexus-one-activation-rollback.service`:
`2c967122edc5602a767df59787969cd0dcdaa4fc3cd1c9dbf057cea58a631019`.
- `nginx-cezen-ordering.conf`:
`0a8ad3528adaa695c8259893143dda062b98cb82c342fded8643419ef2bfba3f`.
- Final structural-validation log:
`d506c14455bf541f876bd925dc44410fc2085553f864fbca72d9fb0437b874d8`.
- Artifact directory entries: 0.
This closes design and structural/cycle validation only. Missing health-window implementation bytes,
unit/script signing, real service executables, category-resolved graph generation, approved timing,
actual service behavior, restart/timeout tests and production security scoring remain blockers.
## Deterministic Activation Health Window Increment 001
`activation-health-window.py` implements the candidate unit's loopback-only semantic window. It
validates a closed component profile, rejects hostname or non-loopback URLs, bounds samples/interval/
per-request timeout and response bytes, requires exact service identity/status JSON and publishes
canonical mode-`0600` evidence with file and directory fsync plus atomic rename. Required degradation
fails; optional degradation requires an explicit invocation flag. Stable exits are 0 for accepted,
23 for semantic rejection/rollback trigger and 2 for invalid configuration or evidence I/O.
Five focused local tests cover consecutive ready samples, explicit optional degradation, timeout,
malformed response, non-loopback/hostname rejection, mode-`0600` evidence and exit-23 behavior. A real
loopback HTTP integration then ran inside a digest-pinned Python container with `--network none`:
- Ready: 3 samples, exit 0, outcome `ready`.
- Optional degraded with explicit policy: 3 samples, exit 0, outcome `degraded`.
- Timeout: rejected at sample 1, exit 23, code `TIMEOUT`.
- Malformed JSON: rejected at sample 1, exit 23, code `MALFORMED_JSON`.
- Forced service failure: rejected at sample 1, exit 23, code `HTTP_503`.
Evidence SHA-256 values:
- Tool: `8fefbbc3ce4fe6034ad1bb4d9710a2b43c03e6dee8775514ab4733daacdffc15`.
- Updated candidate health unit:
`4c0d8cf9969f540376765cd94a5d9a82ee3e34b575d4d21ddd359a2e05d4c91a`.
- Integration log:
`d0bd31faaaca659e6f16356bc20e136a6ea786df5e2a97bd688e96e420751e23`.
- Integration summary:
`bba6ff23b07cda8655153d18a0adba2f96902a99f01fb2dd03fcc61752459d8d`.
- Ready / degraded / timeout / malformed / 503 evidence respectively:
`9b73ff97f500f5ac269e29c7cfb22abf011d4bc09333b6742f43bcb29e7108bf`,
`f629349943e8742b154506ef86e9b0340cdf838fad0cc73419368c814b415bea`,
`5e72919d09a55e9374124caad7bd891d303022d4eb95e0938e34164a788a514d`,
`2b523f7bcad31db6b8298f26767b9f94d6d7c31e350b1eb81eef55e46d9fa3d1`,
`067de6c45079dd2d40e521e473af4df1b1abe93974c3c38cc6277f8e4bf7e76d`.
- Artifact directory entries: 0.
This closes deterministic tool behavior and disconnected loopback parsing only. Signed production
health profiles, approved timing, real Nexus semantic endpoints, restart storms, total-window deadline,
unit/script installation and production security scoring remain open.
The updated candidate unit was rechecked with the correctly named `/tmp` nginx stub; systemd 249
structural verification exited 0 with no candidate error/cycle. After final synchronization, the full
build-host deployment suite passed 36 tests with no skips or failures. Suite log SHA-256:
`4a92a77d516ae9f59cdb724ea798b125d65d3e58439065c5066045db693f85c4`.
Artifact entries remained 0.
## Total-Window Deadline and Restart-Storm Increment 001
The health validator now has a monotonic total-window deadline distinct from its per-request timeout.
Every request is capped by remaining window time; an expired deadline or an interval that makes the
next mandatory sample impossible rejects with `WINDOW_TIMEOUT`. Component profiles also bind a literal
systemd unit and maximum restart count. Before HTTP, a fixed `systemctl show` query validates
`NRestarts`, `ActiveState` and `SubState`; excess restarts reject as `RESTART_STORM` before a stale
endpoint can be accepted.
Seven focused tests pass, including deterministic fake-clock deadline and restart-before-HTTP vectors.
Two uniquely named temporary systemd services then provided real build-host evidence:
- A healthy loopback service passed its systemd state check, but a 3-sample/1-second-interval window
with a 0.5-second total deadline rejected after sample 1 with exit 23 and `WINDOW_TIMEOUT`.
- A failing service configured with `Restart=on-failure` reached `NRestarts=8`; a signed-profile limit
of 1 rejected before HTTP with exit 23 and `RESTART_STORM`, recording restart count 8.
- Deadline evidence SHA-256:
`e1e26cbc66d5d575bf67f1a440b3e7b02179618cbd91c82f98a6edb175462231`.
- Restart-storm evidence SHA-256:
`1f31898643775a55c7f672e375993f696877d6e7c3a8aa032b99a680e80a5063`.
- Deadline and storm CLI logs SHA-256 (each):
`321d974cadea4ad833044190c40514041284a61e303ef3437ce82091d9ddb239`.
- Restart-storm systemd log SHA-256:
`dae95e3fbb0ebbc7e80babef8c5a33a1f52138d1a4c9f664f4f4a9439c4ab408`.
- Updated tool SHA-256:
`c152c7a9ab946a7566d6cf022962a616f54e3aff51ca83b06b872b335152f468`.
- Updated candidate health unit SHA-256:
`3b81ee483a70e120a826bcb99d83b73a329c9b793186c0e78afc96005427bc35`.
- Artifact directory entries: 0.
Both temporary units were removed and daemon state reloaded; static networking remained unchanged.
This closes isolated total-deadline and restart-counter behavior. Signed production limits, restart
counter reset/baseline semantics across activation, real-unit storms and complete rollback chaining
for these two codes remain open.
The updated candidate unit passes `/tmp`-only systemd 249 structural/cycle validation. After final T166
synchronization, the complete build-host deployment suite passed 38 tests with no skips or failures;
suite log SHA-256:
`a5378c125ce65d03ca1f96054feeefdadf2e01b3dfb23b2e2330f5342f10477c`.
Artifact entries remained 0 and no `nexus-test-*` unit remained installed.