166 lines
9.7 KiB
Markdown
166 lines
9.7 KiB
Markdown
# Offline Validation Lab and Evidence Ownership
|
||
|
||
**Status**: Approval and execution plan; no build or deployment authority.
|
||
|
||
## Purpose
|
||
|
||
The lab proves that an artifact satisfies its declared connectivity, hardware, trust, interruption,
|
||
recovery, update/model, entitlement and evidence contracts. A passing container/service check is not
|
||
an installation proof, and a passing source test is not packaged-artifact evidence.
|
||
|
||
## Isolation Topology
|
||
|
||
Each offline run uses a dedicated target attached only to an instrumented isolation switch or test
|
||
host. The isolation environment provides no route to customer, corporate or public networks.
|
||
|
||
- Disable/remove Wi-Fi, Bluetooth networking, cellular, secondary Ethernet, IPMI/BMC shared-network
|
||
routes and hypervisor NAT/bridged adapters before boot.
|
||
- The target MAY connect to a capture-only Ethernet segment with no DHCP, DNS, gateway, proxy, mirror,
|
||
registry, NTP or responder. Packet capture records attempted traffic without satisfying it.
|
||
- Loopback remains available for local services. A local browser may use a directly connected
|
||
capture-segment address only after the OS is installed; this does not create an external dependency.
|
||
- For physical-air-gap confirmation, repeat one accepted run with every data network cable physically
|
||
disconnected and use local console plus post-run local evidence export.
|
||
- BMC management, power control and serial console MUST use a physically/logically separate lab
|
||
management plane that is not visible as a target OS interface.
|
||
|
||
## Attempted-Network Detection
|
||
|
||
Acceptance requires both outside observation and target evidence:
|
||
|
||
1. Capture all packets on the isolation segment from pre-boot through final evidence export using
|
||
synchronized lab timestamps.
|
||
2. Record link state, interface inventory, routes, resolver configuration, proxy variables, sockets,
|
||
firewall counters and package/container/model client logs before and after installation.
|
||
3. Classify every non-loopback packet by source process where locally observable, destination,
|
||
protocol, stage and whether the attempt was required, optional or unexpected.
|
||
4. Any attempted dependency fetch—DNS, HTTP(S), Git, APT, pip/Conda, container registry, model registry,
|
||
license, telemetry or time service—is a failure even when retry/fallback eventually succeeds.
|
||
5. Essential local IPv6 discovery or link-local traffic is documented separately; it cannot carry a
|
||
dependency and SHOULD be disabled when not required by the supported product.
|
||
|
||
Packet capture with zero successful responses is necessary but insufficient: a client that attempted
|
||
an external fetch still fails the offline claim.
|
||
|
||
## Representative Hardware Matrix
|
||
|
||
### Offline Workstation AMD64 — first target
|
||
|
||
| Profile | Required representative target | Acceptance focus |
|
||
|---|---|---|
|
||
| Core/CPU fallback | AMD64, no supported NVIDIA GPU or sub-8 GB usable VRAM | Core platform readiness; truthful disabled local GPU/vLLM state |
|
||
| GPU lite | Supported NVIDIA GPU with 8 GB usable VRAM | Small Ollama chat and embedding; vLLM/fine-tuning disabled |
|
||
| GPU starter | Supported NVIDIA GPU with 24–32 GB usable VRAM | Ollama plus approved vLLM starter inference; RTX 4090/5090-class behavior |
|
||
| GPU standard | Supported NVIDIA GPU with at least 48 GB usable VRAM | Higher-capacity single-GPU profile and memory checks |
|
||
|
||
At least one lower-profile target and one GPU target MUST be physical hardware. VM-only acceptance is
|
||
insufficient for GPU driver, reboot, firmware, storage and power-loss evidence. AMD GPU/ROCm and ARM64
|
||
DGX Spark are outside the first release.
|
||
|
||
### Server artifacts
|
||
|
||
- Online Universal Server: one supported S, M, L and Max selection case across enough hardware to
|
||
exercise license/hardware allowed and denied boundaries; network failure tests use a controlled
|
||
routed lab rather than the offline segment.
|
||
- Offline Server S/M: one representative S and one M physical target.
|
||
- Offline Server L/Max: one representative L and one Max target unless the approved payload/support
|
||
distinction defines a smaller justified matrix. Multi-GPU topology must be real where claimed.
|
||
|
||
Final machine make/model, firmware, storage controller, disk size, GPU, RAM, NIC, Secure Boot state,
|
||
OS architecture and evidence owner are approval fields in the release candidate.
|
||
|
||
## Clean-Run Matrix
|
||
|
||
For each offline artifact/profile combination:
|
||
|
||
1. Securely erase/reset the target to the documented clean baseline.
|
||
2. Verify ISO whole-artifact digest/signature before flashing/attachment.
|
||
3. Record media identity, target hardware/firmware and isolation proof.
|
||
4. Boot, install OS, complete first boot, validate embedded payload, configure license/staging and
|
||
reach truthful ready/degraded state with no undocumented intervention.
|
||
5. Reboot twice: once normally and once at the declared interruption boundary.
|
||
6. Validate portal/API/authentication, entitlement/profile, local help, backup/export, services,
|
||
starter chat and embedding where supported, and vLLM only on approved profiles.
|
||
7. Export redacted evidence locally and reconcile installed inventory with the signed manifest.
|
||
|
||
Minimum release gate is three clean runs per offline ISO, including at least two physical targets and
|
||
one interruption run. Repeated runs on the same uncleared installation do not count as clean runs.
|
||
|
||
## Interruption Injection
|
||
|
||
Injection points are mapped to durable operation phases and use controlled power removal, forced
|
||
reboot, process/service termination, media removal, disk-full/fault simulation, or corrupted test
|
||
object as appropriate.
|
||
|
||
Required boundaries include OS install before/after disk commit, first boot, manifest/object
|
||
verification, extraction/staging, database checkpoint/migration, application pointer switch, service
|
||
restart, awaiting reboot, health validation, model import/activation, update activation, automatic
|
||
rollback and restore-based rollback. Each injection is performed on disposable or recoverable targets
|
||
with the expected outcome written before execution.
|
||
|
||
An interruption passes only when active application inventory, data/schema, selected category/tier or
|
||
Workstation profile, services and durable evidence converge on the declared state without hidden lab
|
||
repair.
|
||
|
||
## Negative Security and Compatibility Matrix
|
||
|
||
Test unsigned/tampered/unknown/revoked/stale-policy artifacts; wrong architecture/category/tier/OS;
|
||
expired/not-yet-valid/mismatched/binding-unavailable license; clock uncertainty/rollback; insufficient
|
||
RAM/VRAM/disk; corrupted cache; duplicate/conflicting operation; unsafe archive paths; extra/missing
|
||
payload objects; rollback below security floor; and unauthorized roles.
|
||
|
||
Negative fixtures use test keys and synthetic data. Production private keys and customer content never
|
||
enter the lab evidence repository.
|
||
|
||
## Evidence Package
|
||
|
||
Each run produces a signed/hashed evidence index containing:
|
||
|
||
- run ID, artifact ID/version/digest/signature, source revision and working-tree state;
|
||
- target hardware/firmware/architecture, storage, interfaces and Secure Boot state;
|
||
- isolation setup, capture-interface identity, capture digest and classified packet-attempt report;
|
||
- exact operator script, timestamps, boot IDs and injected fault;
|
||
- installed object/application/container/runtime/model/schema/license-policy inventory;
|
||
- operation/audit correlation IDs and redacted installer/service logs;
|
||
- readiness, service, portal/API, local inference and reboot results;
|
||
- disk/media/staging/rollback peak measurements;
|
||
- expected versus actual recovery state, deviations, severity and disposition;
|
||
- author, independent witness/reviewer and approval status.
|
||
|
||
Raw packet capture, logs and screenshots remain controlled lab attachments. Repository evidence stores
|
||
non-sensitive summaries, checksums and approved redactions rather than credentials or customer data.
|
||
|
||
## Evidence Gates and Owners
|
||
|
||
| Gate | Responsible | Independent reviewer/accountable approval |
|
||
|---|---|---|
|
||
| Lab isolation and capture readiness | QA/test engineer | Security owner |
|
||
| Hardware/firmware baseline | Delivery engineer | Delivery/support owner |
|
||
| Artifact identity and inventory | Platform release engineer | Security + product owners |
|
||
| Workstation profile/model checks | AI/platform engineer | Product owner |
|
||
| Server tier and entitlement matrix | Platform/license engineer | Product + delivery owners |
|
||
| Interruption/update/rollback | Platform + data engineers | Delivery/support owner |
|
||
| Accessibility/customer journey | QA/product | Product owner |
|
||
| Final release evidence | Release coordinator | Product and delivery/support joint approval; security joins for trust/security-floor changes |
|
||
|
||
The executor cannot be the sole approver of the same evidence. Failed critical safety, integrity,
|
||
entitlement, recovery or zero-network findings block promotion until corrected and independently rerun.
|
||
|
||
## Stop/Retry Rules
|
||
|
||
- Preserve failed evidence before resetting a target.
|
||
- Do not patch the target manually and continue the same acceptance run.
|
||
- A test-tool or lab-infrastructure failure is `INVALID`, not `PASS` or product `FAIL`; rerun from a
|
||
clean baseline after documenting the cause.
|
||
- Three consecutive product failures at the same boundary trigger design review before more retries.
|
||
- Any external dependency attempt, signature bypass, secret exposure, silent data loss or ambiguous
|
||
active version is critical.
|
||
|
||
## Open Assignments
|
||
|
||
- Named people for each responsible/reviewer role and separation-of-duty confirmation.
|
||
- Physical hardware make/models, lab location, switch/capture/BMC topology and reservation dates.
|
||
- Approved capture and fault-injection tools, time synchronization method and evidence retention.
|
||
- Test data/model prompts, expected outputs/tolerances and accessibility participants.
|
||
- Severity/SLA policy, controlled-pilot targets and final evidence repository/access controls.
|