aipackage/specs/001-enterprise-experience/contracts/offline-validation-lab.md

Offline Validation Lab and Evidence Ownership

Status: Approval and execution plan; no build or deployment authority.

Purpose

The lab proves that an artifact satisfies its declared connectivity, hardware, trust, interruption, recovery, update/model, entitlement and evidence contracts. A passing container/service check is not an installation proof, and a passing source test is not packaged-artifact evidence.

Isolation Topology

Each offline run uses a dedicated target attached only to an instrumented isolation switch or test host. The isolation environment provides no route to customer, corporate or public networks.

• Disable/remove Wi-Fi, Bluetooth networking, cellular, secondary Ethernet, IPMI/BMC shared-network routes and hypervisor NAT/bridged adapters before boot.
• The target MAY connect to a capture-only Ethernet segment with no DHCP, DNS, gateway, proxy, mirror, registry, NTP or responder. Packet capture records attempted traffic without satisfying it.
• Loopback remains available for local services. A local browser may use a directly connected capture-segment address only after the OS is installed; this does not create an external dependency.
• For physical-air-gap confirmation, repeat one accepted run with every data network cable physically disconnected and use local console plus post-run local evidence export.
• BMC management, power control and serial console MUST use a physically/logically separate lab management plane that is not visible as a target OS interface.

Attempted-Network Detection

Acceptance requires both outside observation and target evidence:

1. Capture all packets on the isolation segment from pre-boot through final evidence export using synchronized lab timestamps.
2. Record link state, interface inventory, routes, resolver configuration, proxy variables, sockets, firewall counters and package/container/model client logs before and after installation.
3. Classify every non-loopback packet by source process where locally observable, destination, protocol, stage and whether the attempt was required, optional or unexpected.
4. Any attempted dependency fetch—DNS, HTTP(S), Git, APT, pip/Conda, container registry, model registry, license, telemetry or time service—is a failure even when retry/fallback eventually succeeds.
5. Essential local IPv6 discovery or link-local traffic is documented separately; it cannot carry a dependency and SHOULD be disabled when not required by the supported product.

Packet capture with zero successful responses is necessary but insufficient: a client that attempted an external fetch still fails the offline claim.

Representative Hardware Matrix

Offline Workstation AMD64 — first target

Profile	Required representative target	Acceptance focus
Core/CPU fallback	AMD64, no supported NVIDIA GPU or sub-8 GB usable VRAM	Core platform readiness; truthful disabled local GPU/vLLM state
GPU lite	Supported NVIDIA GPU with 8 GB usable VRAM	Small Ollama chat and embedding; vLLM/fine-tuning disabled
GPU starter	Supported NVIDIA GPU with 24–32 GB usable VRAM	Ollama plus approved vLLM starter inference; RTX 4090/5090-class behavior
GPU standard	Supported NVIDIA GPU with at least 48 GB usable VRAM	Higher-capacity single-GPU profile and memory checks

At least one lower-profile target and one GPU target MUST be physical hardware. VM-only acceptance is insufficient for GPU driver, reboot, firmware, storage and power-loss evidence. AMD GPU/ROCm and ARM64 DGX Spark are outside the first release.

Server artifacts

• Online Universal Server: one supported S, M, L and Max selection case across enough hardware to exercise license/hardware allowed and denied boundaries; network failure tests use a controlled routed lab rather than the offline segment.
• Offline Server S/M: one representative S and one M physical target.
• Offline Server L/Max: one representative L and one Max target unless the approved payload/support distinction defines a smaller justified matrix. Multi-GPU topology must be real where claimed.

Final machine make/model, firmware, storage controller, disk size, GPU, RAM, NIC, Secure Boot state, OS architecture and evidence owner are approval fields in the release candidate.

Clean-Run Matrix

For each offline artifact/profile combination:

1. Securely erase/reset the target to the documented clean baseline.
2. Verify ISO whole-artifact digest/signature before flashing/attachment.
3. Record media identity, target hardware/firmware and isolation proof.
4. Boot, install OS, complete first boot, validate embedded payload, configure license/staging and reach truthful ready/degraded state with no undocumented intervention.
5. Reboot twice: once normally and once at the declared interruption boundary.
6. Validate portal/API/authentication, entitlement/profile, local help, backup/export, services, starter chat and embedding where supported, and vLLM only on approved profiles.
7. Export redacted evidence locally and reconcile installed inventory with the signed manifest.

Minimum release gate is three clean runs per offline ISO, including at least two physical targets and one interruption run. Repeated runs on the same uncleared installation do not count as clean runs.

Interruption Injection

Injection points are mapped to durable operation phases and use controlled power removal, forced reboot, process/service termination, media removal, disk-full/fault simulation, or corrupted test object as appropriate.

Required boundaries include OS install before/after disk commit, first boot, manifest/object verification, extraction/staging, database checkpoint/migration, application pointer switch, service restart, awaiting reboot, health validation, model import/activation, update activation, automatic rollback and restore-based rollback. Each injection is performed on disposable or recoverable targets with the expected outcome written before execution.

An interruption passes only when active application inventory, data/schema, selected category/tier or Workstation profile, services and durable evidence converge on the declared state without hidden lab repair.

Negative Security and Compatibility Matrix

Test unsigned/tampered/unknown/revoked/stale-policy artifacts; wrong architecture/category/tier/OS; expired/not-yet-valid/mismatched/binding-unavailable license; clock uncertainty/rollback; insufficient RAM/VRAM/disk; corrupted cache; duplicate/conflicting operation; unsafe archive paths; extra/missing payload objects; rollback below security floor; and unauthorized roles.

Negative fixtures use test keys and synthetic data. Production private keys and customer content never enter the lab evidence repository.

Evidence Package

Each run produces a signed/hashed evidence index containing:

• run ID, artifact ID/version/digest/signature, source revision and working-tree state;
• target hardware/firmware/architecture, storage, interfaces and Secure Boot state;
• isolation setup, capture-interface identity, capture digest and classified packet-attempt report;
• exact operator script, timestamps, boot IDs and injected fault;
• installed object/application/container/runtime/model/schema/license-policy inventory;
• operation/audit correlation IDs and redacted installer/service logs;
• readiness, service, portal/API, local inference and reboot results;
• disk/media/staging/rollback peak measurements;
• expected versus actual recovery state, deviations, severity and disposition;
• author, independent witness/reviewer and approval status.

Raw packet capture, logs and screenshots remain controlled lab attachments. Repository evidence stores non-sensitive summaries, checksums and approved redactions rather than credentials or customer data.

Evidence Gates and Owners

Gate	Responsible	Independent reviewer/accountable approval
Lab isolation and capture readiness	QA/test engineer	Security owner
Hardware/firmware baseline	Delivery engineer	Delivery/support owner
Artifact identity and inventory	Platform release engineer	Security + product owners
Workstation profile/model checks	AI/platform engineer	Product owner
Server tier and entitlement matrix	Platform/license engineer	Product + delivery owners
Interruption/update/rollback	Platform + data engineers	Delivery/support owner
Accessibility/customer journey	QA/product	Product owner
Final release evidence	Release coordinator	Product and delivery/support joint approval; security joins for trust/security-floor changes

The executor cannot be the sole approver of the same evidence. Failed critical safety, integrity, entitlement, recovery or zero-network findings block promotion until corrected and independently rerun.

Stop/Retry Rules

• Preserve failed evidence before resetting a target.
• Do not patch the target manually and continue the same acceptance run.
• A test-tool or lab-infrastructure failure is INVALID, not PASS or product FAIL; rerun from a clean baseline after documenting the cause.
• Three consecutive product failures at the same boundary trigger design review before more retries.
• Any external dependency attempt, signature bypass, secret exposure, silent data loss or ambiguous active version is critical.

Open Assignments

• Named people for each responsible/reviewer role and separation-of-duty confirmation.
• Physical hardware make/models, lab location, switch/capture/BMC topology and reservation dates.
• Approved capture and fault-injection tools, time synchronization method and evidence retention.
• Test data/model prompts, expected outputs/tolerances and accessibility participants.
• Severity/SLA policy, controlled-pilot targets and final evidence repository/access controls.