Fix Phase 1 server validation defects

specs/001-enterprise-experience/evidence/server-phase1.md

@@ -0,0 +1,75 @@
+# Server Phase 1 Validation Evidence
+
+**Validation date:** 2026-07-12 to 2026-07-13
+
+**Target:** Vast.ai full Ubuntu 22.04 VM, RTX 3090 24 GB, 23 vCPU, 52.8 GB usable RAM, 155.9 GB root filesystem
+
+**Package revision at initial install:** `55a2a41` plus the uncommitted corrections listed below
+
+**Install mode:** `install.sh --software-only --tier=starter`
+
+## Executive result
+
+The software package installs and the tested Phase 1 Server workflows operate on the representative
+cloud VM after correcting defects found during validation. This is evidence for a controlled Server
+pilot only. It is not ISO boot evidence, physical hardware evidence, Workstation evidence, a customer
+pilot, WCAG manual approval, or general-release approval.
+
+## Results
+
+| Area | Result | Evidence |
+|---|---|---|
+| Starter installation | PASS after Docker repository fix | Ansible recap: 86 ok, 51 changed, 0 failed, 2 ignored |
+| Source contract suite | PASS | 19 tests passed |
+| Shell/Python/static validation | PASS | `bash -n`, `py_compile`, `git diff --check` |
+| Tier playbook syntax | PASS | starter, entry, pro, max syntax checks |
+| Portal/backend | PASS | HTTP 200 after install and reboot |
+| Authenticated read endpoints | PASS | 51 HTTP 200 responses |
+| Entitlement restrictions | PASS | model router and two fine-tuning routes correctly returned 403 |
+| Authentication and RBAC | PASS | admin login; viewer permitted metrics and denied admin user list |
+| User/session lifecycle | PASS | create, login, revoke by deletion, expired-session rejection, self-delete protection |
+| API key lifecycle | PASS | create, verify, revoke, revoked-key rejection |
+| Guardrails | PASS | temporary keyword rule blocked matching input and was removed |
+| Prompt library | PASS | create, update and delete |
+| Backup/restore | PASS | ZIP created; listed; restored; pre-restore safety snapshot created |
+| Ollama inference | PASS | Qwen 2.5 1.5B GPU inference; model persisted across reboot |
+| Open WebUI | PASS after network correction | healthy container and model visible from WebUI |
+| RAG | PASS after corrections | collection, TXT upload, embedding, ready status, semantic retrieval, delete/cleanup |
+| vLLM | PASS after corrections | Phi-3 readiness and OpenAI-compatible chat completion |
+| Monitoring | PASS | Grafana health 200, Prometheus ready 200, DCGM and node exporter running |
+| Web console | PASS after correction | service active and `/console/` HTTP 200 |
+| Reboot persistence | PASS | all six services and five containers auto-started; models and backup persisted |
+| Concurrent inference | PASS | vLLM used 21.3 GB and Ollama 1.8 GB on RTX 3090; both completed inference |
+
+Post-reboot vLLM cold readiness occurred at approximately 144 seconds after boot.
+
+## Defects discovered and corrected in source
+
+1. Ubuntu ttyd does not support `--writable`, and the distribution ttyd service can occupy port 7681.
+2. Open WebUI could not reach an Ollama process that had not reloaded the appliance service environment.
+3. Backend targeted ChromaDB port 8000 while the service intentionally runs on 8100.
+4. Unpinned latest ChromaDB removed the v1 API used by the backend.
+5. Compatible ChromaDB dependencies conflicted with the shared AI environment; ChromaDB is now isolated.
+6. Short RAG documents caused an infinite chunking loop.
+7. Chroma query/upsert/delete data routes require collection UUIDs rather than collection names.
+8. Starter Phi-3 was incorrectly configured as AWQ.
+9. Blank optional vLLM arguments caused systemd command continuation into `ExecStop`.
+10. `vllm:latest` required an incompatible CUDA/driver combination; the image is now pinned to v0.10.2.
+11. vLLM was enabled but not explicitly started during installation.
+
+## Expected warnings / environment gaps
+
+- No final signed customer license was installed; field-staging mode was used.
+- TLS was not configured; access used SSH tunnels.
+- The VM began without a backup or models; both were created during testing.
+- Direct public port 80 was blocked by the provider; this is not an appliance Nginx failure.
+- Clean ISO boot, disk partitioning, BIOS/UEFI, USB media, physical NICs and offline ISO installation
+  cannot be validated on this rented VM.
+- Workstation validation (T036), manual WCAG 2.2 AA evidence, interruption/restricted-network trials,
+  controlled customer pilot and release approvals remain pending.
+
+## Release decision
+
+**Not ready for customer shipment yet.** The corrected package is suitable for another clean Server
+installation validation and controlled pilot preparation. A new ISO must be built from the corrected
+revision and both Server and Workstation ISO paths must be tested before release consideration.

specs/001-enterprise-experience/tasks.md

@@ -87,7 +87,7 @@ verify the same durable operation, accurate readiness, and safe next action rema
 - [x] T034 [US1] Add local setup, interruption, readiness, and escalation guidance in `cezen-portal/quickstart.html` and `cezen-portal/troubleshooting.html`
 - [x] T035 [US1] Add schema migration, operation recovery, and portal deployment steps to `ansible/roles/cezen-backend/tasks/main.yml` and `ansible/roles/cezen-nginx/tasks/main.yml`
 - [ ] T036 [US1] Verify clean install, interruption, restricted-network guidance, and WCAG evidence on a representative Workstation and record it in `specs/001-enterprise-experience/evidence/workstation-phase1.md`
-- [ ] T037 [US1] Verify clean install, interruption, restricted-network guidance, and WCAG evidence on a representative Server and record it in `specs/001-enterprise-experience/evidence/server-phase1.md`
+- [ ] T037 [US1] Verify clean install, interruption, restricted-network guidance, and WCAG evidence on a representative Server and record it in `specs/001-enterprise-experience/evidence/server-phase1.md` (software-package clean install and reboot evidence recorded; interruption, restricted-network, ISO boot, and manual WCAG evidence remain)
 - [x] T038 [US1] Verify source-to-package and source-to-ISO provenance for all Phase 1 surfaces and record checksums in `specs/001-enterprise-experience/evidence/phase1-manifest.md`
 
 **Checkpoint**: User Story 1 and product Phase 1 pass internal evidence review on Workstation and