Test & V&V Plan — Project VELA
Three gates. A gate must be green before spending on the next. Numbers are commitments, not aspirations — revise them consciously, in writing.
Gate 1 — Public benchmark (Weeks 4–5; Tier-0 demo metrics per PRD §5 come first and do not wait on this gate)
Protocol: ESA-ADB official pipeline and splits, Mission 1 full channel set + Mission 2 lightweight subset. Metrics exactly as the benchmark defines them — corrected event-wise F0.5 and false-alarm measures — so results are comparable to published baselines without argument.
Pass: ensemble (telecommand filter + ECOD + GAT) ≥ the ESA-ADB paper’s shipped baselines — Telemanom-ESA and DC-VAE-ESA (Kotowski et al. 2024, arXiv:2406.17826) — on Mission 1 event-wise F0.5, and total false-alarm duration reduced ≥ 30% vs the strongest single detector at matched recall (the filter’s contribution, reported separately — this is the demo’s headline number). NB: the canonical paper benchmarks no GAT and no ECOD; the often-cited “GAT ~0.78 / ECOD ~0.065” comes from a single secondary preprint (arXiv:2607.07335) at 30s resampling, and no method meets the paper’s native-resolution R8 (all score 0) — so state resampling cadence and compare against the shipped baselines, never a “published GAT number”.
Artifacts: results/bench/<git-sha>.json + auto-generated comparison table in results/bench/README.md. Every detection PR regenerates it.
Gate 2 — Fault-injection simulation (Weeks 6–8)
Rig: NASA NOS3 (cFS-based smallsat simulator) bridged to Yamcs; VELA consumes via the same adapter as production. Where NOS3 lacks a hook, fault is injected at the adapter layer with a labeled tag (honest about injection point in all reporting).
Fault matrix (minimum 12 runs; each fault at 2 severities where meaningful):
| Subsystem | Faults |
|---|---|
| EPS | battery cell degradation (capacity fade), solar array string loss, bus undervoltage transient |
| AOCS | reaction wheel friction increase, gyro bias drift, star tracker dropout |
| TTC | transponder gain degradation, spurious command-response latency |
| TCS | heater stuck-on, sensor open-circuit |
| OBDH | memory scrub error rate rise, watchdog resets |
Metrics & pass thresholds:
- Detection latency: median < 2× channel sampling interval after fault onset (per fault class).
- Root cause: correct failure mode in top-3 for ≥ 70% of runs; top-1 ≥ 45%.
- Procedure quality: retired-ISRO-advisor rubric (1–5 on correctness, safety, completeness); mean ≥ 3.5, no safety score < 3.
- Citation integrity: 100% automated pass (
citation_check); any hallucinated reference = run failed regardless of other scores. - False-alarm honesty: ≥ 6 nominal-only runs (including command-heavy passes); false events per 24h sim ≤ 1.
Gate 3 — Shadow-mode pilot (Months 4–6, first customer)
Protocol: 60 days read-only beside the customer’s ops. VELA logs every event + investigation; weekly joint review tagging each as match / partial / miss / VELA-only-catch. No operator action ever depends on VELA output during shadow.
KPIs for the end-of-pilot report: alarm volume VELA would have suppressed (%), diagnosis agreement rate, median time-to-diagnosis VELA vs operator on matched anomalies, count of VELA-only catches confirmed real, % procedure drafts operators say they’d accept with minor edits. Contract convert trigger: pre-agree the numbers that convert pilot → paid (put them in the pilot MOU, not in a slide).
Continuous eval harness (make eval, runs in CI, both modes)
- Golden cases: ≥ 15 frozen investigations (ESA-ADB replays + sim faults) with expert-approved reference verdicts, citations, and procedures. Diff current output vs reference: deterministic fields exact-match; narratives scored by LLM-judge rubric (verdict agreement, citation coverage, no-new-claims) with monthly human spot-check of ≥ 20% of judgments.
- Regression rules: verdict flip on any golden case blocks merge; judge-score drop > 5% aggregate blocks merge; prompt changes require side-by-side goldens diff in PR.
- Hostile-input suite: malformed MCP args, empty windows, gap-spanning queries, contradictory KG entries → assert structured errors and
UNGROUNDED/insufficientverdicts, never confident fabrication. - Guardrail tests: CI greps for command-issuance surface; import-linter forbids
vela/agents/ -> {neo4j,qdrant,stores}direct imports; airgap CI job runs with network egress disabled except local services.
What we deliberately do NOT test in v1
Live commanding (doesn’t exist), onboard constraints, radiation/hardware effects, and multi-satellite cross-correlation (v2). Written down so demos and sales claims stay inside the tested envelope.