Hazard Analysis — Project VELA
Status: hazard-control maturity is DESIGNED/ROADMAPPED unless marked IMPLEMENTED. The core investigation stack (LangGraph agents + Neo4j KG + Qdrant + 3 FastMCP servers + FastAPI) is implemented and tested, but the specific hazard controls below are a separate surface: most remain DESIGNED or ROADMAPPED, and each row labels its control’s maturity explicitly. Some verifying tests referenced against docs/design/06-TEST-PLAN.md do not yet run.
Last reviewed: (owner to date)
1. Scope and framing
VELA is ground-segment, advisory-only software. In v1 it has zero commanding authority — a human operator reads its output and executes (or not). Its failure mode is therefore epistemic, never kinetic: VELA cannot slew a wheel, trip a heater, or safe a bus. The realistic worst case is that VELA misleads a competent human into a wrong or delayed action, or suppresses a signal a human would otherwise have caught. This document is a tool-FMEA over those epistemic failure modes.
Because VELA is not flight software, the flight-software assurance regimes (DO-178C, NASA NPR 7150.2 Class A, MISRA C, JPL Power-of-Ten, ECSS-Q-ST-80 review boards) do not bind it. They govern code that flies or that autonomously commands a spacecraft; VELA does neither. What we borrow in spirit only: the discipline of a written hazard log with traceable mitigations (this file), FMEA severity/likelihood ranking, and an anomaly-review-board artifact (our per-investigation audit log). Nothing here should be read as a claim of compliance with those standards. See docs/assurance/classification-and-safety-case.md for the full non-applicability argument.
This analysis pairs with the four architecture audit invariants (docs/design/02-ARCHITECTURE.md §3):
- No agent output asserts a telemetry value not returned by a telemetry MCP call in that investigation.
- Every narrative claim carries a citation resolvable against KG/Qdrant (
vela/audit/citation_check.py). - No code path constructs or transmits telecommands (grep-guarded in CI).
- Full audit log (every tool call + hypothesis transition) persisted per investigation.
Several hazards below are the residual left when an invariant is only partially enforceable — invariant 2 stops an unresolvable citation, but not a resolvable-but-wrong one (H2), and invariant 1 stops fabricated values, but not a confident verdict built on genuine-but-weak evidence (H4, H5).
2. Rating scales
Severity — impact if the hazard reaches the operator and is acted on:
| S | Meaning |
|---|---|
| S4 Critical | Operator takes a wrong irreversible action on a real spacecraft, or a real anomaly is silently dropped, on VELA’s advice. |
| S3 Major | Operator meaningfully misdiagnoses or wastes a pass; recovery delayed; trust in the tool justifiably broken. Sev-1 review-board event if citation-integrity is the cause. |
| S2 Moderate | Misleading output caught by the operator/reviewer before action; time lost, no mission effect. |
| S1 Minor | Cosmetic or low-impact; noticed and shrugged off. |
Likelihood — chance per investigation in the designed system with the listed control in place (not the uncontrolled rate):
| L | Meaning |
|---|---|
| L4 Frequent | Expected routinely without a further mitigation. |
| L3 Occasional | Credible on a meaningful minority of investigations. |
| L2 Remote | Needs a specific coincidence of conditions. |
| L1 Improbable | Requires multiple independent failures. |
Ratings are pre-build engineering judgement, to be replaced by measured rates from Gate 1–3 (docs/design/06-TEST-PLAN.md). They are not empirical.
3. Hazard / tool-FMEA table
| ID | Hazard | Cause | Effect on operator / mission | Sev | Lik | Existing / designed control (maturity) | Verifying test | Residual risk & gap |
|---|---|---|---|---|---|---|---|---|
| H1 | Hallucinated citation — narrative cites a DocSection/case/KG node that does not exist. |
LLM invents a plausible doc_id#section or FailureMode id; no grounding lookup performed. |
Operator trusts a claim with no real source. One hallucinated reference before an anomaly review board is a sev-1. | S3 | L3 | Invariant 2: deterministic vela/audit/citation_check.py resolves every citation via resolve_citation MCP tool; unresolvable ⇒ report blocked, not shipped. Citation integrity target 100% at every tier. (DESIGNED) |
Golden-case citation-integrity check (Gate 2, 100% pass = hard gate); hostile-input suite asserts fabrication ⇒ block. | Checker is only as good as its resolver + is unbuilt. Residual is a checker bug or an as-yet-unhashed store; covered by H3. Low residual if checker is correct and runs on every path. |
| H2 | Right-doc-wrong-section — citation resolves (real doc_id, valid sha256) but points to a section that does not support the claim. |
Retrieval returns an adjacent chunk; LLM attaches the nearest citation rather than the supporting one. Passes invariant 2 because the target exists. | Operator who spot-checks the cite sees a real manual page and is falsely reassured; the claim itself is unsupported. | S3 | L3 | resolve_citation returns target_excerpt; drafter grounds on get_doc_section text, not just an id. Semantic support (claim ⊆ excerpt) is not yet checked — only existence is. (DESIGNED for existence; ASPIRATIONAL for support) |
LLM-judge “citation coverage / no-new-claims” rubric on golden narratives; monthly ≥20% human spot-check. | Open gap. Existence ≠ entailment. Judge is probabilistic, not a gate. Mitigation roadmap: an entailment check (claim vs target_excerpt) in citation_check.py, and surfacing the excerpt inline in the UI so the operator verifies support, not just existence. ROADMAPPED. |
| H3 | Tampered / invalid sha256 passes the checker — a DocSection whose content no longer matches its stored hash is accepted. |
Checker compares the wrong bytes, hashes post-normalization, caches a stale valid:true, or the store is mutated after ingest without re-hash. |
Citation appears verified; underlying text has drifted or been altered. Silent corruption of the load-bearing guarantee. | S4 | L2 | Content-addressed DocSections {doc_id, section_ref, page, sha256}; design-graph is human-gated, hash-cited, quasi-static. MCP servers are the sole data-access boundary (agents never touch stores; import-linted). (DESIGNED) |
Unit tests on citation_check with deliberately mutated content and mismatched hash → must fail closed. Not yet written. |
Checker must recompute the hash over canonical bytes at resolve time and fail closed on mismatch — spec this explicitly before build. Residual: normalization ambiguity (what exactly is hashed). Gap: no signing of the ingest pipeline; a compromised ingest writes a self-consistent (content,hash) pair. Threat noted in SECURITY.md. |
| H4 | Plausible-but-wrong root cause accepted by a fatigued operator (over-trust of a “high confidence” verdict). | Verdict is well-cited and internally coherent but wrong (missing hypothesis, biased retrieval). Operator on a night pass defers to the confident-sounding narrative. | Operator executes recovery for the wrong failure mode, or stops investigating too early. On a real bus/AOCS fault this can waste the recovery window. Top residual risk — see §4. | S4 | L3 | Human-in-the-loop interrupt() approval gate before any procedure release; RBAC reviewer role; shadow-mode-first (no action depends on VELA during pilot); top-3 root-cause target ≥70% / top-1 ≥45% (i.e. we design for being wrong a large fraction of the time). (DESIGNED) |
Gate 2 root-cause top-1/top-3 rates; Gate 3 diagnosis-agreement + operator-usefulness ≥70%; procedure rubric mean ≥3.5. | Highest residual in the system. Every technical citation control is intact yet the human is still misled. The label “high” is itself the hazard. Mitigation in §4: confidence calibration, mandatory Assumed-Bindings appendix, uncertainty-forward UI, competing-hypothesis display. ROADMAPPED. |
| H5 | Over-confident verdict on weak channel bindings — a confident narrative rests on BOUND_TO bindings with confidence < 0.5 that should have returned insufficient. |
Anonymized ESA-ADB channels bound to the reference ontology by inference; validator does not discount low-confidence paths; supervisor promotes an under-supported hypothesis. | Operator receives a definite answer where the honest answer is “we can’t tell on this data.” Directly erodes calibration and feeds H4. | S3 | L3 | Epistemic gating (03-DATA-SPEC.md §2b.3): channels_to_components returns binding metadata; any verdict resting solely on bindings <0.5 must return insufficient. Every report carries an “Assumed bindings” appendix (confidence + basis). Exact-binding paths (Yamcs sim, customer pilots via ICD) remove the inference entirely. (DESIGNED) |
Hostile-input / golden case: contradictory & weakly-bound inputs must yield insufficient, never confident narrative. Continuous-eval regression rule. |
Threshold (0.5) is a heuristic, unvalidated pre-build. “Rests solely on” needs a precise definition (what if one strong + one weak path?). Gap: no calibration data yet to justify 0.5. Tighten after Gate 2. |
| H6 | Ungrounded procedure step leaks past the block — a recovery step with no citation reaches the operator without its UNGROUNDED flag. |
Drafter emits a step; flag logic mis-fires, or a step is merged/reformatted post-validation in format_report, or a step is grounded on a weak/irrelevant cite (H2 variant in the procedure path). |
Operator executes an unsourced recovery action believing it is procedure-backed. Highest-consequence path because it is action, not just narrative. | S4 | L2 | Procedure drafter flags every step UNGROUNDED ⇒ blocks draft release; validate_procedure returns per-step {grounded, citations, risk_flags} (irreversible action, missing thermal/power precondition); format_report is rendering-only, no new content. interrupt() approval gate before release. (DESIGNED) |
Guardrail test: a draft containing any ungrounded step must not release. Gate 2 procedure rubric requires no safety score <3. | Two-stage grounding (drafter self-flag + validate_procedure) is good; both are unbuilt. Gap: format_report must be proven non-generative (grep/type guard) so it cannot reintroduce content post-validation. Risk-flag rules are rule-based v1 and incomplete. |
| H7 | Suppression of a real anomaly as a command-response false-negative — the telecommand context filter tags a genuine fault as an expected command response and suppresses it from alarming. | A real deviation coincidentally follows a related telecommand within the configured window; KG AFFECTS relatedness over-broad; window too wide. |
A real anomaly is silently dropped before it ever reaches an agent or operator. The one hazard where VELA’s absence of output is the harm. | S4 | L2 | Suppressed events are suppressed-from-alarming but always logged, not deleted (suppressed_as_command_response on every event). Triage agent explicitly re-checks that command-response suppression wasn’t wrong. Filter’s false-alarm reduction is measured at matched recall (recall is not allowed to pay for precision). (DESIGNED) |
Gate 1: false-alarm-duration −30% at matched recall vs GAT-alone; Gate 2 nominal + command-heavy runs; recall never traded silently. Gate 3 tracks VELA-only-catches and misses in weekly review. | Suppressed ≠ invisible, but a suppressed event with no human ever reviewing the suppressed log is a real miss. Gap: no UI surface yet for “what VELA suppressed and why.” Window/relatedness tuning is unvalidated. Recommend a reviewable suppression queue. ROADMAPPED. |
| H8 | Stale operational-overlay health state — a component health edge or binding confidence in the graph is out of date; agent reasons on expired state. | Health-state edge outlives its validity; an overturned hypothesis’s case writes not inverted; overlay write lags reality. | Operator gets a diagnosis anchored to a component state that is no longer true (e.g. “battery already degraded” when it was since swapped/recovered). | S3 | L2 | Overlay writes go through a typed, invertible operator vocabulary (OpenCase, LinkEvidence, UpdateBindingConfidence, ExpireHealthState) from day one; health-state edges carry temporal validity windows and expire; overturned hypotheses invert their case writes; every write recorded in the audit stream. (DESIGNED) |
Not yet specified. Needs: test that expired health edges are excluded from traversal; test that hypothesis reversal inverts overlay writes. | Design is sound (LiveGraph-style operator algebra) but entirely unbuilt and untested. Gap: no test proving traversal filters on validity window; no monitor for overlay/reality drift. Telemetry itself never enters the graph, which bounds this to overlay state only. |
| H9 | Airgap mode silently degrades output quality vs cloud — sovereign/local Qwen path produces weaker reasoning than the Claude cloud path with no signal to the operator. | Same config flag switches models; local model weaker on investigator/validator/drafter; operator cannot tell which mode produced a report. | Sovereign customer (defense, ISRO/NSIL) gets quietly worse diagnoses while believing parity. Erodes trust asymmetrically for the highest-stakes customers. | S3 | L3 | Mode is a single config flag; CI runs the golden-case eval in both cloud and airgap; airgap CI job runs with egress disabled except local services; bge-m3 embeddings local by design for airgap parity. (DESIGNED) |
Continuous-eval harness runs both modes; regression rules apply per mode. Golden-case judge-score-drop >5% blocks merge (per mode). | Parity is asserted by design, not yet measured. Gap: no per-mode quality delta is published; the report does not state which mode/model produced it. Mitigation: stamp mode + model id into every InvestigationReport and audit record; publish the airgap-vs-cloud golden delta and set a max acceptable gap. ROADMAPPED. |
| H10 | Prompt injection via anomaly / telemetry text — attacker-controlled or corrupt free-text in a doc, anomaly case, or channel/command name steers the agent (fabricate a verdict, exfiltrate, or emit a command-shaped string). | Ingested manuals, past-case summaries, or telemetry metadata contain adversarial instructions the LLM follows. | Manipulated diagnosis, attempted egress (defeated in airgap), or an attempt to produce telecommand-like output. | S3 | L2 | Invariant 3: no code path constructs or transmits telecommands (grep-guarded in CI) — caps the worst outcome structurally. MCP tools are read-only, typed, whitelisted (traverse hop-cap ≤3, no free-form Cypher from LLMs); airgap no-egress mode; ingest graph writes pass human review (extraction precision over recall). (DESIGNED) |
Hostile-input suite (malformed/adversarial args ⇒ structured errors, never fabrication). No dedicated injection-corpus test yet. | Gap: no injection-specific red-team corpus in the test plan; no content sanitization/detection on ingested free-text. Structural caps (read-only, no-command, no-egress) blunt impact but do not stop a manipulated narrative, which reduces to H4. Add an injection suite before pilot. ROADMAPPED. |
4. Top residual risk: operator over-trust (H4)
The most dangerous failure is the one where every technical control passes and the human is still misled. H4 (and its feeders H5, H9) is that failure. Citations resolve, hashes match, no telecommand is emitted, the audit log is complete — and a fatigued operator on a 0300 pass accepts a confident, well-cited, wrong root cause and acts on it. No invariant catches this, because nothing is technically false; the verdict is merely wrong, and the word “high” made it persuasive. Our own success target concedes we design for a top-1 correct rate of only ~45%: VELA will be confidently wrong on a meaningful fraction of investigations, by design. Automation bias does the rest.
This risk cannot be closed in v1 and we do not claim to. The mitigation is to make VELA honest about its own uncertainty and to keep the human epistemically active rather than deferential. Concrete, roadmapped mitigations:
- Confidence calibration, not confidence assertion. The
low|medium|highlabel must be calibrated against measured Gate-2 outcomes (e.g. “high” reserved for the band where measured top-1 accuracy actually exceeds X%), not emitted from the model’s self-report. Until calibration data exists, cap the displayed confidence and never render “high.” Track calibration (reliability curve) as an ongoing metric, not a one-off. (ASPIRATIONAL until Gate 2 data exists.) - Mandatory “Assumed bindings” appendix, foregrounded. Already required in every
InvestigationReport(03-DATA-SPEC.md§2b.3). Elevate it from appendix to a verdict-blocking banner whenever any relied-upon binding is <0.5 (ties H4 to H5): the operator sees “this diagnosis assumes the following unverified channel bindings” before the narrative, not after. - Uncertainty-forward UI. The console must foreground what VELA doesn’t know: competing hypotheses with their support/contradict/insufficient status (not just the winner), the evidence against the top verdict, and the specific telemetry windows checked. Present the diagnosis as “strongest of N hypotheses, here is the runner-up and why it wasn’t ruled out,” never as a single authoritative answer. (ROADMAPPED — the console does not yet do this.)
- Keep the human doing the epistemic work. The
interrupt()approval gate should require the reviewer to acknowledge the assumed bindings and the competing hypothesis before release, not merely click accept. Shadow-mode-first (Gate 3, 60 days, no action depends on VELA) exists precisely so operators calibrate their trust before any live reliance. - Measure over-trust directly. Gate 3 should log where the operator agreed with VELA and was later shown wrong (and vice versa) — the calibration of human trust in VELA, not just VELA’s accuracy. Put an over-trust KPI in the pilot MOU.
Owner action: promote items 1–3 into docs/design/06-TEST-PLAN.md and the UI backlog before the first live-adjacent pilot; treat “renders an uncalibrated ‘high confidence’” as a launch blocker.
5. Cross-cutting notes and gaps
- Maturity reality. The core stack is implemented and tested, but the hazard controls above are mostly DESIGNED or ROADMAPPED and several of their verifying tests in §3 do not yet run. This document is a hazard log tracking those controls to closure, not standalone evidence of a mitigated system.
- The invariants stop fabrication, not error. Invariants 1–3 defeat made-up values, unresolvable citations, and command emission. They are structurally silent on H2 (wrong-but-real citation), H4 (wrong-but-honest verdict), and H5 (confident-but-weak binding). Those three are where the real epistemic risk concentrates and where the mitigations are least mature.
- Fail-closed is the required default. H3, H6, H7, H8 all reduce to “a guard must fail closed” (block the report / keep the suppressed event / exclude expired state). Specify fail-closed behaviour explicitly per guard before implementation; a guard that fails open is a defect, not a degradation.
- Not covered here (out of v1 scope, per
06-TEST-PLAN.md): live commanding hazards (no command path exists — invariant 3), onboard/flight-software hazards (VELA is ground-only), radiation/hardware effects, and multi-satellite cross-correlation (v2). Recording the exclusions keeps sales and demo claims inside the tested envelope.
6. Traceability
Each hazard should carry a row in docs/assurance/rtm.csv linking it to its control, its verifying test id (docs/design/06-TEST-PLAN.md gate), and status. Citation-integrity failures (H1–H3, H6) are governed by docs/policies/incident-response-citation-integrity.md and are sev-1 before an anomaly review board. RBAC roles referenced in H4/H6 are defined in docs/policies/rbac.md; audit-log retention (invariant 4, the review-board artifact) in docs/policies/audit-log-retention.md. Security-threat framing for H3/H10 lives in SECURITY.md; data-isolation for the airgap/sovereign path (H9) in DATA-HANDLING.md and export posture in EXPORT-CONTROL.md.