Incident-Response Runbook — Citation Integrity

Status: pre-build — DESIGNED/ROADMAPPED unless a line is marked IMPLEMENTED. The core investigation stack (LangGraph agents, Neo4j KG, Qdrant, three FastMCP servers, FastAPI) is implemented and tested, including the citation-integrity check (vela/audit/citation_check.py) this runbook depends on. The operational response controls below (on-call rotation, sev-1 escalation, customer notification, formal postmortem process) are still being stood up — treat any control not explicitly marked IMPLEMENTED as a commitment, not a description of a running process.

Last reviewed: (owner to date)

Owner: SKOPAQ AI Pvt Ltd (solo founder / engineering lead). This document is right-sized for a pre-revenue, bootstrapped, one-person team: roles below are functions the founder wears in sequence, not separate headcount. Where a role should later be a distinct person (independent reviewer, incident declarer), it is called out.

1. Why this runbook exists

VELA is a ground-segment, advisory-only anomaly-investigation co-pilot. It has zero commanding authority in v1 — a human executes any procedure. Its entire value rests on one promise: every claim it puts in front of an operator is grounded in a real, resolvable source.

A citation is load-bearing. A satellite operator or an anomaly review board acts on VELA’s narrative because they trust the references under it. One hallucinated or invalid reference reaching that audience can end the product’s credibility permanently — a review board that catches VELA inventing a manual section will (correctly) never trust it again, and will say so to every peer operator in a small market (Dhruva Space, Pixxel, XDLINX, Digantara, later ISRO/NSIL).

Therefore: a citation-integrity violation that reaches an operator or a review board is pre-classified SEV-1. There is no triage discretion to downgrade it. This is a deliberate, standing decision recorded here so no on-call judgment call can soften it under pressure.

Not flight software. VELA is not bound by DO-178C, NASA NPR 7150.2 Class A, MISRA C, JPL Power-of-10, or ECSS software review boards — it neither runs onboard nor commands a spacecraft. We borrow in spirit only: a named severity scale, a written IR process, a golden/adversarial test corpus, and a post-incident review that feeds a hazard log. We do not claim conformance to any of those regimes.

2. What “citation integrity” means here (the thing we defend)

Anchored on the four architecture audit invariants (docs/design/02-ARCHITECTURE.md §3; enforced in code + CI in the production design):

#	Invariant	Runbook relevance
1	No agent output asserts a telemetry value not returned by a telemetry MCP call in that investigation.	A fabricated value is a citation-integrity violation even if a doc ref is present.
2	Every narrative claim carries a citation resolvable against KG/Qdrant (`vela/audit/citation_check.py`).	The primary invariant this runbook exists to protect.
3	No code path constructs or transmits telecommands (grep-guarded in CI).	Separate hazard, separate handling — see `docs/assurance/hazard-analysis.md`. Not this runbook.
4	Full audit log (every tool call + hypothesis transition) persisted per investigation.	The forensic substrate every step below depends on. If the audit log is incomplete, treat integrity as unverifiable = fail-closed.

A citation-integrity violation is any of:

A Citation (ref, optional sha256) that does not resolve via vela-knowledge-mcp.resolve_citation → {valid: false}.
A narrative sentence or procedure step with no citation where one is required (empty citation list on a claim).
A citation that resolves to the wrong content (right doc_id, wrong section_ref; or sha256 mismatch against the content-addressed DocSection).
A telemetry value in the narrative that no telemetry MCP call in that investigation returned (invariant 1).
A citation to a source outside the tenant’s own KG for a spacecraft-specific claim (external MCP context sources — NASA/CelesTrak — are never citation sources for spacecraft-specific claims; docs/design/04-MCP-SERVERS.md §4).

The 100% bar holds at every PRD tier (Tier-0 Demo, Tier-1 Sim, Tier-2 Pilot). Integrity is not a metric that trades off against usefulness. One hallucinated reference before a review board = SEV-1, full stop.

3. Severity levels

Sev	Definition	Example	Target response start
SEV-1	An invalid/fabricated citation, a fabricated telemetry value, or an ungrounded claim reached a human decision-maker (operator, reviewer, or review board) in a released report or live UI — OR strong evidence it did and we cannot rule it out.	VELA’s report to a pilot customer cited “EPS-GUIDE §4.2.1”; that section does not exist. Reviewer notices.	Immediate (drop other work).
SEV-2	A citation-integrity violation was produced by the pipeline but caught before human release by a control that is supposed to catch it (release gate, runtime `citation_check`), yet reveals a real checker or prompt weakness. Contained, but the failure mode is live.	Release gate blocked a report because `resolve_citation` returned `valid:false` on a homoglyph `doc_id`. Blocked correctly, but the agent produced it.	Same business day.
SEV-3	A checker/tooling defect, a gap in the adversarial golden suite, or a near-miss with no bad output produced — a latent hole.	A new golden test shows `citation_check` accepts a tampered `sha256` under one code path. No investigation ever exercised it.	Next working session; tracked to closure.
SEV-4	Cosmetic or documentation issue in citation rendering with no integrity impact (e.g. a correctly-resolving citation displays an ugly label).	Reviewer confused by citation format; content correct.	Backlog.

Pre-classification rule (standing): Anything matching the SEV-1 definition is SEV-1 on detection. On-call may escalate SEV-2→SEV-1 if human reach cannot be excluded, but may never downgrade a SEV-1. When reach is uncertain, fail-closed: assume it reached a human until the audit log proves otherwise.

4. Roles

Per docs/policies/rbac.md. Product RBAC roles are operator, reviewer, admin. IR roles map onto them; at solo-founder scale one person holds several, but the decisions stay separated in the record.

IR role	Who (v1)	RBAC mapping	Responsibility
Detector	Anyone: CI, runtime gate, a customer operator, a reviewer.	any / automated	Raises the signal.
Declarer	Founder (eng lead). Roadmapped: an independent reviewer once headcount exists.	`admin`	Declares severity, owns the incident, owns customer disclosure decision.
Fixer	Founder (eng).	`admin`	Roots-causes the checker gap, writes the adversarial golden test, ships the fix.
Verifier	Ideally not the Fixer. Solo-founder reality: the adversarial golden test is the independent verifier — the fix is not “done” until a new, failing-first test proves it.	`reviewer`	Confirms fix + confirms no other affected investigations.
Customer liaison	Founder.	`admin`	Owns review-board / customer comms, honesty guardrail enforcement.

Separation we hold even solo: Fixer’s own assertion “it’s fixed” is not acceptance. Acceptance = a committed adversarial test that failed before the fix and passes after. The test is the second person we don’t have yet.

5. Runbook

5.0 Fail-closed default

At every phase where state is ambiguous — audit log incomplete, resolve_citation erroring, reach-to-human unknown — the safe action is to block release and treat integrity as failed. VELA never emits a confident narrative on uncertain ground; the pipeline’s own insufficient/UNGROUNDED verdicts are the in-band expression of this, and the human process mirrors it.

5.1 Detection

Source	Mechanism	Status
CI	`vela/audit/citation_check.py` run over golden cases + adversarial corpus; any fail blocks merge (`docs/design/06-TEST-PLAN.md`, guardrail tests).	DESIGNED
Release gate (runtime)	Before any `format_report` output or `interrupt()` procedure-release resumes, `citation_check` runs over the full `InvestigationReport`; a violation blocks release and routes to the supervisor’s degraded path (`insufficient`).	DESIGNED
Runtime self-verify	Agents may call `vela-knowledge-mcp.resolve_citation` to self-check before proposing (`docs/design/04-MCP-SERVERS.md`).	DESIGNED
Operator / reviewer report	A human spots a citation that looks wrong in the UI or a delivered report. Highest-severity path — implies possible human reach.	DESIGNED (process)
Review board finding	Worst case: caught by the customer’s anomaly review board. Automatic SEV-1.	Process

Every detection must capture, at minimum: tenant_id, satellite_id, investigation / thread_id, audit_log_ref, the offending Citation (or the ungrounded claim), and how it was detected. These come free from the OpenTelemetry trace that backs the audit log (docs/design/02-ARCHITECTURE.md §6) — if they don’t, that missing-audit gap is itself a finding.

5.2 Triage

Classify severity using §3. Apply the pre-classification rule — do not deliberate about downgrading a SEV-1.
Establish reach. From the audit log: did this report/claim get released to a human (operator UI view, delivered PDF, review-board packet)? If the log cannot answer → fail-closed → SEV-1.
Scope the blast radius. Same defect class could affect other investigations. Query the audit store for other reports sharing: the same doc_id/section_ref, the same agent/prompt version, the same code path, the same tenant. Record the candidate set — you will re-verify it in Recovery.
Assign Declarer, Fixer, Verifier (§4). Open an incident record (tenant-scoped; see retention rules in docs/policies/audit-log-retention.md).

5.3 Containment (stop the bleeding, before root cause)

Block release of the affected investigation and any report in the candidate set — flip them to insufficient/held. No further human delivery until cleared.
The UNGROUNDED / insufficient mechanism is the containment primitive. Any claim that cannot be re-resolved is force-flagged UNGROUNDED (procedure steps) or the whole verdict downgraded to insufficient. This is exactly the in-band control the architecture already defines for uncertain output — containment reuses it, it is not a special mode.
Disable the affected path if the defect is systemic (a checker bug that could pass many bad citations, or a prompt regression producing fabrications): take the affected agent/tool path or the whole release surface offline via config until the fix lands. For a pilot, that means VELA output pauses; that is acceptable — a paused co-pilot is safe, a lying one is not. Advisory-only means there is no commanding to fail dangerous; the failure mode is credibility, and silence protects it.
Airgap / sovereign deployments: the customer runs their own instance. Containment there is a customer-executed action we must be able to instruct remotely and fast (config flag to hold releases). This dependency is DESIGNED — the runbook assumes we can push a “hold releases” instruction to a sovereign install; validating that channel is a build task.

5.4 Eradication (root-cause + immunize)

Root-cause the checker gap, not just the bad citation. The bad citation is a symptom. The defect is: why did citation_check / the release gate let it through, or why did the agent produce it? Every SEV-1/SEV-2 must name the specific gap (a resolver edge case, a prompt that invited fabrication, a missing sha256 comparison, an audit-log hole).

Add an adversarial golden test that fails first. No fix merges without a new test in the adversarial citation corpus that reproduces the exact hole and fails against the pre-fix code. The corpus must cover, at minimum, these attack classes:

Adversarial class	What it injects	Expected control response
Fabricated ref	`ref` to a `doc_id`/`section_ref` that does not exist.	`resolve_citation` → `valid:false`; release blocked.
Right-doc-wrong-section	Real `doc_id`, `section_ref` that doesn’t support the claim (or doesn’t exist).	Resolver rejects; content match fails.
Tampered sha256	Valid `doc_id`/`section_ref`, `sha256` not matching the content-addressed `DocSection`.	Hash mismatch → reject. Content-addressing (`docs/design/03-DATA-SPEC.md`) makes this deterministic.
Homoglyph doc_id	`doc_id` using look-alike Unicode chars to impersonate a real doc.	Normalize + exact-match against known IDs; reject unknown.
Empty citation list	A narrative/procedure claim with no citation at all.	Invariant 2 fail; `UNGROUNDED`/`insufficient`.
Phantom telemetry value	Narrative asserts a value no telemetry MCP call in the investigation returned.	Invariant 1 fail; block.
Out-of-tenant source	Spacecraft-specific claim cited to an external/context MCP source.	Source-scope check rejects (`04-MCP-SERVERS.md` §4).
Cross-tenant ref	A `doc_id` belonging to a different `tenant_id`.	`tenant_id` scoping (mandatory day one) rejects.

This table is the standing minimum; each incident adds to it, never subtracts. Status of the corpus itself: DESIGNED — it must exist and be green in CI before Tier-0 demo.

Harden the checker deterministically. citation_check is deterministic by design (content-addressed sha256, exact resolver) — keep it that way. No LLM in the integrity-checking loop; the check must not itself be capable of hallucinating.

5.5 Recovery

Re-verify the candidate set from triage: run the hardened citation_check over every investigation that shares the defect’s fingerprint. Clear the clean ones; re-issue corrected reports for any others found bad (each re-issue is itself a customer-comms event — §6).
Re-enable the disabled path only after: (a) the failing-first adversarial test now passes, (b) the full golden + adversarial suite is green in both cloud and airgap CI, (c) the candidate set is cleared.
Use forensic replay. The LangGraph durable-execution checkpointer supports time-travel replay (“replay this investigation”) — replay the incident investigation to confirm the fix changes the outcome, and to produce a clean before/after for the review-board packet if one is owed.
Confirm the audit log is complete for the incident window. An incomplete audit log during a citation incident is a second finding (invariant 4).

5.6 Post-incident review (PIR)

Within one working week of resolution, a short written PIR (no blameless-culture theater needed at solo scale — just honest):

Timeline: produced → detected → contained → eradicated → recovered.
The specific checker/prompt/audit gap. The adversarial test(s) added.
Did it reach a human? If yes, what was disclosed and to whom.
Feed docs/assurance/hazard-analysis.md: every SEV-1/SEV-2 updates the citation-integrity hazard entry — likelihood, existing controls, residual risk, new mitigations. Update the RTM (docs/assurance/rtm.csv) if a new requirement/test row is created. This closes the loop from incident → hazard log → requirement → test.
If the incident revealed the class of defect is broader (e.g. resolver trusts caller-supplied hashes), file it as a design finding against docs/design/02-ARCHITECTURE.md / 04-MCP-SERVERS.md, not just a patch.

6. Communications

6.1 Internal

Solo-founder: the “internal comms” is the incident record itself — declared severity, timeline, PIR — kept in the tenant-scoped incident log. Roadmapped: as the team grows, a same-day internal note on any SEV-1/2 to all engineers before external disclosure.

6.2 Customer / review-board disclosure

Disclosure is owed and proactive for any SEV-1 (and any SEV-2 where output reached the customer environment, even if their gate caught it). We do not wait to be asked.

Situation	Action
SEV-1, reached a customer operator or their review board	Notify the customer liaison contact within one business day. State plainly: what was wrong, which investigation(s), that VELA is advisory-only so no command was affected, containment taken, fix + adversarial test added, whether re-issued reports follow.
SEV-1/2 caught by our gate, customer environment involved (e.g. their airgap install held a report)	Notify with same content, framed as “our control worked and here is the hardening.”
Corrected re-issue of a previously delivered report	Each re-issue explicitly flagged as a correction with a one-line reason; never silently swapped.
Shadow-mode pilot	Log it in the weekly joint review (`docs/design/06-TEST-PLAN.md`, Gate 3); shadow mode means no operator acted on it, but disclosure still happens — the pilot’s whole point is trust calibration.

Honesty guardrail on all comms (enforced, non-negotiable): Disclosure language never claims a certification or audit we do not hold. We do not say “SOC 2”, “ISO 27001 certified”, or “independently audited”. We say what is true: what failed, what we did, what is now tested. Referenced posture lives in SECURITY.md and COMPLIANCE-ROADMAP.md (both roadmap-framed). Overstating our assurance posture during an integrity incident would compound exactly the credibility damage the incident already caused.

7. Cross-references

docs/design/02-ARCHITECTURE.md — the four audit invariants; MCP audit boundary; durable-execution replay; interrupt() gates.
docs/design/03-DATA-SPEC.md — Citation / InvestigationReport contracts; content-addressed DocSection sha256.
docs/design/04-MCP-SERVERS.md — resolve_citation; external-source exclusion rule.
docs/design/06-TEST-PLAN.md — CI guardrail tests; golden + hostile-input suites; tier gates.
docs/assurance/hazard-analysis.md — citation-integrity hazard entry the PIR feeds.
docs/assurance/classification-and-safety-case.md — why VELA is advisory ground software, not flight software.
docs/assurance/rtm.csv — requirement→test traceability updated by PIRs.
docs/policies/rbac.md — operator/reviewer/admin roles behind the IR roles.
docs/policies/audit-log-retention.md — retention of audit logs + incident records.
SECURITY.md, COMPLIANCE-ROADMAP.md — honesty-bounded external posture.

8. Status ledger (control maturity)

Control	Maturity
`vela/audit/citation_check.py` deterministic checker	DESIGNED
Adversarial golden corpus (the 8 classes in §5.4)	DESIGNED — must be green before Tier-0 demo
Runtime release gate + `UNGROUNDED`/`insufficient` containment	DESIGNED
`resolve_citation` MCP tool	DESIGNED
Audit log via OpenTelemetry traces	DESIGNED
Forensic time-travel replay	DESIGNED
Remote “hold releases” instruction to sovereign installs	DESIGNED — channel not yet built
Independent Verifier as a distinct person	ROADMAPPED (post-headcount)
Internal same-day SEV note to team	ROADMAPPED (post-headcount)
This runbook exercised against a real incident	ASPIRATIONAL (no pipeline yet)