enhance rag retrieval + summary

specs/004-rag-retrieval-quality/checklists/requirements.md

@@ -0,0 +1,34 @@
+# Specification Quality Checklist: RAG Retrieval Quality Improvements
+
+**Purpose**: Validate specification completeness and quality before proceeding to planning
+**Created**: 2026-04-06
+**Feature**: [spec.md](../spec.md)
+
+## Content Quality
+
+- [x] No implementation details (languages, frameworks, APIs)
+- [x] Focused on user value and business needs
+- [x] Written for non-technical stakeholders
+- [x] All mandatory sections completed
+
+## Requirement Completeness
+
+- [x] No [NEEDS CLARIFICATION] markers remain
+- [x] Requirements are testable and unambiguous
+- [x] Success criteria are measurable
+- [x] Success criteria are technology-agnostic (no implementation details)
+- [x] All acceptance scenarios are defined
+- [x] Edge cases are identified
+- [x] Scope is clearly bounded
+- [x] Dependencies and assumptions identified
+
+## Feature Readiness
+
+- [x] All functional requirements have clear acceptance criteria
+- [x] User scenarios cover primary flows
+- [x] Feature meets measurable outcomes defined in Success Criteria
+- [x] No implementation details leak into specification
+
+## Notes
+
+- All items pass. Ready to proceed to `/speckit.clarify` or `/speckit.plan`.

specs/004-rag-retrieval-quality/contracts/chat-api.md

@@ -0,0 +1,33 @@
+# API Contract: Chat (unchanged endpoints)
+
+**Branch**: `004-rag-retrieval-quality` | **Date**: 2026-04-06
+
+## No endpoint changes
+
+This feature makes no changes to the public API surface. All existing `/api/v1/chat/...` endpoints remain identical in path, method, request body, and response shape.
+
+## Message response — `sources` field addition
+
+The only observable change to callers is that each source entry in `Message.sources` gains an optional `refLabel` field. This is backwards-compatible (additive only).
+
+### Existing contract (unchanged)
+
+```
+POST /api/v1/chat/sessions/{sessionId}/messages
+Body: { "content": "..." }
+Response: Message { id, sessionId, role, content, sources: [...], createdAt }
+```
+
+### Source entry schema (additive change)
+
+Before:
+```json
+{ "type": "TEXT", "bookTitle": "...", "page": 142, "chunkText": "..." }
+```
+
+After (new optional field):
+```json
+{ "type": "TEXT", "refLabel": "S1", "bookTitle": "...", "page": 142, "chunkText": "..." }
+```
+
+Frontend consumers that ignore unknown fields are unaffected. `ChatMessage.vue` may optionally use `refLabel` for future inline citation linking.

specs/004-rag-retrieval-quality/contracts/topics-api.md

@@ -0,0 +1,68 @@
+# API Contract: Topics — Summary Persistence
+
+**Base path**: `/api/v1/topics`
+
+---
+
+## Existing endpoint (unchanged behaviour, extended response)
+
+### `POST /api/v1/topics/{id}/summary`
+
+Generates a new summary for the topic, **persists it**, and returns it.
+
+**Path param**: `id` — topic id (e.g. `intracranial-aneurysms`)
+
+**Response** `200 OK`:
+```json
+{
+  "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
+  "summaryNumber": 3,
+  "topicId": "intracranial-aneurysms",
+  "topicName": "Intracranial Aneurysms",
+  "summary": "## Key Points\n...",
+  "sources": [
+    { "bookId": "uuid", "bookTitle": "Youmans & Winn", "page": 142 }
+  ],
+  "generatedAt": "2026-04-07T10:23:00Z"
+}
+```
+
+**Error responses**:
+- `404 Not Found` — topic id does not exist
+- `503 Service Unavailable` — no books ready (existing `NoKnowledgeSourceException`)
+
+---
+
+## New endpoints
+
+### `GET /api/v1/topics/{id}/summaries`
+
+Returns the list of saved summaries for a topic (no full text — list metadata only).
+
+**Path param**: `id` — topic id
+
+**Response** `200 OK`:
+```json
+[
+  { "id": "uuid-1", "summaryNumber": 1, "generatedAt": "2026-04-06T08:00:00Z" },
+  { "id": "uuid-2", "summaryNumber": 2, "generatedAt": "2026-04-06T09:15:00Z" }
+]
+```
+
+Ordered ascending by `summaryNumber`. Empty array if no summaries saved yet.
+
+**Error responses**:
+- `404 Not Found` — topic id does not exist
+
+---
+
+### `GET /api/v1/topics/{id}/summaries/{summaryId}`
+
+Fetches the full content of a specific saved summary.
+
+**Path params**: `id` — topic id, `summaryId` — summary UUID
+
+**Response** `200 OK`: same shape as `POST /summary` response above.
+
+**Error responses**:
+- `404 Not Found` — topic or summary not found

specs/004-rag-retrieval-quality/data-model.md

@@ -0,0 +1,91 @@
+# Data Model: RAG Retrieval Quality + Topic Summary Persistence
+
+**Branch**: `004-rag-retrieval-quality` | **Date**: 2026-04-07
+
+---
+
+## New persistent entity: TopicSummaryEntity
+
+**Table**: `topic_summary`  
+**Migration**: `V6__topic_summary.sql`
+
+| Column | Type | Notes |
+|--------|------|-------|
+| `id` | `UUID` PK | `gen_random_uuid()` default |
+| `topic_id` | `VARCHAR(100)` NOT NULL | FK to `topic.id` |
+| `summary_number` | `INT` NOT NULL | Sequential per topic (1, 2, 3, …). Set at insert time: `COUNT(*) WHERE topic_id = ? + 1`. |
+| `summary` | `TEXT` NOT NULL | Full markdown summary text |
+| `sources_json` | `TEXT` NOT NULL | JSON array of `SourceReference` objects (same structure as `TopicSummaryResponse.sources`) |
+| `generated_at` | `TIMESTAMPTZ` NOT NULL | UTC timestamp of generation |
+
+**Constraints**: no unique constraint on `summary_number` (sequential, not concurrent-safe for POC). No FK constraint enforced at DB level (topic ids are static seed data).
+
+---
+
+## In-memory objects (new, from RAG quality work)
+
+### ExpandedQuery (value object, not persisted)
+
+Produced by `QueryExpansionService` for each user message.
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `original` | `String` | The user's literal question |
+| `rewritten` | `String` | Clinically rewritten version used for vector search |
+
+---
+
+### LabelledContext (value object, not persisted)
+
+Produced by `ChatService.buildContextPrompt()` to track the mapping from ref-labels to source entities.
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `sectionLabels` | `Map<String, SectionEntity>` | e.g. `{"S1" → SectionEntity, "S2" → SectionEntity}` |
+| `figureLabels` | `Map<String, FigureEntity>` | e.g. `{"F1" → FigureEntity}` |
+| `promptText` | `String` | The fully formatted context prompt including `[S1]`, `[F1]` tags |
+
+---
+
+## New API DTOs
+
+### SavedSummaryItem (list view — no full text)
+
+```java
+record SavedSummaryItem(UUID id, int summaryNumber, Instant generatedAt) {}
+```
+
+Used in `GET /api/v1/topics/{id}/summaries` to show the summary history list without transmitting full text.
+
+### TopicSummaryResponse (existing, extended)
+
+Adds `id` (UUID) and `summaryNumber` (int) fields so the frontend knows which saved record was just created.
+
+---
+
+## Existing entities (unchanged)
+
+| Entity | Table | Change |
+|--------|-------|--------|
+| `SectionEntity` | `section` | None |
+| `FigureEntity` | `figure` | None |
+| `Message` | `message` | `sources` field gets `refLabel` key added per entry |
+| `ChatSession` | `chat_session` | None |
+| `Book` | `book` | None |
+| `Topic` | `topic` | None |
+
+---
+
+## Message.sources structure (existing, clarified)
+
+After the RAG quality feature each entry includes `refLabel`:
+
+```json
+{
+  "type": "TEXT",
+  "refLabel": "S1",
+  "bookTitle": "Youmans & Winn Neurological Surgery",
+  "page": 142,
+  "chunkText": "..."
+}
+```

specs/004-rag-retrieval-quality/plan.md

@@ -0,0 +1,108 @@
+# Implementation Plan: RAG Retrieval Quality + Topic Summary Persistence
+
+**Branch**: `004-rag-retrieval-quality` | **Date**: 2026-04-07 | **Spec**: [spec.md](spec.md)  
+**Input**: Feature specification from `/specs/004-rag-retrieval-quality/spec.md` + user request: "Save summaries by topic; list previously saved summaries; button to generate new."
+
+## Summary
+
+Improve RAG retrieval quality by (1) expanding user queries via LLM rewrite to bridge vocabulary gaps and (2) validating generated citations against the retrieved context to eliminate hallucinated references. Additionally, persist generated topic summaries to the database so students can revisit past summaries and generate new ones on demand.
+
+## Technical Context
+
+**Language/Version**: Java 21 (backend), TypeScript / Node 20 (frontend)  
+**Primary Dependencies**: Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API (chat + embeddings), Vue 3.4, Pinia 2.1, Axios 1.7  
+**Storage**: PostgreSQL (JPA + Flyway), pgvector (`VectorStore`)  
+**Testing**: JUnit 5, Spring Boot Test, Vitest  
+**Target Platform**: Linux server  
+**Project Type**: Web application (backend API + Vue frontend)  
+**Performance Goals**: RAG query latency remains acceptable (one additional LLM call for query expansion, ~1–2s overhead)  
+**Constraints**: KISS — no new architectural layers; no new external services; single deployable backend + frontend
+
+## Constitution Check
+
+| Principle | Status | Notes |
+|-----------|--------|-------|
+| I — KISS | ✅ Pass | Query expansion = one LLM call. Citation validation = string scan. Summary persistence = one new table + minimal service change. No new layers. |
+| II — Easy to Change | ✅ Pass | `QueryExpansionService`, `CitationValidatorService`, `TopicSummaryRepository` are small, focused interfaces. Swappable without touching callers. |
+| III — Web-First | ✅ Pass | All new functionality exposed via REST at `/api/v1/...`. Frontend-only UI changes. |
+| IV — Documentation as Architecture | ⚠️ Action needed | README must be updated to reflect new `topic_summary` table and the summary history flow. Update in the same PR. |
+
+## Project Structure
+
+### Documentation (this feature)
+
+```text
+specs/004-rag-retrieval-quality/
+├── plan.md              # This file
+├── research.md          # Phase 0 — decisions on query expansion, citation grounding, summary persistence
+├── data-model.md        # Phase 1 — topic_summary table, ExpandedQuery, LabelledContext DTOs
+├── contracts/
+│   ├── chat-api.md      # Existing chat API (updated for labelled sources)
+│   └── topics-api.md    # New/updated topics API (summary list + detail endpoints)
+└── tasks.md             # Phase 2 output (/speckit.tasks command)
+```
+
+### Source Code
+
+```text
+backend/
+├── src/main/java/com/aiteacher/
+│   ├── retrieval/
+│   │   ├── QueryExpansionService.java        (NEW) — LLM rewrite of user query
+│   │   ├── ExpandedQuery.java                (NEW) — value object {original, rewritten}
+│   │   ├── LabelledContext.java              (NEW) — value object {sectionLabels, figureLabels, promptText}
+│   │   └── CitationValidatorService.java     (NEW) — strip unknown [Sx]/[Fx] refs from answer text
+│   └── topic/
+│       ├── TopicSummaryEntity.java           (NEW) — JPA entity for topic_summary table
+│       ├── TopicSummaryRepository.java       (NEW) — findByTopicIdOrderBySummaryNumberAsc
+│       ├── SavedSummaryItem.java             (NEW) — DTO for list view (id, summaryNumber, generatedAt)
+│       ├── TopicSummaryResponse.java         (MODIFY) — add id: UUID, summaryNumber: int fields
+│       ├── TopicSummaryService.java          (MODIFY) — persist after generation; add list/get methods
+│       └── TopicController.java             (MODIFY) — add GET /summaries and GET /summaries/{id}
+├── src/main/resources/db/migration/
+│   └── V6__topic_summary.sql               (NEW) — create topic_summary table
+└── chat/
+    └── ChatService.java                    (MODIFY) — use QueryExpansionService + CitationValidatorService
+
+frontend/
+└── src/
+    ├── stores/
+    │   └── topicStore.ts                   (MODIFY) — add fetchSummaries(), fetchSummaryDetail(), summaryList state
+    └── views/
+        └── TopicsView.vue                  (MODIFY) — show summary list when topic selected, "Generate New" button
+```
+
+**Structure Decision**: Option 2 (web application). Existing `backend/` + `frontend/` directories. No new top-level directories.
+
+## Complexity Tracking
+
+> No constitution violations introduced.
+
+---
+
+## Phase 0: Research (complete)
+
+See [research.md](research.md). All decisions resolved:
+- Query expansion: single LLM rewrite call
+- Citation grounding: ref-label tagging + post-generation string scan
+- Summary persistence: new `topic_summary` table, sequential numbering at insert time
+
+## Phase 1: Design (complete)
+
+### Data model
+See [data-model.md](data-model.md). New `topic_summary` table; two new in-memory value objects; `TopicSummaryResponse` extended with `id` + `summaryNumber`.
+
+### API contracts
+See [contracts/topics-api.md](contracts/topics-api.md):
+- `POST /api/v1/topics/{id}/summary` — now persists and returns `id` + `summaryNumber`
+- `GET /api/v1/topics/{id}/summaries` — list metadata (no full text)
+- `GET /api/v1/topics/{id}/summaries/{summaryId}` — full detail
+
+### Frontend behaviour
+
+When a **topic card is clicked** (before generating):
+1. Call `GET /api/v1/topics/{id}/summaries`.
+2. If summaries exist → show list panel with chips: "Summary #1 · Apr 6", "Summary #2 · Apr 7", … + a "Generate New" button.
+3. If no summaries → show "Generate Summary" button (current behaviour).
+4. Clicking a chip → call `GET /api/v1/topics/{id}/summaries/{summaryId}` → display the saved summary.
+5. Clicking "Generate New" → call `POST /api/v1/topics/{id}/summary` → display new summary, refresh list.

specs/004-rag-retrieval-quality/research.md

@@ -0,0 +1,71 @@
+# Research: RAG Retrieval Quality Improvements
+
+**Branch**: `004-rag-retrieval-quality` | **Date**: 2026-04-06
+
+## Decision 1: Query Expansion Strategy
+
+**Decision**: Single LLM rewrite — ask the model to restate the user's question using clinical/technical terminology before retrieval. Use the rewritten query for vector search instead of (or in addition to) the original.
+
+**Rationale**: The simplest approach that directly addresses vocabulary mismatch. A single extra LLM call rewrites the query into the language of the documentation (clinical terms), so the embedding similarity search has a much better chance of matching. No new dependencies, no index changes.
+
+**Alternatives considered**:
+- *HyDE (Hypothetical Document Embeddings)*: Ask the model to write a hypothetical answer, then embed that. More powerful but adds latency and the answer may itself hallucinate clinical content — rejected for POC.
+- *Multi-query retrieval*: Generate 3–5 alternative queries and merge results. Effective but multiplies retrieval calls and deduplication complexity — rejected (KISS).
+- *Synonym dictionary*: Pre-built medical thesaurus mapping. No new dependencies but requires maintenance and won't generalise — rejected.
+- *Re-ranking*: Run a cross-encoder after retrieval. Addresses ordering, not vocabulary gap — deferred.
+
+**Implementation note**: The rewrite prompt should be a short, focused instruction: "Rewrite the following question using precise medical/surgical terminology as it would appear in a neurosurgery textbook index. Output only the rewritten question." Use the same `ChatClient` bean; no new API client needed.
+
+---
+
+## Decision 2: Citation Grounding Strategy
+
+**Decision**: Tag each retrieved context section with a short ref-label (`[S1]`, `[S2]`, …, `[Fn]` for figures) in the prompt. Instruct the model to cite only using those labels. Post-process the generated answer to detect and strip any citation that does not correspond to a known label.
+
+**Rationale**: The simplest way to make citations verifiable — all valid citation targets are enumerated in the prompt itself. Post-processing is a pure string operation; no second LLM call needed for validation.
+
+**Alternatives considered**:
+- *Ask the model to self-check citations*: Second LLM call to verify each claim. More accurate but doubles cost and latency — rejected for POC.
+- *Structured output (JSON)*: Return answer as JSON with claim/source pairs. Most precise but requires significant prompt engineering and frontend changes — deferred.
+- *No citation enforcement*: Let the model cite freely and show retrieved sources separately. Already the current state — rejected because it doesn't solve citation hallucination.
+
+**Implementation note**:
+- Context sections labelled `[S1] Section Title, p.N` through `[Sk]` in `buildContextPrompt()`.
+- Figures labelled `[F1]`, `[F2]` etc.
+- System prompt updated: "Cite claims using ONLY the reference labels [S1]…[Sk] and [F1]…[Fj] provided in the context. Do not invent page numbers or section titles."
+- `CitationValidatorService` scans generated text for `[Sx]` / `[Fx]` patterns, checks each against the known label set, and removes unknown ones.
+- `Message.sources` is already a `List<Map<String,Object>>`. No schema change needed — but we store the ref-label alongside each source so the frontend can correlate.
+
+---
+
+## Decision 3: Frontend Source Display
+
+**Decision**: No frontend change needed for MVP. The backend already filters out hallucinated citations via `CitationValidatorService` before saving the message. The `sources` list attached to the message is already the retrieved set. The existing `ChatMessage.vue` source panel continues to show all retrieved sources (which is correct — they were all used as context).
+
+**Rationale**: KISS. The critical correctness fix (no hallucinated citations in answer text) is fully backend-side. The sources panel shows context that was genuinely available to the model — that is accurate and useful. Linking specific claims to specific sources (inline highlighting) is a UX enhancement that can be a follow-on feature.
+
+**Alternatives considered**:
+- *Inline citation links*: Highlight `[S1]` in answer text and link to the source card. Better UX but requires markdown parsing and component changes — deferred.
+- *Show only cited sources*: Filter `sources` to only those actually cited in the answer. Marginally more accurate but the uncited context is still legitimately retrieved — deferred.
+
+---
+
+## Decision 4: Persisting Topic Summaries
+
+**Decision**: Add a new `topic_summary` table that stores each generated summary. Summaries are numbered sequentially per topic (summary #1, #2, …). The POST endpoint continues to generate and now also persists. A new GET endpoint lists saved summaries for a topic.
+
+**Rationale**: The simplest approach — one new table, one new repository, minimal changes to the existing service. No caching layer, no event system. The sequential number is derived at query time via `ROW_NUMBER` or counted in the repository, keeping the schema minimal.
+
+**Alternatives considered**:
+- *Store in frontend state only (session memory)*: Already the current state — summaries are lost on reload. Rejected because the user explicitly wants persistence.
+- *Store summary_number as a persisted column*: Avoids a query-time count but risks gaps/duplicates on concurrent writes. For a POC with single-user use, a `SELECT COUNT(*) + 1` approach at insert time is sufficient.
+- *Versioning / soft-delete*: Overkill for a POC where the user just wants to re-read old summaries. No delete endpoint needed initially.
+
+**Implementation note**:
+- New Flyway migration `V6__topic_summary.sql`.
+- `TopicSummaryEntity` (JPA `@Entity` → table `topic_summary`): `id` (UUID), `topicId` (VARCHAR FK), `summaryNumber` (INT), `summary` (TEXT), `sourcesJson` (TEXT — JSON array), `generatedAt` (TIMESTAMP).
+- `summaryNumber` set at insert time: `COUNT(*) WHERE topic_id = ?` + 1.
+- `TopicSummaryRepository extends JpaRepository<TopicSummaryEntity, UUID>` with a `findByTopicIdOrderBySummaryNumberAsc` query.
+- `TopicSummaryService.generateSummary()` saves the result before returning it.
+- `TopicController` gains `GET /api/v1/topics/{id}/summaries` returning `List<SavedSummaryResponse>` (id, summaryNumber, generatedAt — no full text, for list efficiency) and `GET /api/v1/topics/{id}/summaries/{summaryId}` for the full detail.
+- Frontend: when a topic card is clicked, fetch the summary list. Show "Summary #1", "Summary #2" chips + "Generate New" button. Clicking a chip loads the full summary.

specs/004-rag-retrieval-quality/spec.md

@@ -0,0 +1,111 @@
+# Feature Specification: RAG Retrieval Quality Improvements
+
+**Feature Branch**: `004-rag-retrieval-quality`  
+**Created**: 2026-04-06  
+**Status**: Draft  
+**Input**: User description: "I want to enhance the current RAG system, to avoid common pitfalls like: Vocabulary mismatch in retrieval, where the user's language doesn't overlap with the documentation's terminology and Citation errors in generation, where the model cites a chunk that either wasn't retrieved or doesn't support the specific claim being made"
+
+## Overview
+
+The AI Teacher's question-answering system retrieves relevant content from neurosurgery documentation and generates answers. Two reliability problems reduce trust in the system:
+
+1. **Vocabulary mismatch**: A student asking "what happens after cutting the skull?" may use everyday language while the documentation uses clinical terms like "craniotomy" — causing relevant passages to be missed entirely.
+2. **Citation hallucination**: The model sometimes references a section or page in its answer that was not actually retrieved or that does not support the specific claim made, misleading students.
+
+This feature improves both the accuracy of what gets retrieved and the integrity of what the model claims as its sources.
+
+## User Scenarios & Testing *(mandatory)*
+
+### User Story 1 - Accurate Retrieval Despite Different Terminology (Priority: P1)
+
+A medical student asks a question using lay or imprecise language. The system bridges the gap between their vocabulary and the technical terminology used in the textbook, returning contextually relevant passages even when there is no word overlap between the question and the document text.
+
+**Why this priority**: Vocabulary mismatch is the most frequent silent failure — the system returns an answer but based on wrong or empty context, so students receive incorrect information without realising it.
+
+**Independent Test**: Ask the system a question using a common synonym or lay term for a concept that appears in the documentation only under a clinical name. Verify that the retrieved passages contain relevant content about that concept.
+
+**Acceptance Scenarios**:
+
+1. **Given** a question uses a lay term (e.g., "brain swelling"), **When** the system retrieves content, **Then** it returns passages discussing the medically indexed equivalent ("cerebral edema") even though that phrase was not in the question.
+2. **Given** a question uses an acronym the documentation spells out fully, **When** retrieval runs, **Then** relevant passages are still found.
+3. **Given** a highly technical question that perfectly matches documentation language, **When** retrieval runs, **Then** quality does not regress compared to current behavior.
+
+---
+
+### User Story 2 - Grounded Citation in Generated Answers (Priority: P1)
+
+When the model produces an answer and references a source (section, page, or figure), that source must have been part of the retrieved context and must genuinely support the specific claim being cited.
+
+**Why this priority**: A hallucinated citation is worse than no citation — it gives students false confidence that a claim is documented when it is not.
+
+**Independent Test**: Submit a query, capture the retrieved context passed to the model, and verify that every source reference in the generated answer maps to an identifier present in that context.
+
+**Acceptance Scenarios**:
+
+1. **Given** retrieved context contains sections A, B, and C, **When** the model generates an answer citing "Section D", **Then** that citation is either removed or flagged before the answer is shown to the user.
+2. **Given** a retrieved passage is used as context, **When** the model cites it, **Then** the cited passage actually contains the information the citation supports.
+3. **Given** no relevant context was retrieved for part of a query, **When** the model responds, **Then** it acknowledges uncertainty rather than fabricating a source.
+
+---
+
+### User Story 3 - User Visibility into Retrieval Confidence (Priority: P2)
+
+A student can see which parts of the answer are well-supported by the retrieved material and which parts carry lower confidence, allowing them to judge how much to rely on each claim.
+
+**Why this priority**: Transparency builds appropriate trust — students should know when the system is uncertain rather than presenting all answers with equal authority.
+
+**Independent Test**: Ask a question where only partial relevant content exists in the corpus. Verify the answer visually differentiates well-supported claims from lower-confidence statements.
+
+**Acceptance Scenarios**:
+
+1. **Given** an answer contains a claim backed by a directly retrieved passage, **When** displayed, **Then** the source is shown alongside the claim.
+2. **Given** an answer contains a claim not directly covered by retrieved content, **When** displayed, **Then** the system signals lower confidence or absence of a source for that claim.
+
+---
+
+### Edge Cases
+
+- What happens when query enrichment produces terms that retrieve completely unrelated passages?
+- How does the system behave when the retrieved context is very short or empty?
+- What if two retrieved sections contradict each other — how does citation work in that case?
+- What if citation verification removes all citations from an answer (model cited nothing valid)?
+- How does the system handle questions in languages other than the documentation language?
+
+## Requirements *(mandatory)*
+
+### Functional Requirements
+
+- **FR-001**: The system MUST enrich user queries to bridge vocabulary gaps before retrieval, using the domain context of the book being queried.
+- **FR-002**: The system MUST retrieve passages based on the enriched query, not solely on the literal user input.
+- **FR-003**: The system MUST pass only verified, retrieved passage identifiers to the generation step as eligible citation targets.
+- **FR-004**: The system MUST validate that every source reference in a generated answer corresponds to a passage present in the retrieved context.
+- **FR-005**: The system MUST suppress or flag any citation in the generated answer that refers to a passage not present in the retrieved context.
+- **FR-006**: The system MUST surface to the user which retrieved passages were used to support each claim (or indicate absence of supporting source).
+- **FR-007**: When no relevant passages are retrieved, the system MUST communicate this clearly rather than generating an unsupported answer.
+- **FR-008**: Query enrichment MUST be scoped to the active book to avoid introducing terminology from unrelated domains.
+
+### Key Entities
+
+- **Enriched Query**: The augmented version of the user's original question, including synonyms, alternate phrasings, or domain-aligned terms used for retrieval.
+- **Retrieved Context**: The set of passages (sections and figures) returned by retrieval, each identified by a unique source reference, passed to generation.
+- **Citation**: A reference in the generated answer to a specific source; must be traceable to a member of the Retrieved Context.
+- **Citation Validation Result**: A per-citation judgment of whether the cited source was retrieved and supports the claim.
+
+## Success Criteria *(mandatory)*
+
+### Measurable Outcomes
+
+- **SC-001**: Questions using common lay synonyms for clinical terms retrieve relevant passages at least 80% of the time (verified on a manually curated test set of synonym-query pairs).
+- **SC-002**: Zero citations appear in generated answers that reference a source not present in the retrieved context passed to the model.
+- **SC-003**: For every generated answer, 100% of displayed citations can be traced to a retrieved passage identifier.
+- **SC-004**: Retrieval quality for queries that already match documentation vocabulary does not degrade (baseline score maintained or improved).
+- **SC-005**: Users can identify, for each factual claim in an answer, whether a supporting source was found — without needing to ask a follow-up question.
+
+## Assumptions
+
+- The existing retrieval pipeline returns section and figure identifiers that can be used as citation anchors.
+- Query enrichment operates at query time (not ingestion time), so no changes to stored embeddings are needed.
+- The generation model can be instructed via prompt to restrict citations to a provided list of identifiers.
+- Citation validation is performed after generation but before the answer is shown to the user (post-processing step).
+- Mobile or offline support is out of scope for this feature.
+- Multi-language support (non-English questions against English documentation) is a future concern and not addressed here.

specs/004-rag-retrieval-quality/tasks.md

@@ -0,0 +1,250 @@
+# Tasks: RAG Retrieval Quality Improvements
+
+**Input**: Design documents from `/specs/004-rag-retrieval-quality/`
+**Prerequisites**: plan.md ✅, spec.md ✅, research.md ✅, data-model.md ✅, contracts/ ✅
+
+**Organization**: Tasks are grouped by user story to enable independent implementation and testing of each story.
+**Tests**: Not requested in spec — no test tasks generated.
+
+## Format: `[ID] [P?] [Story] Description`
+
+- **[P]**: Can run in parallel (different files, no dependencies)
+- **[Story]**: Which user story this task belongs to (US1, US2, US3)
+
+---
+
+## Phase 1: Setup (Shared Infrastructure)
+
+**Purpose**: No new project structure needed — services are added to the existing `retrieval/` package. This phase is a single verification step.
+
+- [x] T001 Verify active branch is `004-rag-retrieval-quality` and `backend/src/main/java/com/aiteacher/retrieval/` exists
+
+---
+
+## Phase 2: Foundational (Blocking Prerequisites)
+
+**Purpose**: Two lightweight value objects shared by both user stories.
+
+**⚠️ CRITICAL**: Both user story phases depend on these records being present.
+
+- [x] T002 Create `ExpandedQuery` record in `backend/src/main/java/com/aiteacher/retrieval/ExpandedQuery.java` with fields `String original` and `String rewritten`
+- [x] T003 [P] Create `LabelledContext` record in `backend/src/main/java/com/aiteacher/retrieval/LabelledContext.java` with fields `Map<String, SectionEntity> sectionLabels`, `Map<String, FigureEntity> figureLabels`, and `String promptText`
+
+**Checkpoint**: Foundation ready — US1 and US2 implementation can begin in parallel
+
+---
+
+## Phase 3: User Story 1 — Accurate Retrieval Despite Different Terminology (Priority: P1) 🎯 MVP
+
+**Goal**: Before each retrieval call, rewrite the user's question into clinical terminology so that vector search finds relevant sections even when the user uses lay language.
+
+**Independent Test**: Ask "what happens after cutting the skull?" — verify retrieved sections contain content about craniotomy without that word appearing in the query.
+
+### Implementation for User Story 1
+
+- [x] T004 [US1] Create `QueryExpansionService` in `backend/src/main/java/com/aiteacher/retrieval/QueryExpansionService.java`:
+  - Constructor-inject `ChatClient`
+  - Method `expand(String query): ExpandedQuery`
+  - LLM prompt: *"Rewrite the following question using precise medical/surgical terminology as it would appear in a neurosurgery textbook index. Output only the rewritten question, nothing else. Question: {query}"*
+  - Return `new ExpandedQuery(query, rewrittenText)`
+  - Annotate with `@Service`
+
+- [x] T005 [US1] Modify `NeurosurgeryRetriever.retrieve()` in `backend/src/main/java/com/aiteacher/retrieval/NeurosurgeryRetriever.java`:
+  - Change method signature from `retrieve(String query, UUID bookId)` to `retrieve(String query, UUID bookId)` — no signature change; just use `query` for vector search (already correct; no change needed here unless query is pre-expanded by caller)
+  - *Note*: expansion is done in ChatService before calling retrieve, so no change to NeurosurgeryRetriever is required
+
+- [x] T006 [US1] Modify `ChatService` in `backend/src/main/java/com/aiteacher/chat/ChatService.java`:
+  - Constructor-inject `QueryExpansionService`
+  - In `sendMessage()`, call `queryExpansionService.expand(fullQuestion)` before the retrieval loop
+  - Pass `expandedQuery.rewritten()` to `retriever.retrieve()` instead of `fullQuestion`
+  - Keep passing `fullQuestion` (original) to `buildContextPrompt()` so the QUESTION block shown to the model reflects what the user actually asked
+
+**Checkpoint**: User Story 1 fully functional — retrieval now uses clinically rewritten queries
+
+---
+
+## Phase 4: User Story 2 — Grounded Citation in Generated Answers (Priority: P1)
+
+**Goal**: Tag all retrieved sections and figures with short ref-labels (`[S1]`, `[F1]`…) in the prompt, instruct the model to cite only those labels, then post-process the answer to strip any citation referencing a label that was not provided.
+
+**Independent Test**: Trigger a question where only sections S1–S3 are retrieved. Verify the generated answer contains no citation outside that set, and the `sources` list in the response carries `refLabel` fields.
+
+### Implementation for User Story 2
+
+- [x] T007 [US2] Create `CitationValidatorService` in `backend/src/main/java/com/aiteacher/retrieval/CitationValidatorService.java`:
+  - Annotate with `@Service`
+  - Method `validate(String generatedAnswer, Set<String> validLabels): String`
+  - Scan `generatedAnswer` for occurrences of `[Sn]` and `[Fn]` patterns using a regex like `\[(S|F)\d+\]`
+  - Remove (or replace with empty string) any match whose label is not in `validLabels`
+  - Return the cleaned answer text
+
+- [x] T008 [US2] Modify `ChatService.buildContextPrompt()` in `backend/src/main/java/com/aiteacher/chat/ChatService.java`:
+  - Change signature to return `LabelledContext` instead of `String`
+  - Assign sequential labels: sections get `S1`, `S2`, …; figures get `F1`, `F2`, …
+  - Prefix each section block with its label: `[S1] Section Title, p.N\n{fullText}\n\n`
+  - Prefix each figure line with its label: `[F1] Fig. X (p.N): caption`
+  - Populate `sectionLabels` and `figureLabels` maps in the returned `LabelledContext`
+  - Store the full formatted prompt in `LabelledContext.promptText()`
+
+- [x] T009 [US2] Update system prompt constant in `backend/src/main/java/com/aiteacher/chat/ChatService.java`:
+  - Replace the citation rule *"Cite sources for each major point (book title and page number from the context)"* with: *"Cite claims using ONLY the reference labels provided in the context (e.g. [S1], [F2]). Do not invent page numbers, section titles, or labels not present in the CONTEXT block."*
+
+- [x] T010 [US2] Wire `CitationValidatorService` into `ChatService.sendMessage()` in `backend/src/main/java/com/aiteacher/chat/ChatService.java`:
+  - Constructor-inject `CitationValidatorService`
+  - After the `chatClient.prompt()...call().content()` call, pass `assistantContent` and the label set from `LabelledContext` to `citationValidatorService.validate()`
+  - Use the validated string as `assistantContent` going forward
+
+- [x] T011 [US2] Modify `buildSources()` in `backend/src/main/java/com/aiteacher/chat/ChatService.java`:
+  - Accept the `LabelledContext` (or its two maps) as an additional parameter
+  - Add `"refLabel"` entry to each source map: e.g. `source.put("refLabel", "S1")` for sections, `source.put("refLabel", "F1")` for figures
+  - Keep all other existing fields unchanged
+
+- [x] T012 [US2] Update `sendMessage()` call chain in `backend/src/main/java/com/aiteacher/chat/ChatService.java` to thread `LabelledContext` through steps T008–T011:
+  - `LabelledContext ctx = buildContextPrompt(fullQuestion, allSections, allFigures)`
+  - Pass `ctx.promptText()` to the LLM call
+  - Pass `ctx` label maps to `validate()` and `buildSources()`
+
+**Checkpoint**: User Stories 1 and 2 both fully functional — queries are expanded, citations are grounded
+
+---
+
+## Phase 5: User Story 3 — User Visibility into Retrieval Confidence (Priority: P2)
+
+**Goal**: The answer text contains `[S1]`-style labels (after US2). This phase exposes them in the frontend so users can see which claim maps to which source card.
+
+**Independent Test**: Send a question, receive an answer with inline `[S1]` labels visible in the rendered text, and confirm clicking/hovering the label highlights the corresponding source card.
+
+**Note**: Per research.md, the backend is already complete after US1+US2. US3 is a frontend-only UX enhancement.
+
+### Implementation for User Story 3
+
+- [x] T013 [US3] Modify `ChatMessage.vue` in `frontend/src/components/ChatMessage.vue`:
+  - Parse the answer text for `[Sn]` and `[Fn]` citation labels using a regex
+  - Render each label as a styled inline badge (e.g. `<span class="citation-badge">[S1]</span>`)
+  - When a badge is clicked or hovered, highlight the corresponding source card (match by `source.refLabel`)
+
+- [x] T014 [US3] Update source card rendering in `frontend/src/components/ChatMessage.vue`:
+  - Add a `data-ref-label` attribute to each source card element so it can be targeted by the citation badge interaction
+  - Apply a visual highlight style (CSS class) when the card is active
+
+**Checkpoint**: All three user stories functional — full end-to-end quality improvements delivered
+
+---
+
+## Phase 6: User Story 4 — Topic Summary Persistence & History (user-requested)
+
+**Goal**: Every generated topic summary is saved to the database. When a topic is selected the UI shows a numbered history list; the student can view any past summary or generate a new one.
+
+**Independent Test**: Generate a summary for "Intracranial Aneurysms", reload the page, click the topic — verify "Summary #1" appears. Generate again — verify "Summary #2" appears. Click "Summary #1" — verify the original text loads without regeneration.
+
+- [x] T018 Create Flyway migration `backend/src/main/resources/db/migration/V6__topic_summary.sql` — table `topic_summary` with columns: `id UUID PRIMARY KEY DEFAULT gen_random_uuid()`, `topic_id VARCHAR(100) NOT NULL`, `summary_number INT NOT NULL`, `summary TEXT NOT NULL`, `sources_json TEXT NOT NULL`, `generated_at TIMESTAMPTZ NOT NULL`
+- [x] T019 [P] [US4] Create `TopicSummaryEntity.java` in `backend/src/main/java/com/aiteacher/topic/TopicSummaryEntity.java` — JPA `@Entity` mapped to table `topic_summary`; fields: `@Id UUID id`, `String topicId`, `int summaryNumber`, `String summary`, `String sourcesJson`, `Instant generatedAt`; no-arg + all-args constructor
+- [x] T02X [P] [US4] Create `SavedSummaryItem.java` record in `backend/src/main/java/com/aiteacher/topic/SavedSummaryItem.java` — fields: `UUID id`, `int summaryNumber`, `Instant generatedAt` (list-view DTO, no full text)
+- [x] T02X [US4] Create `TopicSummaryRepository.java` in `backend/src/main/java/com/aiteacher/topic/TopicSummaryRepository.java` — `extends JpaRepository<TopicSummaryEntity, UUID>`; add `List<TopicSummaryEntity> findByTopicIdOrderBySummaryNumberAsc(String topicId)` and `long countByTopicId(String topicId)`
+- [x] T02X [US4] Modify `TopicSummaryResponse.java` in `backend/src/main/java/com/aiteacher/topic/TopicSummaryResponse.java` — add fields `UUID id` and `int summaryNumber` to the record components
+- [x] T02X [US4] Modify `TopicSummaryService.java` in `backend/src/main/java/com/aiteacher/topic/TopicSummaryService.java` — inject `TopicSummaryRepository` and `ObjectMapper`; at end of `generateSummary()` compute `summaryNumber = (int) repository.countByTopicId(topicId) + 1`, persist a `TopicSummaryEntity` (serialise `sources` list to JSON via `objectMapper.writeValueAsString()`), and include `id` + `summaryNumber` in the returned `TopicSummaryResponse`; add `List<SavedSummaryItem> listSummaries(String topicId)` and `TopicSummaryResponse getSummary(UUID summaryId)` methods
+- [x] T02X [US4] Modify `TopicController.java` in `backend/src/main/java/com/aiteacher/topic/TopicController.java` — add `@GetMapping("/{id}/summaries")` returning `List<SavedSummaryItem>` (delegates to `listSummaries`); add `@GetMapping("/{id}/summaries/{summaryId}")` returning `TopicSummaryResponse` (delegates to `getSummary`); both return 404 via `NoSuchElementException` when topic or summary not found
+- [x] T02X [US4] Modify `topicStore.ts` in `frontend/src/stores/topicStore.ts` — add state `summaryList: SavedSummaryItem[]`; add `fetchSummaries(topicId)` action calling `GET /api/v1/topics/{topicId}/summaries`; add `fetchSummaryDetail(topicId, summaryId)` action calling `GET /api/v1/topics/{topicId}/summaries/{summaryId}` and setting `activeSummary`; clear `summaryList` when a different topic is selected
+- [x] T02X [US4] Modify `TopicsView.vue` in `frontend/src/views/TopicsView.vue` — when a topic card is clicked: (1) call `topicStore.fetchSummaries(topicId)` first; (2) if summaries exist, display a summary history list showing chips "Summary #1 · [date]", "Summary #2 · [date]", … + a "Generate New" button; (3) clicking a chip calls `fetchSummaryDetail()` and renders the saved summary in the existing panel; (4) clicking "Generate New" calls `handleGenerate()` then re-calls `fetchSummaries()` to refresh the list; (5) if no summaries exist, show only the "Generate Summary" button (current behaviour)
+
+**Checkpoint**: Summary persistence fully working end-to-end. US4 independently testable.
+
+---
+
+## Phase 7: Polish & Cross-Cutting Concerns
+
+**Purpose**: Constitution IV compliance and cleanup.
+
+- [x] T027 Update `README.md` Mermaid architecture diagram to add `QueryExpansionService` and `CitationValidatorService` to the chat pipeline flow, and the `topic_summary` table to the data diagram (required by Constitution Principle IV — must be in the same PR)
+- [x] T028 [P] Log the expanded query at DEBUG level in `QueryExpansionService` (e.g. `log.debug("Query expanded: '{}' → '{}'", original, rewritten)`) for observability
+- [x] T029 [P] Log stripped citation labels at WARN level in `CitationValidatorService` when any labels are removed (e.g. `log.warn("Stripped hallucinated citations: {}", removedLabels)`)
+
+---
+
+## Dependencies & Execution Order
+
+### Phase Dependencies
+
+- **Phase 1 (Setup)**: No dependencies — start immediately
+- **Phase 2 (Foundational)**: Depends on Phase 1 — blocks all user story phases
+- **Phase 3 (US1)**: Depends on Phase 2 (needs `ExpandedQuery`)
+- **Phase 4 (US2)**: Depends on Phase 2 (needs `LabelledContext`); can run in parallel with Phase 3
+- **Phase 5 (US3)**: Depends on Phase 4 (needs `refLabel` in sources)
+- **Phase 6 (US4)**: No dependency on Phase 2 for the migration (T018); entity/service work (T019+) depends on T018
+- **Phase 7 (Polish)**: Depends on all implementation phases complete
+
+### User Story Dependencies
+
+- **User Story 1 (P1)**: Depends on Phase 2 only — no dependency on US2 or US3
+- **User Story 2 (P1)**: Depends on Phase 2 only — can run in parallel with US1
+- **User Story 3 (P2)**: Depends on US2 (needs `refLabel` in the API response)
+- **User Story 4**: Independent of US1–US3 — can start immediately after T018 migration
+
+### Within Each User Story
+
+- T004 → T006 (QueryExpansionService must exist before ChatService wiring)
+- T007 → T010 → T012 (CitationValidatorService → wire into sendMessage → thread context)
+- T008 → T012 (LabelledContext must be built before threading through)
+- T013 → T014 (badge rendering before card targeting)
+
+### Parallel Opportunities
+
+- T002 and T003 (Phase 2) can run in parallel — different files
+- Phase 3 (US1) and Phase 4 (US2) can run in parallel after Phase 2 — all different files
+- T015, T016, T017 (Polish) can run in parallel — different files
+
+---
+
+## Parallel Example: US1 + US2
+
+```
+After Phase 2 completes:
+
+Track A (US1):
+  T004 — Create QueryExpansionService
+  T005 — (no change to NeurosurgeryRetriever)
+  T006 — Wire into ChatService
+
+Track B (US2):
+  T007 — Create CitationValidatorService
+  T008 — Modify buildContextPrompt() → LabelledContext
+  T009 — Update system prompt
+  T010 — Wire CitationValidatorService into sendMessage()
+  T011 — Add refLabel to buildSources()
+  T012 — Thread LabelledContext through call chain
+
+Merge point: Both tracks modify ChatService — coordinate T006 and T012
+to avoid conflicts (implement T006 first or use feature branches).
+```
+
+---
+
+## Implementation Strategy
+
+### MVP First (User Stories 1 + 2 — both P1)
+
+1. Complete Phase 1: Setup (T001)
+2. Complete Phase 2: Foundational (T002, T003)
+3. Complete Phase 3: US1 — query expansion (T004–T006)
+4. **VALIDATE**: Ask a lay-language question; confirm relevant clinical passages are retrieved
+5. Complete Phase 4: US2 — citation grounding (T007–T012)
+6. **VALIDATE**: Confirm no `[Sx]` label appears in the answer that wasn't in the retrieved set
+7. **STOP and DEMO**: Both P1 stories deliver the core reliability improvements
+
+### Incremental Delivery
+
+1. Phase 1 + 2 → infrastructure ready
+2. Phase 3 → vocabulary mismatch fixed → demo-able
+3. Phase 4 → citation hallucination fixed → demo-able
+4. Phase 5 → citation badges in UI → UX polish
+5. Phase 6 → README + logging → PR-ready
+
+---
+
+## Notes
+
+- `ChatService` is modified by both US1 (T006) and US2 (T008–T012) — coordinate edits or implement sequentially
+- `buildContextPrompt()` changes return type from `String` to `LabelledContext` (T008) — update all callers in the same task
+- The system prompt change (T009) is a one-line string edit inside `ChatService`; no separate class needed
+- `CitationValidatorService` operates purely on strings — no DB or AI dependency, easy to unit-test manually
+- US3 frontend tasks (T013–T014) are entirely in `ChatMessage.vue` — no backend change