enhance rag retrieval + summary

This commit is contained in:
Adrien
2026-04-07 22:39:28 +02:00
parent 0cf318f0a7
commit aee6a9dfba
34 changed files with 2306 additions and 279 deletions
+7 -3
View File
@@ -1,6 +1,6 @@
# ai-teacher Development Guidelines # ai-teacher Development Guidelines
Auto-generated from all feature plans. Last updated: 2026-04-06 Auto-generated from all feature plans. Last updated: 2026-04-07
## Active Technologies ## Active Technologies
- Java 25 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API (embeddings + chat), PDFBox (via Spring AI PDF reader dependency) (002-image-aware-embedding) - Java 25 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API (embeddings + chat), PDFBox (via Spring AI PDF reader dependency) (002-image-aware-embedding)
@@ -10,6 +10,10 @@ Auto-generated from all feature plans. Last updated: 2026-04-06
- PostgreSQL (JPA + Flyway), pgvector (Spring AI `VectorStore`), S3-compatible (002-image-aware-embedding) - PostgreSQL (JPA + Flyway), pgvector (Spring AI `VectorStore`), S3-compatible (002-image-aware-embedding)
- Java 21 (backend) / TypeScript + Node 20 (frontend) + Spring Boot 4.0.5, Spring Security (already included), Vue 3.4, Vue Router 4.3, Pinia 2.1, Axios 1.7 (003-basic-login) - Java 21 (backend) / TypeScript + Node 20 (frontend) + Spring Boot 4.0.5, Spring Security (already included), Vue 3.4, Vue Router 4.3, Pinia 2.1, Axios 1.7 (003-basic-login)
- No new storage — credentials held in browser `sessionStorage` (frontend only) (003-basic-login) - No new storage — credentials held in browser `sessionStorage` (frontend only) (003-basic-login)
- Java 21 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API (chat + embeddings), pgvector, Vue 3.4, Pinia 2.1 (004-rag-retrieval-quality)
- PostgreSQL (sections, figures, messages — unchanged). No new tables needed. (004-rag-retrieval-quality)
- Java 21 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API (chat + embeddings), Vue 3.4, Pinia 2.1, Axios 1.7 (004-rag-retrieval-quality)
- PostgreSQL (JPA + Flyway), pgvector (`VectorStore`) (004-rag-retrieval-quality)
- Java 21 (backend), TypeScript / Node 20 (frontend) (001-neuro-rag-learning) - Java 21 (backend), TypeScript / Node 20 (frontend) (001-neuro-rag-learning)
@@ -29,9 +33,9 @@ npm test && npm run lint
Java 21 (backend), TypeScript / Node 20 (frontend): Follow standard conventions Java 21 (backend), TypeScript / Node 20 (frontend): Follow standard conventions
## Recent Changes ## Recent Changes
- 004-rag-retrieval-quality: Added Java 21 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API (chat + embeddings), Vue 3.4, Pinia 2.1, Axios 1.7
- 004-rag-retrieval-quality: Added Java 21 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API (chat + embeddings), pgvector, Vue 3.4, Pinia 2.1
- 003-basic-login: Added Java 21 (backend) / TypeScript + Node 20 (frontend) + Spring Boot 4.0.5, Spring Security (already included), Vue 3.4, Vue Router 4.3, Pinia 2.1, Axios 1.7 - 003-basic-login: Added Java 21 (backend) / TypeScript + Node 20 (frontend) + Spring Boot 4.0.5, Spring Security (already included), Vue 3.4, Vue Router 4.3, Pinia 2.1, Axios 1.7
- 002-image-aware-embedding: Added Java 25 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API (embeddings +
- 002-image-aware-embedding: Added Java 25 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API, PDFBox (rendering only), `com.google.cloud:google-cloud-documentai` (~2.40.x)
<!-- MANUAL ADDITIONS START --> <!-- MANUAL ADDITIONS START -->
+9 -2
View File
@@ -43,18 +43,25 @@ graph TD
end end
subgraph "Retrieval Pipeline (per chat query)" subgraph "Retrieval Pipeline (per chat query)"
RP0["Query expansion\n(QueryExpansionService)\nlay → clinical terms"]
RP1["Text chunk search (topK=5)"] RP1["Text chunk search (topK=5)"]
RP2["Figure caption search (topK=3)"] RP2["Figure caption search (topK=3)"]
RP3["Expand chunks → full section text"] RP3["Expand chunks → ±1-page section text"]
RP4["Fetch linked figures (chunk_figure_ref)"] RP4["Fetch linked figures (chunk_figure_ref)"]
RP5["Merge + deduplicate figures"] RP5["Merge + deduplicate figures"]
RP6["Build LLM prompt + call"] RP6["Build labelled prompt\n[S1],[F1]… tags"]
RP7["LLM chat call"]
RP8["Citation validation\n(CitationValidatorService)\nstrip hallucinated refs"]
RP0 --> RP1
RP0 --> RP2
RP1 --> RP3 RP1 --> RP3
RP1 --> RP4 RP1 --> RP4
RP2 --> RP5 RP2 --> RP5
RP4 --> RP5 RP4 --> RP5
RP3 --> RP6 RP3 --> RP6
RP5 --> RP6 RP5 --> RP6
RP6 --> RP7
RP7 --> RP8
end end
``` ```
@@ -92,7 +92,7 @@ public class BookEmbeddingService {
ChapterEntity chapter = new ChapterEntity(chapterId, bookId, 1, bookTitle, 1); ChapterEntity chapter = new ChapterEntity(chapterId, bookId, 1, bookTitle, 1);
chapterRepository.save(chapter); chapterRepository.save(chapter);
// Step 1: Parse with Marker — JSON (structured) + Markdown (per-page) in parallel // Step 1: Parse with Marker — split into 100-page chunks, then merge results
ParsedBook parsed = markerPageParser.parse(pdfPath); ParsedBook parsed = markerPageParser.parse(pdfPath);
List<PageResult> pageResults = parsed.pages(); List<PageResult> pageResults = parsed.pages();
@@ -125,25 +125,32 @@ public class BookEmbeddingService {
log.info("Saved {} HTML pages to S3 for book {}", parsed.htmlByPage().size(), bookId); log.info("Saved {} HTML pages to S3 for book {}", parsed.htmlByPage().size(), bookId);
// Step 5: Vision analysis (description + visible text) → embed figure chunks // Step 5: Vision analysis (description + visible text) → embed figure chunks
Map<String, SectionEntity> sectionById = new HashMap<>();
for (SectionEntity s : sections) sectionById.put(s.getId(), s);
for (FigureEntity figure : figures) { for (FigureEntity figure : figures) {
// Prefer caption extracted from the linked section's full text
if (figure.getCaption() == null || figure.getCaption().isBlank()) {
String sectionCaption = extractCaptionFromSection(sectionById.get(figure.getSectionId()));
if (sectionCaption != null) {
figure.setCaption(sectionCaption);
figureRepository.save(figure);
} else {
byte[] imageBytes = figureStorageService.getBytes(figure.getImagePath()); byte[] imageBytes = figureStorageService.getBytes(figure.getImagePath());
VisionDescriptionService.ImageAnalysis analysis = VisionDescriptionService.ImageAnalysis analysis =
visionDescriptionService.analyze(imageBytes, figure.getCaption()); visionDescriptionService.analyze(imageBytes, figure.getCaption());
if (figure.getCaption() == null || figure.getCaption().isBlank()) {
figure.setCaption(analysis.description()); figure.setCaption(analysis.description());
figureRepository.save(figure); figureRepository.save(figure);
} }
}
// Embedding content: description + caption + visible image text // Embedding content: description
String embeddingContent = analysis.description() String embeddingContent = (figure.getCaption() != null ? "\n" + figure.getCaption() : "");
+ (figure.getCaption() != null ? "\n" + figure.getCaption() : "")
+ (analysis.imageText().isEmpty() ? "" : "\n" + analysis.imageText());
String embeddingId = UUID.randomUUID().toString(); String embeddingId = UUID.randomUUID().toString();
if (!skipEmbedding) { if (!skipEmbedding) {
Document figureDoc = new Document(embeddingId, embeddingContent, Document figureDoc = new Document(embeddingId, embeddingContent,
buildFigureMetadata(figure, bookTitle, embeddingId, analysis.imageText())); buildFigureMetadata(figure, bookTitle, embeddingId, ""));
vectorStore.add(List.of(figureDoc)); vectorStore.add(List.of(figureDoc));
figure.setCaptionEmbeddingId(UUID.fromString(embeddingId)); figure.setCaptionEmbeddingId(UUID.fromString(embeddingId));
} }
@@ -163,7 +170,7 @@ public class BookEmbeddingService {
} }
book.setStatus(BookStatus.READY); book.setStatus(BookStatus.READY);
book.setPageCount(sections.size()); book.setPageCount(parsed.htmlByPage().size());
book.setProcessedAt(Instant.now()); book.setProcessedAt(Instant.now());
bookRepository.save(book); bookRepository.save(book);
@@ -210,7 +217,7 @@ public class BookEmbeddingService {
if (page.orderedText().isBlank()) continue; if (page.orderedText().isBlank()) continue;
String sectionId = bookId + "-p" + page.pageNumber(); String sectionId = bookId + "-p" + page.pageNumber();
String title = page.headingTitle() != null ? page.headingTitle() : "Page " + page.pageNumber(); String title = truncate(page.headingTitle() != null ? page.headingTitle() : "Page " + page.pageNumber(), 500);
SectionEntity section = new SectionEntity( SectionEntity section = new SectionEntity(
sectionId, chapterId, bookId, sectionId, chapterId, bookId,
@@ -271,6 +278,17 @@ public class BookEmbeddingService {
return html; return html;
} }
private String extractCaptionFromSection(SectionEntity section) {
if (section == null) return null;
for (String line : section.getFullText().split("\n")) {
String trimmed = line.strip();
if (trimmed.startsWith("Fig.") || trimmed.startsWith("Figure") || trimmed.startsWith("Algorithm")) {
return trimmed;
}
}
return null;
}
private String truncate(String msg, int max) { private String truncate(String msg, int max) {
if (msg == null) return null; if (msg == null) return null;
return msg.length() <= max ? msg : msg.substring(0, max); return msg.length() <= max ? msg : msg.substring(0, max);
@@ -5,10 +5,11 @@ import com.aiteacher.book.BookStatus;
import com.aiteacher.book.NoKnowledgeSourceException; import com.aiteacher.book.NoKnowledgeSourceException;
import com.aiteacher.document.FigureEntity; import com.aiteacher.document.FigureEntity;
import com.aiteacher.document.SectionEntity; import com.aiteacher.document.SectionEntity;
import com.aiteacher.retrieval.CitationValidatorService;
import com.aiteacher.retrieval.LabelledContext;
import com.aiteacher.retrieval.NeurosurgeryRetriever; import com.aiteacher.retrieval.NeurosurgeryRetriever;
import com.aiteacher.retrieval.QueryExpansionService;
import com.aiteacher.retrieval.RetrievalResult; import com.aiteacher.retrieval.RetrievalResult;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.ai.chat.client.ChatClient; import org.springframework.ai.chat.client.ChatClient;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
@@ -17,8 +18,6 @@ import java.util.*;
@Service @Service
public class ChatService { public class ChatService {
private static final Logger log = LoggerFactory.getLogger(ChatService.class);
private static final String SYSTEM_PROMPT = """ private static final String SYSTEM_PROMPT = """
You are an expert neurosurgery educator assistant. Answer questions using the You are an expert neurosurgery educator assistant. Answer questions using the
medical textbook content provided to you as context. medical textbook content provided to you as context.
@@ -29,8 +28,8 @@ public class ChatService {
- Build answers from what is present: procedures, conditions, techniques, and descriptions all contribute; combine them into a rich, structured response - Build answers from what is present: procedures, conditions, techniques, and descriptions all contribute; combine them into a rich, structured response
- Use clear structure: headings, bullet points, or numbered steps where appropriate to maximize clarity - Use clear structure: headings, bullet points, or numbered steps where appropriate to maximize clarity
- Only say you cannot answer if the context is entirely unrelated to the question - Only say you cannot answer if the context is entirely unrelated to the question
- Cite sources for each major point (book title and page number from the context) - Cite sources for each major claim using the reference labels from the context (e.g. [S1], [F2]). Prefer these labels over inventing page numbers, but you may also describe the source naturally if needed.
- When referencing diagrams or figures, cite them as [Fig. X, p.N] - When referencing diagrams or figures, prefer their label from the context (e.g. [F1])
- Maintain continuity with the conversation history - Maintain continuity with the conversation history
- Never fabricate clinical information not present in the context - Never fabricate clinical information not present in the context
"""; """;
@@ -40,17 +39,23 @@ public class ChatService {
private final ChatSessionRepository sessionRepository; private final ChatSessionRepository sessionRepository;
private final MessageRepository messageRepository; private final MessageRepository messageRepository;
private final NeurosurgeryRetriever retriever; private final NeurosurgeryRetriever retriever;
private final QueryExpansionService queryExpansionService;
private final CitationValidatorService citationValidatorService;
public ChatService(ChatClient chatClient, public ChatService(ChatClient chatClient,
BookRepository bookRepository, BookRepository bookRepository,
ChatSessionRepository sessionRepository, ChatSessionRepository sessionRepository,
MessageRepository messageRepository, MessageRepository messageRepository,
NeurosurgeryRetriever retriever) { NeurosurgeryRetriever retriever,
QueryExpansionService queryExpansionService,
CitationValidatorService citationValidatorService) {
this.chatClient = chatClient; this.chatClient = chatClient;
this.bookRepository = bookRepository; this.bookRepository = bookRepository;
this.sessionRepository = sessionRepository; this.sessionRepository = sessionRepository;
this.messageRepository = messageRepository; this.messageRepository = messageRepository;
this.retriever = retriever; this.retriever = retriever;
this.queryExpansionService = queryExpansionService;
this.citationValidatorService = citationValidatorService;
} }
public ChatSession createSession(String topicId) { public ChatSession createSession(String topicId) {
@@ -85,25 +90,34 @@ public class ChatService {
List<Message> history = messageRepository.findBySessionIdOrderByCreatedAtAsc(sessionId); List<Message> history = messageRepository.findBySessionIdOrderByCreatedAtAsc(sessionId);
String fullQuestion = buildQuestionWithHistory(history, userContent, session.getTopicId()); String fullQuestion = buildQuestionWithHistory(history, userContent, session.getTopicId());
// Retrieve context from all ready books (aggregate across books) // Expand only the current user question to clinical terminology for retrieval (US1).
// fullQuestion (which includes conversation history) is used for the LLM context prompt,
// but retrieval should be driven by a concise clinical rewrite of the actual question.
String retrievalQuery = queryExpansionService.expand(userContent).rewritten();
// Retrieve context from all ready books using the expanded query
List<SectionEntity> allSections = new ArrayList<>(); List<SectionEntity> allSections = new ArrayList<>();
List<FigureEntity> allFigures = new ArrayList<>(); List<FigureEntity> allFigures = new ArrayList<>();
for (com.aiteacher.book.Book book : readyBooks) { for (com.aiteacher.book.Book book : readyBooks) {
RetrievalResult result = retriever.retrieve(fullQuestion, book.getId()); RetrievalResult result = retriever.retrieve(retrievalQuery, book.getId());
allSections.addAll(result.parentSections()); allSections.addAll(result.parentSections());
allFigures.addAll(result.figures()); allFigures.addAll(result.figures());
} }
// Build LLM prompt with section full texts and figure references // Build labelled context prompt (US2): assigns [S1]/[F1] labels to each source
String contextPrompt = buildContextPrompt(fullQuestion, allSections, allFigures); LabelledContext ctx = buildContextPrompt(fullQuestion, allSections, allFigures);
String assistantContent = chatClient.prompt() // Generate answer
String rawContent = chatClient.prompt()
.system(SYSTEM_PROMPT) .system(SYSTEM_PROMPT)
.user(contextPrompt) .user(ctx.promptText())
.call() .call()
.content(); .content();
// Build sources list with TEXT and FIGURE entries // Strip any citation labels not present in the retrieved context (US2)
String assistantContent = citationValidatorService.validate(rawContent, ctx.allLabels());
// Attach sources with their ref-labels for frontend traceability
List<Map<String, Object>> sources = buildSources(allSections, allFigures); List<Map<String, Object>> sources = buildSources(allSections, allFigures);
Message assistantMessage = new Message(sessionId, MessageRole.ASSISTANT, assistantContent); Message assistantMessage = new Message(sessionId, MessageRole.ASSISTANT, assistantContent);
@@ -126,51 +140,71 @@ public class ChatService {
// Private helpers // Private helpers
// ------------------------------------------------------------------------- // -------------------------------------------------------------------------
private String buildContextPrompt(String question, /**
* Builds the LLM context prompt, tagging each section as [S1], [S2]… and
* each figure as [F1], [F2]… so the model can cite only known sources.
*/
private LabelledContext buildContextPrompt(String question,
List<SectionEntity> sections, List<SectionEntity> sections,
List<FigureEntity> figures) { List<FigureEntity> figures) {
Map<String, SectionEntity> sectionLabels = new LinkedHashMap<>();
Map<String, FigureEntity> figureLabels = new LinkedHashMap<>();
StringBuilder sb = new StringBuilder(); StringBuilder sb = new StringBuilder();
if (!sections.isEmpty()) { if (!sections.isEmpty()) {
sb.append("CONTEXT:\n\n"); sb.append("CONTEXT:\n\n");
for (SectionEntity section : sections) { for (int i = 0; i < sections.size(); i++) {
sb.append("[").append(section.getTitle()) SectionEntity section = sections.get(i);
.append(", p.").append(section.getPageStart()).append("]\n"); String label = "S" + (i + 1);
sectionLabels.put(label, section);
sb.append("[").append(label).append("] ")
.append(section.getTitle())
.append(", p.").append(section.getPageStart()).append("\n");
sb.append(section.getFullText()).append("\n\n"); sb.append(section.getFullText()).append("\n\n");
} }
} }
if (!figures.isEmpty()) { if (!figures.isEmpty()) {
sb.append("AVAILABLE FIGURES:\n"); sb.append("AVAILABLE FIGURES:\n");
for (FigureEntity figure : figures) { for (int i = 0; i < figures.size(); i++) {
sb.append("- ").append(figure.getLabel() != null ? figure.getLabel() : "Figure") FigureEntity figure = figures.get(i);
String label = "F" + (i + 1);
figureLabels.put(label, figure);
sb.append("[").append(label).append("] ")
.append(figure.getLabel() != null ? figure.getLabel() : "Figure")
.append(" (p.").append(figure.getPage()).append("): ") .append(" (p.").append(figure.getPage()).append("): ")
.append(figure.getCaption() != null ? figure.getCaption() : "") .append(figure.getCaption() != null ? figure.getCaption() : "")
.append("\n"); .append("\n");
} }
sb.append("\nWhen referencing diagrams, cite them as [Fig. X, p.N].\n\n"); sb.append("\nWhen referencing diagrams, use their label from the context (e.g. [F1]).\n\n");
} }
sb.append("QUESTION:\n").append(question); sb.append("QUESTION:\n").append(question);
return sb.toString(); return new LabelledContext(sectionLabels, figureLabels, sb.toString());
} }
private List<Map<String, Object>> buildSources(List<SectionEntity> sections, private List<Map<String, Object>> buildSources(List<SectionEntity> sections,
List<FigureEntity> figures) { List<FigureEntity> figures) {
List<Map<String, Object>> sources = new ArrayList<>(); List<Map<String, Object>> sources = new ArrayList<>();
for (SectionEntity section : sections) { for (int i = 0; i < sections.size(); i++) {
SectionEntity section = sections.get(i);
Map<String, Object> source = new LinkedHashMap<>(); Map<String, Object> source = new LinkedHashMap<>();
source.put("type", "TEXT"); source.put("type", "TEXT");
source.put("refLabel", "S" + (i + 1));
source.put("bookId", section.getBookId());
source.put("bookTitle", deriveTitleFromSection(section)); source.put("bookTitle", deriveTitleFromSection(section));
source.put("page", section.getPageStart()); source.put("page", section.getPageStart());
source.put("chunkText", truncate(section.getFullText(), 500)); source.put("chunkText", truncate(section.getFullText(), 500));
sources.add(source); sources.add(source);
} }
for (FigureEntity figure : figures) { for (int i = 0; i < figures.size(); i++) {
FigureEntity figure = figures.get(i);
Map<String, Object> source = new LinkedHashMap<>(); Map<String, Object> source = new LinkedHashMap<>();
source.put("type", "FIGURE"); source.put("type", "FIGURE");
source.put("refLabel", "F" + (i + 1));
source.put("bookId", figure.getBookId());
source.put("bookTitle", bookRepository.findById(figure.getBookId()) source.put("bookTitle", bookRepository.findById(figure.getBookId())
.map(com.aiteacher.book.Book::getTitle).orElse("Book")); .map(com.aiteacher.book.Book::getTitle).orElse("Book"));
source.put("page", figure.getPage()); source.put("page", figure.getPage());
@@ -178,7 +212,6 @@ public class ChatService {
source.put("label", figure.getLabel() != null ? figure.getLabel() : ""); source.put("label", figure.getLabel() != null ? figure.getLabel() : "");
source.put("caption", figure.getCaption() != null ? figure.getCaption() : ""); source.put("caption", figure.getCaption() != null ? figure.getCaption() : "");
source.put("figureType", figure.getFigureType().name()); source.put("figureType", figure.getFigureType().name());
// imageUrl assembled from relative path: figures/{bookId}/{filename}
String filename = figure.getImagePath().substring( String filename = figure.getImagePath().substring(
figure.getImagePath().lastIndexOf('/') + 1); figure.getImagePath().lastIndexOf('/') + 1);
source.put("imageUrl", "/api/v1/figures/" + figure.getBookId() + "/" + filename); source.put("imageUrl", "/api/v1/figures/" + figure.getBookId() + "/" + filename);
@@ -17,6 +17,7 @@ import java.nio.file.Files;
import java.nio.file.Path; import java.nio.file.Path;
import java.util.*; import java.util.*;
/** /**
* Parses a PDF with a single call to the Marker server using {@code output_format=json}. * Parses a PDF with a single call to the Marker server using {@code output_format=json}.
* *
@@ -46,19 +47,65 @@ public class MarkerPageParser {
); );
private static final Set<String> FIGURE_BLOCK_TYPES = Set.of("Figure", "Picture", "FigureGroup", "PictureGroup"); private static final Set<String> FIGURE_BLOCK_TYPES = Set.of("Figure", "Picture", "FigureGroup", "PictureGroup");
private static final int CHUNK_SIZE = 100;
private static final ObjectMapper MAPPER = new ObjectMapper(); private static final ObjectMapper MAPPER = new ObjectMapper();
private final RestClient restClient; private final RestClient restClient;
private final PdfSplitterService pdfSplitterService;
public MarkerPageParser(@Qualifier("markerRestClient") RestClient restClient) { public MarkerPageParser(@Qualifier("markerRestClient") RestClient restClient,
PdfSplitterService pdfSplitterService) {
this.restClient = restClient; this.restClient = restClient;
this.pdfSplitterService = pdfSplitterService;
} }
public ParsedBook parse(Path pdfPath) { /**
log.info("Submitting {} to Marker (json)", pdfPath.getFileName()); * Parses a PDF by splitting it into {@value #CHUNK_SIZE}-page chunks, submitting each
* chunk to Marker individually, and merging the results into a single {@link ParsedBook}.
* Page numbers in the merged result are absolute (1-based across the whole document).
*/
public ParsedBook parse(Path pdfPath) throws IOException {
List<PdfSplitterService.PdfChunk> chunks = pdfSplitterService.split(pdfPath, CHUNK_SIZE);
log.info("Processing {} chunk(s) for {}", chunks.size(), pdfPath.getFileName());
List<PageResult> allPages = new ArrayList<>();
Map<Integer, String> allHtml = new LinkedHashMap<>();
try {
for (int c = 0; c < chunks.size(); c++) {
PdfSplitterService.PdfChunk chunk = chunks.get(c);
log.info("Submitting chunk {}/{} to Marker (page offset {})", c + 1, chunks.size(), chunk.pageOffset());
ParsedBook chunkResult = submitChunk(chunk.tempFile());
// Rebase page numbers from chunk-relative to document-absolute
for (PageResult page : chunkResult.pages()) {
int absolutePage = chunk.pageOffset() + page.pageNumber();
allPages.add(new PageResult(absolutePage, page.orderedText(), page.headingTitle(), page.figures()));
}
chunkResult.htmlByPage().forEach((chunkPage, html) ->
allHtml.put(chunk.pageOffset() + chunkPage, html));
}
} finally {
// Delete temporary chunk files (skip if the chunk is the original PDF)
for (PdfSplitterService.PdfChunk chunk : chunks) {
if (!chunk.tempFile().equals(pdfPath)) {
try { Files.deleteIfExists(chunk.tempFile()); }
catch (IOException e) { log.warn("Could not delete temp chunk {}", chunk.tempFile()); }
}
}
}
log.info("Marker produced {} non-empty pages from {} chunk(s) of {}",
allPages.size(), chunks.size(), pdfPath.getFileName());
return new ParsedBook(allPages, allHtml);
}
/** Submits a single PDF file to Marker and returns the parsed result with chunk-relative page numbers. */
private ParsedBook submitChunk(Path chunkPath) {
MultiValueMap<String, Object> body = new LinkedMultiValueMap<>(); MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
body.add("file", new FileSystemResource(pdfPath)); body.add("file", new FileSystemResource(chunkPath));
body.add("output_format", "json"); body.add("output_format", "json");
JsonNode response = restClient.post() JsonNode response = restClient.post()
@@ -76,28 +123,29 @@ public class MarkerPageParser {
List<JsonNode> pageNodes = extractPages(response); List<JsonNode> pageNodes = extractPages(response);
if (pageNodes.isEmpty()) { if (pageNodes.isEmpty()) {
log.warn("Marker returned no pages for {}", pdfPath.getFileName()); log.warn("Marker returned no pages for chunk {}", chunkPath.getFileName());
return new ParsedBook(List.of(), Map.of()); return new ParsedBook(List.of(), Map.of());
} }
log.info("Marker returned {} pages for {}", pageNodes.size(), pdfPath.getFileName());
List<PageResult> pages = new ArrayList<>(); List<PageResult> pages = new ArrayList<>();
Map<Integer, String> htmlByPage = new LinkedHashMap<>(); Map<Integer, String> htmlByPage = new LinkedHashMap<>();
for (int i = 0; i < pageNodes.size(); i++) { for (int i = 0; i < pageNodes.size(); i++) {
JsonNode pageNode = pageNodes.get(i); JsonNode pageNode = pageNodes.get(i);
int pageNumber = i + 1; // 1-based int pageNumber = i + 1; // 1-based, chunk-relative
PageResult result = buildPageResult(pageNode, pageNumber); PageResult result = buildPageResult(pageNode, pageNumber);
String html = jsonToHtml(pageNode); String html = jsonToHtml(pageNode);
// Always save HTML so the reader can navigate to every page
htmlByPage.put(pageNumber, html);
// Only queue for embedding if the page has extractable content
if (!result.orderedText().isBlank() || !result.figures().isEmpty()) { if (!result.orderedText().isBlank() || !result.figures().isEmpty()) {
pages.add(result); pages.add(result);
htmlByPage.put(pageNumber, html);
} }
} }
log.info("Marker produced {} non-empty pages from {}", pages.size(), pdfPath.getFileName());
return new ParsedBook(pages, htmlByPage); return new ParsedBook(pages, htmlByPage);
} }
@@ -0,0 +1,72 @@
package com.aiteacher.document;
import org.apache.pdfbox.io.RandomAccessReadBufferedFile;
import org.apache.pdfbox.multipdf.Splitter;
import org.apache.pdfbox.pdfparser.PDFParser;
import org.apache.pdfbox.pdmodel.PDDocument;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Service;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
/**
* Splits a PDF file into fixed-size chunks using PDFBox.
* Each chunk is saved as a temporary file so it can be submitted independently to Marker.
*/
@Service
public class PdfSplitterService {
private static final Logger log = LoggerFactory.getLogger(PdfSplitterService.class);
/**
* A chunk of a split PDF.
*
* @param tempFile path to the temporary PDF file (caller must delete when done)
* @param pageOffset 0-based index of the first page in this chunk within the original document
*/
public record PdfChunk(Path tempFile, int pageOffset) {}
/**
* Splits {@code pdfPath} into chunks of at most {@code maxPagesPerChunk} pages.
* Returns a single-element list when the document fits in one chunk.
*
* @param pdfPath source PDF
* @param maxPagesPerChunk maximum pages per chunk
* @return ordered list of chunks; caller is responsible for deleting {@code tempFile}s
*/
public List<PdfChunk> split(Path pdfPath, int maxPagesPerChunk) throws IOException {
try (PDDocument doc = new PDFParser(new RandomAccessReadBufferedFile(pdfPath.toFile())).parse()) {
int totalPages = doc.getNumberOfPages();
log.info("PDF {} has {} pages, splitting into chunks of {}", pdfPath.getFileName(), totalPages, maxPagesPerChunk);
if (totalPages <= maxPagesPerChunk) {
// No split needed — return the original file as a single virtual chunk
return List.of(new PdfChunk(pdfPath, 0));
}
Splitter splitter = new Splitter();
splitter.setSplitAtPage(maxPagesPerChunk);
List<PDDocument> parts = splitter.split(doc);
List<PdfChunk> chunks = new ArrayList<>(parts.size());
int offset = 0;
for (PDDocument part : parts) {
try {
Path tmp = Files.createTempFile("marker-chunk-", ".pdf");
part.save(tmp.toFile());
chunks.add(new PdfChunk(tmp, offset));
log.debug("Created chunk at {} (page offset {})", tmp, offset);
offset += part.getNumberOfPages();
} finally {
part.close();
}
}
return chunks;
}
}
}
@@ -1,6 +1,8 @@
package com.aiteacher.document; package com.aiteacher.document;
import org.springframework.data.jpa.repository.JpaRepository; import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.Query;
import org.springframework.data.repository.query.Param;
import java.util.List; import java.util.List;
import java.util.UUID; import java.util.UUID;
@@ -8,4 +10,10 @@ import java.util.UUID;
public interface SectionRepository extends JpaRepository<SectionEntity, String> { public interface SectionRepository extends JpaRepository<SectionEntity, String> {
List<SectionEntity> findAllByBookId(UUID bookId); List<SectionEntity> findAllByBookId(UUID bookId);
void deleteAllByBookId(UUID bookId); void deleteAllByBookId(UUID bookId);
@Query("SELECT s FROM SectionEntity s WHERE s.bookId = :bookId AND s.pageStart <= :windowEnd AND s.pageEnd >= :windowStart ORDER BY s.pageStart")
List<SectionEntity> findByBookIdAndPageOverlap(
@Param("bookId") UUID bookId,
@Param("windowStart") int windowStart,
@Param("windowEnd") int windowEnd);
} }
@@ -3,6 +3,7 @@ package com.aiteacher.document;
import org.slf4j.Logger; import org.slf4j.Logger;
import org.slf4j.LoggerFactory; import org.slf4j.LoggerFactory;
import org.springframework.ai.chat.client.ChatClient; import org.springframework.ai.chat.client.ChatClient;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.core.io.ByteArrayResource; import org.springframework.core.io.ByteArrayResource;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import org.springframework.util.MimeTypeUtils; import org.springframework.util.MimeTypeUtils;
@@ -32,10 +33,16 @@ public class VisionDescriptionService {
IMAGE_TEXT: <all visible text, labels, measurements, and annotations copied verbatim, comma-separated; write NONE if no text visible> IMAGE_TEXT: <all visible text, labels, measurements, and annotations copied verbatim, comma-separated; write NONE if no text visible>
"""; """;
/** Minimum ms between vision API calls. Configurable via app.vision.min-interval-ms. */
private final long minIntervalMs;
private final ChatClient chatClient; private final ChatClient chatClient;
private volatile long lastCallAt = 0;
public VisionDescriptionService(ChatClient chatClient) { public VisionDescriptionService(
ChatClient chatClient,
@Value("${app.vision.min-interval-ms:2000}") long minIntervalMs) {
this.chatClient = chatClient; this.chatClient = chatClient;
this.minIntervalMs = minIntervalMs;
} }
/** /**
@@ -55,6 +62,7 @@ public class VisionDescriptionService {
* @param captionFallback caption detected from surrounding text, may be null * @param captionFallback caption detected from surrounding text, may be null
*/ */
public ImageAnalysis analyze(byte[] imageBytes, String captionFallback) { public ImageAnalysis analyze(byte[] imageBytes, String captionFallback) {
throttle();
try { try {
String raw = chatClient.prompt() String raw = chatClient.prompt()
.user(u -> u .user(u -> u
@@ -71,6 +79,15 @@ public class VisionDescriptionService {
} }
} }
private synchronized void throttle() {
long now = System.currentTimeMillis();
long wait = minIntervalMs - (now - lastCallAt);
if (wait > 0) {
try { Thread.sleep(wait); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
}
lastCallAt = System.currentTimeMillis();
}
private ImageAnalysis parse(String raw, String captionFallback) { private ImageAnalysis parse(String raw, String captionFallback) {
String description = captionFallback != null ? captionFallback : "Figure"; String description = captionFallback != null ? captionFallback : "Figure";
String imageText = ""; String imageText = "";
@@ -0,0 +1,59 @@
package com.aiteacher.retrieval;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Service;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
/**
* Post-processes generated answers to strip citation labels that do not
* correspond to any passage retrieved for the current query, preventing
* hallucinated source references from reaching the user.
*/
@Service
public class CitationValidatorService {
private static final Logger log = LoggerFactory.getLogger(CitationValidatorService.class);
/** Matches citation labels of the form [S1], [F2], [S12], etc. */
private static final Pattern CITATION_PATTERN = Pattern.compile("\\[(S|F)\\d+\\]");
/**
* Removes any {@code [Sx]} / {@code [Fx]} citation in {@code generatedAnswer}
* whose label is not contained in {@code validLabels}.
*
* @param generatedAnswer raw model output
* @param validLabels set of labels present in the retrieved context
* @return cleaned answer text with hallucinated citations removed
*/
public String validate(String generatedAnswer, Set<String> validLabels) {
if (generatedAnswer == null) return "";
Matcher matcher = CITATION_PATTERN.matcher(generatedAnswer);
List<String> removed = new ArrayList<>();
StringBuffer sb = new StringBuffer();
while (matcher.find()) {
String label = matcher.group();
String inner = label.substring(1, label.length() - 1); // strip [ ]
if (validLabels.contains(inner)) {
matcher.appendReplacement(sb, Matcher.quoteReplacement(label));
} else {
removed.add(inner);
matcher.appendReplacement(sb, "");
}
}
matcher.appendTail(sb);
if (!removed.isEmpty()) {
log.warn("Stripped hallucinated citations: {}", removed);
}
return sb.toString();
}
}
@@ -0,0 +1,7 @@
package com.aiteacher.retrieval;
/**
* Value object holding the original user query alongside its clinically
* rewritten variant used for vector-store retrieval.
*/
public record ExpandedQuery(String original, String rewritten) {}
@@ -0,0 +1,27 @@
package com.aiteacher.retrieval;
import com.aiteacher.document.FigureEntity;
import com.aiteacher.document.SectionEntity;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
/**
* Value object produced when building the LLM context prompt.
* Maps short ref-labels (S1, S2… / F1, F2…) to their source entities
* and carries the fully formatted prompt text.
*/
public record LabelledContext(
Map<String, SectionEntity> sectionLabels,
Map<String, FigureEntity> figureLabels,
String promptText) {
/** Returns the union of all valid citation labels for this context. */
public Set<String> allLabels() {
Set<String> labels = new HashSet<>();
labels.addAll(sectionLabels.keySet());
labels.addAll(figureLabels.keySet());
return labels;
}
}
@@ -0,0 +1,47 @@
package com.aiteacher.retrieval;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.stereotype.Service;
/**
* Rewrites a user query into precise clinical/surgical terminology so that
* vector-store retrieval can match textbook language even when the user's
* phrasing differs from the documentation vocabulary.
*/
@Service
public class QueryExpansionService {
private static final Logger log = LoggerFactory.getLogger(QueryExpansionService.class);
private static final String EXPANSION_PROMPT = """
Rewrite the following question using precise medical and surgical terminology \
as it would appear in a neurosurgery textbook index. \
Output only the rewritten question, nothing else.
Question: %s""";
private final ChatClient chatClient;
public QueryExpansionService(ChatClient chatClient) {
this.chatClient = chatClient;
}
/**
* Returns an {@link ExpandedQuery} whose {@code rewritten} field contains
* the clinically rephrased version of {@code query}.
*/
public ExpandedQuery expand(String query) {
String rewritten = chatClient.prompt()
.user(EXPANSION_PROMPT.formatted(query))
.call()
.content();
if (rewritten == null || rewritten.isBlank()) {
rewritten = query;
}
log.debug("Query expanded: '{}' → '{}'", query, rewritten);
return new ExpandedQuery(query, rewritten);
}
}
@@ -0,0 +1,7 @@
package com.aiteacher.topic;
import java.time.Instant;
import java.util.UUID;
public record SavedSummaryItem(UUID id, int summaryNumber, Instant generatedAt) {
}
@@ -5,6 +5,7 @@ import org.springframework.web.bind.annotation.*;
import java.util.List; import java.util.List;
import java.util.NoSuchElementException; import java.util.NoSuchElementException;
import java.util.UUID;
@RestController @RestController
@RequestMapping("/api/v1/topics") @RequestMapping("/api/v1/topics")
@@ -32,4 +33,21 @@ public class TopicController {
TopicSummaryResponse response = topicSummaryService.generateSummary(topic); TopicSummaryResponse response = topicSummaryService.generateSummary(topic);
return ResponseEntity.ok(response); return ResponseEntity.ok(response);
} }
@GetMapping("/{id}/summaries")
public ResponseEntity<List<SavedSummaryItem>> listSummaries(@PathVariable String id) {
topicRepository.findById(id)
.orElseThrow(() -> new NoSuchElementException("Topic not found."));
return ResponseEntity.ok(topicSummaryService.listSummaries(id));
}
@GetMapping("/{id}/summaries/{summaryId}")
public ResponseEntity<TopicSummaryResponse> getSummary(@PathVariable String id,
@PathVariable UUID summaryId) {
topicRepository.findById(id)
.orElseThrow(() -> new NoSuchElementException("Topic not found."));
return ResponseEntity.ok(topicSummaryService.getSummary(summaryId));
}
} }
@@ -0,0 +1,53 @@
package com.aiteacher.topic;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
import java.time.Instant;
import java.util.UUID;
@Entity
@Table(name = "topic_summary")
public class TopicSummaryEntity {
@Id
@GeneratedValue(strategy = GenerationType.UUID)
private UUID id;
@Column(name = "topic_id", nullable = false)
private String topicId;
@Column(name = "summary_number", nullable = false)
private int summaryNumber;
@Column(nullable = false, columnDefinition = "TEXT")
private String summary;
@Column(name = "sources_json", nullable = false, columnDefinition = "TEXT")
private String sourcesJson;
@Column(name = "generated_at", nullable = false)
private Instant generatedAt;
protected TopicSummaryEntity() {}
public TopicSummaryEntity(String topicId, int summaryNumber, String summary,
String sourcesJson, Instant generatedAt) {
this.topicId = topicId;
this.summaryNumber = summaryNumber;
this.summary = summary;
this.sourcesJson = sourcesJson;
this.generatedAt = generatedAt;
}
public UUID getId() { return id; }
public String getTopicId() { return topicId; }
public int getSummaryNumber() { return summaryNumber; }
public String getSummary() { return summary; }
public String getSourcesJson() { return sourcesJson; }
public Instant getGeneratedAt() { return generatedAt; }
}
@@ -0,0 +1,13 @@
package com.aiteacher.topic;
import org.springframework.data.jpa.repository.JpaRepository;
import java.util.List;
import java.util.UUID;
public interface TopicSummaryRepository extends JpaRepository<TopicSummaryEntity, UUID> {
List<TopicSummaryEntity> findByTopicIdOrderBySummaryNumberAsc(String topicId);
long countByTopicId(String topicId);
}
@@ -2,8 +2,11 @@ package com.aiteacher.topic;
import java.time.Instant; import java.time.Instant;
import java.util.List; import java.util.List;
import java.util.UUID;
public record TopicSummaryResponse( public record TopicSummaryResponse(
UUID id,
int summaryNumber,
String topicId, String topicId,
String topicName, String topicName,
String summary, String summary,
@@ -11,6 +14,7 @@ public record TopicSummaryResponse(
Instant generatedAt Instant generatedAt
) { ) {
public record SourceReference( public record SourceReference(
String bookId,
String bookTitle, String bookTitle,
Integer page Integer page
) { ) {
@@ -1,21 +1,25 @@
package com.aiteacher.topic; package com.aiteacher.topic;
import com.aiteacher.book.Book;
import com.aiteacher.book.BookRepository; import com.aiteacher.book.BookRepository;
import com.aiteacher.book.BookStatus; import com.aiteacher.book.BookStatus;
import com.aiteacher.book.NoKnowledgeSourceException; import com.aiteacher.book.NoKnowledgeSourceException;
import com.aiteacher.document.FigureEntity;
import com.aiteacher.document.SectionEntity;
import com.aiteacher.retrieval.NeurosurgeryRetriever;
import com.aiteacher.retrieval.RetrievalResult;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.slf4j.Logger; import org.slf4j.Logger;
import org.slf4j.LoggerFactory; import org.slf4j.LoggerFactory;
import org.springframework.ai.chat.client.ChatClient; import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.document.Document;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import java.time.Instant; import java.time.Instant;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.List; import java.util.List;
import java.util.Map; import java.util.NoSuchElementException;
import java.util.UUID;
@Service @Service
public class TopicSummaryService { public class TopicSummaryService {
@@ -29,80 +33,190 @@ public class TopicSummaryService {
When answering: When answering:
- Structure your response clearly with key points - Structure your response clearly with key points
- If the context mentions specific book titles and page numbers, reference them - Cite claims using ONLY the reference labels provided in the context (e.g. [S1], [F2]).
Do not invent page numbers, section titles, or labels not present in the CONTEXT block.
- If the retrieved context does not contain sufficient information on the topic, - If the retrieved context does not contain sufficient information on the topic,
explicitly state: "The uploaded books do not contain sufficient information on this topic." explicitly state: "The uploaded books do not contain sufficient information on this topic."
- Never hallucinate or fabricate clinical information - Never hallucinate or fabricate clinical information
"""; """;
private final ChatClient chatClient; private final ChatClient chatClient;
private final VectorStore vectorStore;
private final BookRepository bookRepository; private final BookRepository bookRepository;
private final NeurosurgeryRetriever retriever;
private final TopicSummaryRepository summaryRepository;
private final ObjectMapper objectMapper;
public TopicSummaryService(ChatClient chatClient, VectorStore vectorStore, public TopicSummaryService(ChatClient chatClient,
BookRepository bookRepository) { BookRepository bookRepository,
NeurosurgeryRetriever retriever,
TopicSummaryRepository summaryRepository,
ObjectMapper objectMapper) {
this.chatClient = chatClient; this.chatClient = chatClient;
this.vectorStore = vectorStore;
this.bookRepository = bookRepository; this.bookRepository = bookRepository;
this.retriever = retriever;
this.summaryRepository = summaryRepository;
this.objectMapper = objectMapper;
} }
public TopicSummaryResponse generateSummary(Topic topic) { public TopicSummaryResponse generateSummary(Topic topic) {
if (!bookRepository.existsByStatus(BookStatus.READY)) { List<Book> readyBooks = bookRepository.findAll().stream()
.filter(b -> b.getStatus() == BookStatus.READY)
.toList();
if (readyBooks.isEmpty()) {
throw new NoKnowledgeSourceException( throw new NoKnowledgeSourceException(
"No books are available as knowledge sources. Please upload and process at least one book."); "No books are available as knowledge sources. Please upload and process at least one book.");
} }
String question = buildQuestion(topic); String question = buildQuestion(topic);
ChatResponse response = chatClient.prompt() List<SectionEntity> allSections = new ArrayList<>();
.system(SYSTEM_PROMPT) List<FigureEntity> allFigures = new ArrayList<>();
.advisors(QuestionAnswerAdvisor.builder(vectorStore).build()) for (Book book : readyBooks) {
.user(question) RetrievalResult result = retriever.retrieve(question, book.getId());
.call() allSections.addAll(result.parentSections());
.chatResponse(); allFigures.addAll(result.figures());
}
String summary = response.getResult().getOutput().getText(); log.debug("Topic summary for '{}': {} sections, {} figures retrieved",
List<TopicSummaryResponse.SourceReference> sources = extractSources(response); topic.getName(), allSections.size(), allFigures.size());
String contextPrompt = buildContextPrompt(question, allSections, allFigures);
String summary = chatClient.prompt()
.system(SYSTEM_PROMPT)
.user(contextPrompt)
.call()
.content();
List<TopicSummaryResponse.SourceReference> sources = buildSources(allSections, allFigures, readyBooks);
Instant generatedAt = Instant.now();
int summaryNumber = (int) summaryRepository.countByTopicId(topic.getId()) + 1;
String sourcesJson = serializeSources(sources);
TopicSummaryEntity entity = new TopicSummaryEntity(
topic.getId(), summaryNumber, summary, sourcesJson, generatedAt);
entity = summaryRepository.save(entity);
return new TopicSummaryResponse( return new TopicSummaryResponse(
entity.getId(),
summaryNumber,
topic.getId(), topic.getId(),
topic.getName(), topic.getName(),
summary, summary,
sources, sources,
Instant.now() generatedAt
);
}
public List<SavedSummaryItem> listSummaries(String topicId) {
return summaryRepository.findByTopicIdOrderBySummaryNumberAsc(topicId).stream()
.map(e -> new SavedSummaryItem(e.getId(), e.getSummaryNumber(), e.getGeneratedAt()))
.toList();
}
public TopicSummaryResponse getSummary(UUID summaryId) {
TopicSummaryEntity entity = summaryRepository.findById(summaryId)
.orElseThrow(() -> new NoSuchElementException("Summary not found."));
List<TopicSummaryResponse.SourceReference> sources = deserializeSources(entity.getSourcesJson());
return new TopicSummaryResponse(
entity.getId(),
entity.getSummaryNumber(),
entity.getTopicId(),
entity.getTopicId(),
entity.getSummary(),
sources,
entity.getGeneratedAt()
); );
} }
private String buildQuestion(Topic topic) { private String buildQuestion(Topic topic) {
return String.format( return String.format(
"Please provide a comprehensive educational summary of the following neurosurgery topic: " + "Provide a comprehensive educational summary of the following neurosurgery topic: " +
"%s. Topic description: %s. " + "%s. Topic description: %s. " +
"Include key concepts, clinical considerations, and important details that a neurosurgeon should know.", "Include key concepts, clinical considerations, and important details that a neurosurgeon should know.",
topic.getName(), topic.getDescription() topic.getName(), topic.getDescription()
); );
} }
private List<TopicSummaryResponse.SourceReference> extractSources(ChatResponse response) { private String buildContextPrompt(String question,
List<SectionEntity> sections,
List<FigureEntity> figures) {
StringBuilder sb = new StringBuilder();
if (!sections.isEmpty()) {
sb.append("CONTEXT:\n\n");
for (int i = 0; i < sections.size(); i++) {
SectionEntity s = sections.get(i);
sb.append("[S").append(i + 1).append("] ")
.append(s.getTitle()).append(", p.").append(s.getPageStart()).append("\n");
sb.append(s.getFullText()).append("\n\n");
}
}
if (!figures.isEmpty()) {
sb.append("AVAILABLE FIGURES:\n");
for (int i = 0; i < figures.size(); i++) {
FigureEntity f = figures.get(i);
sb.append("[F").append(i + 1).append("] ")
.append(f.getLabel() != null ? f.getLabel() : "Figure")
.append(" (p.").append(f.getPage()).append("): ")
.append(f.getCaption() != null ? f.getCaption() : "")
.append("\n");
}
sb.append("\n");
}
sb.append("QUESTION:\n").append(question);
return sb.toString();
}
private List<TopicSummaryResponse.SourceReference> buildSources(List<SectionEntity> sections,
List<FigureEntity> figures,
List<Book> readyBooks) {
List<TopicSummaryResponse.SourceReference> sources = new ArrayList<>(); List<TopicSummaryResponse.SourceReference> sources = new ArrayList<>();
if (response.getMetadata() != null) { for (SectionEntity s : sections) {
Object retrieved = response.getMetadata().get(QuestionAnswerAdvisor.RETRIEVED_DOCUMENTS); Book book = readyBooks.stream()
if (retrieved instanceof List<?> docs) { .filter(b -> b.getId().equals(s.getBookId()))
for (Object docObj : docs) { .findFirst()
if (docObj instanceof Document doc) { .orElse(null);
Map<String, Object> metadata = doc.getMetadata(); String title = book != null ? book.getTitle() : "Book";
String bookTitle = (String) metadata.get("book_title"); String bookId = book != null ? book.getId().toString() : null;
Object pageObj = metadata.get("page_number"); sources.add(new TopicSummaryResponse.SourceReference(bookId, title, s.getPageStart()));
Integer page = pageObj instanceof Number n ? n.intValue() : null;
if (bookTitle != null) {
sources.add(new TopicSummaryResponse.SourceReference(bookTitle, page));
} }
for (FigureEntity f : figures) {
Book book = readyBooks.stream()
.filter(b -> b.getId().equals(f.getBookId()))
.findFirst()
.orElse(null);
String title = book != null ? book.getTitle() : "Book";
String bookId = book != null ? book.getId().toString() : null;
sources.add(new TopicSummaryResponse.SourceReference(bookId, title, f.getPage()));
} }
return sources.stream().distinct().toList();
} }
private String serializeSources(List<TopicSummaryResponse.SourceReference> sources) {
try {
return objectMapper.writeValueAsString(sources);
} catch (JsonProcessingException e) {
log.warn("Failed to serialize sources, storing empty array", e);
return "[]";
} }
} }
// Deduplicate by bookTitle + page private List<TopicSummaryResponse.SourceReference> deserializeSources(String json) {
return sources.stream().distinct().toList(); try {
return objectMapper.readValue(json,
objectMapper.getTypeFactory().constructCollectionType(
List.class, TopicSummaryResponse.SourceReference.class));
} catch (JsonProcessingException e) {
log.warn("Failed to deserialize sources from stored JSON", e);
return List.of();
}
} }
} }
+4 -2
View File
@@ -30,7 +30,7 @@ spring:
api-key: ${OPENAI_API_KEY} api-key: ${OPENAI_API_KEY}
chat: chat:
options: options:
model: gpt-4o model: gpt-4o-mini
embedding: embedding:
options: options:
model: "text-embedding-3-small" model: "text-embedding-3-small"
@@ -68,6 +68,8 @@ app:
embedding: embedding:
batch-size: 20 batch-size: 20
batch-delay-ms: 2000 batch-delay-ms: 2000
skip-embedding: true skip-embedding: false
marker: marker:
base-url: ${MARKER_BASE_URL:http://192.168.1.105:8000} base-url: ${MARKER_BASE_URL:http://192.168.1.105:8000}
vision:
min-interval-ms: ${VISION_MIN_INTERVAL_MS:2000}
@@ -0,0 +1,10 @@
CREATE TABLE topic_summary (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
topic_id VARCHAR(100) NOT NULL,
summary_number INT NOT NULL,
summary TEXT NOT NULL,
sources_json TEXT NOT NULL,
generated_at TIMESTAMPTZ NOT NULL
);
CREATE INDEX idx_topic_summary_topic_id ON topic_summary(topic_id, summary_number);
+239
View File
@@ -0,0 +1,239 @@
<template>
<div class="book-panel">
<div class="book-panel-header">
<span class="book-panel-title">{{ bookTitle || 'Book' }} p.&nbsp;{{ page }}</span>
<div class="book-panel-nav">
<button class="nav-btn" :disabled="page <= 1" @click="emit('navigate', page - 1)">&#8592;</button>
<button class="nav-btn" @click="emit('navigate', page + 1)">&#8594;</button>
</div>
<button class="close-btn" @click="emit('close')" title="Close">&#x2715;</button>
</div>
<div class="book-panel-body">
<div v-if="loading" class="panel-loading">
<div class="spinner spinner-dark" style="width:24px;height:24px;margin:0 auto 0.5rem;"></div>
<p>Loading page {{ page }}</p>
</div>
<div v-else-if="error" class="panel-error">{{ error }}</div>
<div v-else class="markdown-body" v-html="renderedHtml"></div>
</div>
</div>
</template>
<script setup lang="ts">
import { ref, watch, onMounted, onUnmounted } from 'vue'
import { api } from '@/services/api'
const props = defineProps<{
bookId: string
page: number
bookTitle?: string
}>()
const emit = defineEmits<{
close: []
navigate: [page: number]
}>()
const loading = ref(false)
const error = ref<string | null>(null)
const renderedHtml = ref('')
let activeBlobUrls: string[] = []
onMounted(() => loadPage(props.page))
watch(() => [props.bookId, props.page], () => loadPage(props.page))
onUnmounted(() => {
activeBlobUrls.forEach(u => URL.revokeObjectURL(u))
})
async function loadPage(page: number) {
loading.value = true
error.value = null
renderedHtml.value = ''
activeBlobUrls.forEach(u => URL.revokeObjectURL(u))
activeBlobUrls = []
try {
const res = await api.get<string>(`/books/${props.bookId}/pages/${page}/html`, {
headers: { Accept: 'text/html' },
responseType: 'text'
})
renderedHtml.value = await resolveImages(res.data)
} catch (e: any) {
error.value = e.message ?? 'Failed to load page.'
} finally {
loading.value = false
}
}
async function resolveImages(html: string): Promise<string> {
const srcPattern = /src="(\/api\/v1\/figures\/[^"]+)"/g
const matches = [...html.matchAll(srcPattern)]
if (matches.length === 0) return html
const unique = [...new Set(matches.map(m => m[1]))]
const blobMap: Record<string, string> = {}
await Promise.all(
unique.map(async (src) => {
try {
const res = await api.get(src.replace(/^\/api\/v1/, ''), { responseType: 'blob' })
const blobUrl = URL.createObjectURL(res.data)
activeBlobUrls.push(blobUrl)
blobMap[src] = blobUrl
} catch {
// leave original src
}
})
)
return html.replace(/src="(\/api\/v1\/figures\/[^"]+)"/g, (_, src) =>
blobMap[src] ? `src="${blobMap[src]}"` : `src="${src}"`
)
}
</script>
<style scoped>
.book-panel {
display: flex;
flex-direction: column;
height: 100%;
background: white;
border-left: 1px solid #e2e8f0;
border-radius: 0 10px 10px 0;
overflow: hidden;
}
.book-panel-header {
display: flex;
align-items: center;
gap: 0.5rem;
padding: 0.6rem 0.75rem;
background: #f7fafc;
border-bottom: 1px solid #e2e8f0;
flex-shrink: 0;
}
.book-panel-title {
flex: 1;
font-size: 0.8rem;
font-weight: 600;
color: #2b6cb0;
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
}
.book-panel-nav {
display: flex;
gap: 0.25rem;
}
.nav-btn {
width: 1.75rem;
height: 1.75rem;
border: 1px solid #cbd5e0;
border-radius: 5px;
background: white;
cursor: pointer;
font-size: 0.85rem;
display: flex;
align-items: center;
justify-content: center;
transition: background 0.15s;
}
.nav-btn:hover:not(:disabled) { background: #ebf8ff; border-color: #3182ce; }
.nav-btn:disabled { opacity: 0.4; cursor: not-allowed; }
.close-btn {
width: 1.75rem;
height: 1.75rem;
border: none;
border-radius: 5px;
background: none;
cursor: pointer;
font-size: 1rem;
color: #718096;
display: flex;
align-items: center;
justify-content: center;
transition: background 0.15s, color 0.15s;
}
.close-btn:hover { background: #fed7d7; color: #742a2a; }
.book-panel-body {
flex: 1;
overflow-y: auto;
padding: 1rem 1.25rem;
}
.panel-loading {
text-align: center;
padding: 2rem;
color: #718096;
font-size: 0.875rem;
}
.panel-error {
padding: 1rem;
background: #fff5f5;
border: 1px solid #fed7d7;
color: #742a2a;
border-radius: 6px;
font-size: 0.875rem;
}
.markdown-body {
font-size: 0.9rem;
line-height: 1.75;
color: #2d3748;
}
.markdown-body :deep(h1),
.markdown-body :deep(h2),
.markdown-body :deep(h3) {
color: #1a365d;
font-weight: 600;
margin: 1.25rem 0 0.5rem;
}
.markdown-body :deep(h2) { font-size: 1.05rem; border-bottom: 1px solid #e2e8f0; padding-bottom: 0.3rem; }
.markdown-body :deep(h3) { font-size: 0.95rem; }
.markdown-body :deep(p) { margin: 0.6rem 0; }
.markdown-body :deep(img) {
max-width: 100%;
border-radius: 6px;
display: block;
margin: 0.75rem auto;
box-shadow: 0 1px 4px rgba(0,0,0,0.12);
}
.markdown-body :deep(ul),
.markdown-body :deep(ol) { padding-left: 1.4rem; margin: 0.5rem 0; }
.markdown-body :deep(code) {
background: #f7fafc;
border: 1px solid #e2e8f0;
border-radius: 3px;
padding: 0.1em 0.3em;
font-size: 0.85em;
}
.markdown-body :deep(blockquote) {
border-left: 3px solid #3182ce;
padding-left: 0.75rem;
color: #4a5568;
margin: 0.5rem 0;
}
.markdown-body :deep(table) {
width: 100%;
border-collapse: collapse;
font-size: 0.875em;
margin: 0.75rem 0;
}
.markdown-body :deep(th),
.markdown-body :deep(td) {
border: 1px solid #e2e8f0;
padding: 0.35rem 0.6rem;
text-align: left;
}
.markdown-body :deep(th) { background: #f7fafc; font-weight: 600; }
</style>
+110 -6
View File
@@ -3,22 +3,30 @@
<div class="message-bubble" :class="isUser ? 'bubble-user' : 'bubble-assistant'"> <div class="message-bubble" :class="isUser ? 'bubble-user' : 'bubble-assistant'">
<div class="message-role">{{ isUser ? 'You' : 'AI Teacher' }}</div> <div class="message-role">{{ isUser ? 'You' : 'AI Teacher' }}</div>
<div v-if="isUser" class="message-content">{{ message.content }}</div> <div v-if="isUser" class="message-content">{{ message.content }}</div>
<div v-else class="message-content message-content--markdown" v-html="renderedContent"></div> <div v-else class="message-content message-content--markdown" v-html="renderedWithBadges" @click="onContentClick"></div>
<!-- Sources for assistant messages --> <!-- Sources for assistant messages -->
<div v-if="!isUser && message.sources && message.sources.length > 0" class="message-sources"> <div v-if="!isUser && message.sources && message.sources.length > 0" class="message-sources">
<div class="sources-label">Sources:</div> <div class="sources-label">Sources:</div>
<div class="source-list"> <div class="source-list" ref="sourceListEl">
<!-- TEXT sources --> <!-- TEXT sources -->
<div <div
v-for="(source, idx) in textSources" v-for="(source, idx) in textSources"
:key="'text-' + idx" :key="'text-' + idx"
class="source-item" class="source-item"
:class="{ 'source-item--active': activeRef === source.refLabel }"
:data-ref-label="source.refLabel"
>
<div
class="source-chip source-chip--text"
:class="{ 'source-chip--clickable': source.bookId && source.page }"
@click="source.bookId && source.page ? emit('open-source', source.bookId, source.page) : undefined"
> >
<div class="source-chip source-chip--text">
<span class="source-icon">📖</span> <span class="source-icon">📖</span>
<span v-if="source.refLabel" class="source-ref-label">{{ source.refLabel }}</span>
<span class="source-book-title">{{ source.bookTitle }}</span> <span class="source-book-title">{{ source.bookTitle }}</span>
<span v-if="source.page" class="source-page">p.&nbsp;{{ source.page }}</span> <span v-if="source.page" class="source-page">p.&nbsp;{{ source.page }}</span>
<span v-if="source.bookId && source.page" class="source-open-hint"></span>
</div> </div>
<div v-if="source.chunkText" class="source-chunk">{{ source.chunkText }}</div> <div v-if="source.chunkText" class="source-chunk">{{ source.chunkText }}</div>
</div> </div>
@@ -28,12 +36,20 @@
v-for="(source, idx) in figureSources" v-for="(source, idx) in figureSources"
:key="'fig-' + idx" :key="'fig-' + idx"
class="source-item source-item--figure" class="source-item source-item--figure"
:class="{ 'source-item--active': activeRef === source.refLabel }"
:data-ref-label="source.refLabel"
>
<div
class="source-chip source-chip--figure"
:class="{ 'source-chip--clickable': source.bookId && source.page }"
@click="source.bookId && source.page ? emit('open-source', source.bookId, source.page) : undefined"
> >
<div class="source-chip source-chip--figure">
<span class="source-icon">🖼</span> <span class="source-icon">🖼</span>
<span v-if="source.refLabel" class="source-ref-label source-ref-label--figure">{{ source.refLabel }}</span>
<span class="source-figure-label">{{ source.label || 'Figure' }}</span> <span class="source-figure-label">{{ source.label || 'Figure' }}</span>
<span v-if="source.page" class="source-page">p.&nbsp;{{ source.page }}</span> <span v-if="source.page" class="source-page">p.&nbsp;{{ source.page }}</span>
<span v-if="source.figureType" class="source-figure-type">{{ formatFigureType(source.figureType) }}</span> <span v-if="source.figureType" class="source-figure-type">{{ formatFigureType(source.figureType) }}</span>
<span v-if="source.bookId && source.page" class="source-open-hint"></span>
</div> </div>
<div v-if="source.caption" class="source-caption">{{ source.caption }}</div> <div v-if="source.caption" class="source-caption">{{ source.caption }}</div>
<div class="source-figure-image"> <div class="source-figure-image">
@@ -55,7 +71,7 @@
</template> </template>
<script setup lang="ts"> <script setup lang="ts">
import { computed } from 'vue' import { computed, ref } from 'vue'
import { marked } from 'marked' import { marked } from 'marked'
import type { ChatMessage, ChatSource } from '@/stores/chatStore' import type { ChatMessage, ChatSource } from '@/stores/chatStore'
@@ -63,8 +79,43 @@ const props = defineProps<{
message: ChatMessage message: ChatMessage
}>() }>()
const emit = defineEmits<{
'open-source': [bookId: string, page: number]
}>()
const isUser = computed(() => props.message.role === 'USER') const isUser = computed(() => props.message.role === 'USER')
const renderedContent = computed(() => marked.parse(props.message.content) as string) const activeRef = ref<string | null>(null)
const sourceListEl = ref<HTMLElement | null>(null)
/** Replaces [S1]/[F1]-style labels in the rendered HTML with clickable badges. */
const renderedWithBadges = computed(() => {
const html = marked.parse(props.message.content) as string
return html.replace(/\[(S|F)\d+\]/g, (match) => {
const inner = match.slice(1, -1) // e.g. "S1"
return `<span class="citation-badge" data-ref="${inner}" title="Jump to source ${inner}">${match}</span>`
})
})
function onContentClick(e: MouseEvent) {
const target = e.target as HTMLElement
if (!target.classList.contains('citation-badge')) return
const label = target.getAttribute('data-ref') // e.g. "S1" or "F1"
if (!label) return
activeRef.value = activeRef.value === label ? null : label
// Scroll to the matching source chip
const sourceEl = sourceListEl.value?.querySelector(`[data-ref-label="${label}"]`) as HTMLElement | null
sourceEl?.scrollIntoView({ behavior: 'smooth', block: 'start' })
// Open the book at the referenced page
const allSources = props.message.sources ?? []
const source = allSources.find((s: ChatSource) => s.refLabel === label)
if (source?.bookId && source.page) {
emit('open-source', source.bookId, source.page)
}
}
const textSources = computed(() => const textSources = computed(() =>
(props.message.sources ?? []).filter((s: ChatSource) => s.type === 'TEXT' || !s.type) (props.message.sources ?? []).filter((s: ChatSource) => s.type === 'TEXT' || !s.type)
@@ -255,6 +306,22 @@ function formatTime(iso: string): string {
border: 1px solid #bee3f8; border: 1px solid #bee3f8;
} }
.source-chip--clickable {
cursor: pointer;
transition: background 0.15s, border-color 0.15s;
}
.source-chip--clickable:hover {
background: #bee3f8;
border-color: #90cdf4;
}
.source-open-hint {
font-size: 0.75rem;
color: #3182ce;
margin-left: 0.1rem;
}
.source-chip--figure { .source-chip--figure {
background: #f0fff4; background: #f0fff4;
border: 1px solid #9ae6b4; border: 1px solid #9ae6b4;
@@ -322,6 +389,43 @@ function formatTime(iso: string): string {
font-style: italic; font-style: italic;
} }
.message-content--markdown :deep(.citation-badge) {
display: inline-block;
background: #ebf8ff;
border: 1px solid #90cdf4;
border-radius: 3px;
padding: 0 0.3em;
font-size: 0.78em;
font-weight: 600;
color: #2b6cb0;
cursor: pointer;
user-select: none;
transition: background 0.15s;
}
.message-content--markdown :deep(.citation-badge:hover) {
background: #bee3f8;
}
.source-item--active {
outline: 2px solid #4299e1;
border-radius: 6px;
}
.source-ref-label {
font-size: 0.72rem;
font-weight: 700;
background: #bee3f8;
color: #2b6cb0;
border-radius: 3px;
padding: 0 0.3rem;
}
.source-ref-label--figure {
background: #9ae6b4;
color: #276749;
}
.message-timestamp { .message-timestamp {
font-size: 0.7rem; font-size: 0.7rem;
opacity: 0.6; opacity: 0.6;
+2
View File
@@ -4,8 +4,10 @@ import { api } from '@/services/api'
export interface ChatSource { export interface ChatSource {
type: 'TEXT' | 'FIGURE' type: 'TEXT' | 'FIGURE'
bookId?: string
bookTitle: string bookTitle: string
page: number | null page: number | null
refLabel?: string
// TEXT-specific // TEXT-specific
chunkText?: string chunkText?: string
// FIGURE-specific // FIGURE-specific
+45
View File
@@ -10,11 +10,14 @@ export interface Topic {
} }
export interface SourceReference { export interface SourceReference {
bookId: string | null
bookTitle: string bookTitle: string
page: number | null page: number | null
} }
export interface TopicSummary { export interface TopicSummary {
id: string
summaryNumber: number
topicId: string topicId: string
topicName: string topicName: string
summary: string summary: string
@@ -22,12 +25,20 @@ export interface TopicSummary {
generatedAt: string generatedAt: string
} }
export interface SavedSummaryItem {
id: string
summaryNumber: number
generatedAt: string
}
export const useTopicStore = defineStore('topics', () => { export const useTopicStore = defineStore('topics', () => {
const topics = ref<Topic[]>([]) const topics = ref<Topic[]>([])
const activeSummary = ref<TopicSummary | null>(null) const activeSummary = ref<TopicSummary | null>(null)
const activeSummaryTopicId = ref<string | null>(null) const activeSummaryTopicId = ref<string | null>(null)
const summaryList = ref<SavedSummaryItem[]>([])
const loading = ref(false) const loading = ref(false)
const summaryLoading = ref(false) const summaryLoading = ref(false)
const summaryListLoading = ref(false)
const error = ref<string | null>(null) const error = ref<string | null>(null)
async function fetchTopics() { async function fetchTopics() {
@@ -43,6 +54,36 @@ export const useTopicStore = defineStore('topics', () => {
} }
} }
async function fetchSummaries(topicId: string) {
summaryListLoading.value = true
summaryList.value = []
error.value = null
try {
const response = await api.get<SavedSummaryItem[]>(`/topics/${topicId}/summaries`)
summaryList.value = response.data
} catch (err: any) {
error.value = err.message
} finally {
summaryListLoading.value = false
}
}
async function fetchSummaryDetail(topicId: string, summaryId: string): Promise<TopicSummary | null> {
summaryLoading.value = true
activeSummary.value = null
error.value = null
try {
const response = await api.get<TopicSummary>(`/topics/${topicId}/summaries/${summaryId}`)
activeSummary.value = response.data
return response.data
} catch (err: any) {
error.value = err.message
return null
} finally {
summaryLoading.value = false
}
}
async function generateSummary(topicId: string): Promise<TopicSummary | null> { async function generateSummary(topicId: string): Promise<TopicSummary | null> {
summaryLoading.value = true summaryLoading.value = true
activeSummaryTopicId.value = topicId activeSummaryTopicId.value = topicId
@@ -65,10 +106,14 @@ export const useTopicStore = defineStore('topics', () => {
topics, topics,
activeSummary, activeSummary,
activeSummaryTopicId, activeSummaryTopicId,
summaryList,
loading, loading,
summaryLoading, summaryLoading,
summaryListLoading,
error, error,
fetchTopics, fetchTopics,
fetchSummaries,
fetchSummaryDetail,
generateSummary generateSummary
} }
}) })
+59 -79
View File
@@ -3,27 +3,10 @@
<h1 class="page-title">Knowledge Chat</h1> <h1 class="page-title">Knowledge Chat</h1>
<p class="page-subtitle">Ask questions grounded in your uploaded medical textbooks.</p> <p class="page-subtitle">Ask questions grounded in your uploaded medical textbooks.</p>
<!-- Step 1: Topic Selection --> <!-- Session selection -->
<div v-if="!chatStore.session && !selectedTopic" class="topic-selection"> <div v-if="!chatStore.session" class="session-setup card">
<h2 class="section-title">Select a Topic</h2>
<div class="topic-grid">
<button
v-for="topic in topicStore.topics"
:key="topic.id"
:class="['topic-tile', { 'topic-tile-freeform': topic.id === 'free-form' }]"
@click="handleTopicSelect(topic)"
>
<span class="topic-tile-name">{{ topic.name }}</span>
<span v-if="topic.id === 'free-form'" class="topic-tile-hint">Any neurosurgery question</span>
</button>
</div>
</div>
<!-- Step 2: Topic selected previous sessions + new chat -->
<div v-else-if="!chatStore.session && selectedTopic" class="session-setup card">
<div class="setup-header"> <div class="setup-header">
<button class="btn-back" @click="handleBack"> Topics</button> <h2 class="section-title">Free-form Chat</h2>
<h2 class="section-title">{{ selectedTopic.name }}</h2>
</div> </div>
<div v-if="chatStore.error" class="error-banner">{{ chatStore.error }}</div> <div v-if="chatStore.error" class="error-banner">{{ chatStore.error }}</div>
@@ -71,6 +54,10 @@
</div> </div>
</div> </div>
<!-- Chat + Reader split -->
<div class="chat-reader-split">
<!-- Messages + Input -->
<div class="chat-column">
<!-- Messages Area --> <!-- Messages Area -->
<div class="messages-container" ref="messagesContainer"> <div class="messages-container" ref="messagesContainer">
<div v-if="chatStore.loading && chatStore.messages.length === 0" class="empty-state"> <div v-if="chatStore.loading && chatStore.messages.length === 0" class="empty-state">
@@ -89,6 +76,7 @@
v-for="message in chatStore.messages" v-for="message in chatStore.messages"
:key="message.id" :key="message.id"
:message="message" :message="message"
@open-source="handleOpenSource"
/> />
<div v-if="chatStore.sending" class="typing-indicator"> <div v-if="chatStore.sending" class="typing-indicator">
<div class="typing-bubble"> <div class="typing-bubble">
@@ -123,6 +111,19 @@
<p class="input-hint">Press Enter to send, Shift+Enter for new line.</p> <p class="input-hint">Press Enter to send, Shift+Enter for new line.</p>
</div> </div>
</div> </div>
<!-- Inline book reader panel -->
<BookPagePanel
v-if="readerPanel"
:book-id="readerPanel.bookId"
:page="readerPanel.page"
:book-title="readerPanel.bookTitle"
class="reader-panel"
@close="readerPanel = null"
@navigate="(p) => readerPanel && (readerPanel.page = p)"
/>
</div>
</div>
</div> </div>
</template> </template>
@@ -130,8 +131,10 @@
import { ref, nextTick, onMounted, watch, inject } from 'vue' import { ref, nextTick, onMounted, watch, inject } from 'vue'
import { useChatStore } from '@/stores/chatStore' import { useChatStore } from '@/stores/chatStore'
import { useTopicStore } from '@/stores/topicStore' import { useTopicStore } from '@/stores/topicStore'
import { useBookStore } from '@/stores/bookStore'
import type { ChatSession } from '@/stores/chatStore' import type { ChatSession } from '@/stores/chatStore'
import ChatMessage from '@/components/ChatMessage.vue' import ChatMessage from '@/components/ChatMessage.vue'
import BookPagePanel from '@/components/BookPagePanel.vue'
interface Topic { interface Topic {
id: string id: string
@@ -142,6 +145,7 @@ interface Topic {
const chatStore = useChatStore() const chatStore = useChatStore()
const topicStore = useTopicStore() const topicStore = useTopicStore()
const bookStore = useBookStore()
const showToast = inject<(msg: string, type?: 'error' | 'success') => void>('showToast') const showToast = inject<(msg: string, type?: 'error' | 'success') => void>('showToast')
const selectedTopic = ref<Topic | null>(null) const selectedTopic = ref<Topic | null>(null)
@@ -150,10 +154,22 @@ const loadingTopicSessions = ref(false)
const inputText = ref('') const inputText = ref('')
const messagesContainer = ref<HTMLElement | null>(null) const messagesContainer = ref<HTMLElement | null>(null)
interface ReaderPanel { bookId: string; page: number; bookTitle?: string }
const readerPanel = ref<ReaderPanel | null>(null)
function handleOpenSource(bookId: string, page: number) {
const book = bookStore.books.find(b => b.id === bookId)
readerPanel.value = { bookId, page, bookTitle: book?.title }
}
onMounted(async () => { onMounted(async () => {
if (topicStore.topics.length === 0) { if (topicStore.topics.length === 0) {
await topicStore.fetchTopics() await topicStore.fetchTopics()
} }
const freeForm = topicStore.topics.find((t) => t.id === 'free-form')
if (freeForm) {
await handleTopicSelect(freeForm)
}
}) })
watch( watch(
@@ -189,11 +205,6 @@ async function handleTopicSelect(topic: Topic) {
loadingTopicSessions.value = false loadingTopicSessions.value = false
} }
function handleBack() {
selectedTopic.value = null
topicSessions.value = []
}
async function handleNewChat() { async function handleNewChat() {
const ok = await chatStore.createSession(selectedTopic.value!.id) const ok = await chatStore.createSession(selectedTopic.value!.id)
if (!ok) { if (!ok) {
@@ -207,9 +218,7 @@ async function handleResumeSession(session: ChatSession) {
} }
function handleLeaveSession() { function handleLeaveSession() {
// Leave without deleting — session stays in DB and will appear in "Previous Chats"
chatStore.leaveSession() chatStore.leaveSession()
// Refresh the sessions list for the current topic
if (selectedTopic.value) { if (selectedTopic.value) {
loadingTopicSessions.value = true loadingTopicSessions.value = true
chatStore.fetchSessionsByTopic(selectedTopic.value.id).then((sessions) => { chatStore.fetchSessionsByTopic(selectedTopic.value.id).then((sessions) => {
@@ -231,12 +240,6 @@ async function handleSend() {
</script> </script>
<style scoped> <style scoped>
.topic-selection {
display: flex;
flex-direction: column;
gap: 1.25rem;
}
.section-title { .section-title {
font-size: 1.1rem; font-size: 1.1rem;
font-weight: 600; font-weight: 600;
@@ -244,52 +247,6 @@ async function handleSend() {
margin: 0; margin: 0;
} }
.topic-grid {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(220px, 1fr));
gap: 0.75rem;
}
.topic-tile {
display: flex;
flex-direction: column;
align-items: flex-start;
gap: 0.25rem;
padding: 1rem 1.1rem;
background: white;
border: 1px solid #e2e8f0;
border-radius: 8px;
cursor: pointer;
text-align: left;
transition: border-color 0.15s, box-shadow 0.15s;
}
.topic-tile:hover {
border-color: #3182ce;
box-shadow: 0 2px 8px rgba(49, 130, 206, 0.15);
}
.topic-tile-freeform {
border-style: dashed;
border-color: #a0aec0;
}
.topic-tile-freeform:hover {
border-color: #718096;
box-shadow: 0 2px 8px rgba(0, 0, 0, 0.08);
}
.topic-tile-name {
font-size: 0.9rem;
font-weight: 600;
color: #2d3748;
}
.topic-tile-hint {
font-size: 0.78rem;
color: #a0aec0;
}
.session-setup { .session-setup {
max-width: 540px; max-width: 540px;
} }
@@ -381,6 +338,29 @@ async function handleSend() {
min-height: 500px; min-height: 500px;
} }
.chat-reader-split {
display: flex;
flex: 1;
min-height: 0;
gap: 0;
}
.chat-column {
display: flex;
flex-direction: column;
flex: 1;
min-width: 0;
gap: 1rem;
}
.reader-panel {
width: 420px;
flex-shrink: 0;
border-radius: 10px;
margin-left: 1rem;
box-shadow: -2px 0 8px rgba(0, 0, 0, 0.07);
}
.session-bar { .session-bar {
display: flex; display: flex;
align-items: center; align-items: center;
+343 -20
View File
@@ -1,7 +1,7 @@
<template> <template>
<div class="topics-view"> <div class="topics-view">
<h1 class="page-title">Topics</h1> <h1 class="page-title">Topics</h1>
<p class="page-subtitle">Select a topic to generate an AI-powered summary from uploaded books.</p> <p class="page-subtitle">Select a topic to view or generate an AI-powered summary from uploaded books.</p>
<!-- Loading state --> <!-- Loading state -->
<div v-if="topicStore.loading" class="empty-state"> <div v-if="topicStore.loading" class="empty-state">
@@ -18,15 +18,39 @@
</div> </div>
<div v-else class="topics-layout"> <div v-else class="topics-layout">
<!-- Topic Grid --> <div class="topics-main">
<div class="topic-grid">
<TopicCard <!-- Summary history list -->
v-for="topic in topicStore.topics" <div v-if="selectedTopicId" class="history-panel card">
:key="topic.id" <div class="history-header">
:topic="topic" <span class="history-title">Saved summaries</span>
:is-generating="topicStore.activeSummaryTopicId === topic.id" <button class="btn btn-primary btn-sm" :disabled="topicStore.summaryLoading" @click="handleGenerate(selectedTopicId!)">
@generate="handleGenerate" <span v-if="topicStore.summaryLoading" class="spinner" style="width:14px;height:14px;display:inline-block;vertical-align:middle;margin-right:4px;"></span>
/> Generate New
</button>
</div>
<div v-if="topicStore.summaryListLoading" class="history-loading">
<div class="spinner spinner-dark" style="width:20px;height:20px;margin-right:8px;display:inline-block;vertical-align:middle;"></div>
Loading...
</div>
<div v-else-if="topicStore.summaryList.length === 0" class="history-empty">
No summaries yet. Click "Generate New" to create one.
</div>
<div v-else class="history-list">
<button
v-for="item in topicStore.summaryList"
:key="item.id"
class="history-chip"
:class="{ 'history-chip--active': topicStore.activeSummary?.id === item.id }"
@click="handleLoadSummary(item)"
>
Summary #{{ item.summaryNumber }}
<span class="history-chip-date">· {{ formatDateShort(item.generatedAt) }}</span>
</button>
</div>
</div> </div>
<!-- Summary Panel --> <!-- Summary Panel -->
@@ -48,15 +72,26 @@
</p> </p>
</div> </div>
<div v-else-if="topicStore.activeSummary" class="summary-panel card"> <div v-else-if="!topicStore.activeSummary" class="summary-panel card summary-placeholder">
<div class="summary-header"> <p class="summary-placeholder-text">
<h2 class="summary-topic-name">{{ topicStore.activeSummary.topicName }}</h2> {{ selectedTopicId ? 'Select a saved summary or generate a new one.' : 'Select a topic to get started.' }}
<span class="summary-timestamp">{{ formatDate(topicStore.activeSummary.generatedAt) }}</span> </p>
</div> </div>
<div class="summary-text">{{ topicStore.activeSummary.summary }}</div> <div v-else class="summary-panel card">
<div class="summary-header">
<h2 class="summary-topic-name">{{ topicStore.activeSummary.topicName }}</h2>
<div class="summary-meta">
<span v-if="topicStore.activeSummary.summaryNumber" class="summary-number">
Summary #{{ topicStore.activeSummary.summaryNumber }}
</span>
<span class="summary-timestamp">{{ formatDate(topicStore.activeSummary.generatedAt) }}</span>
</div>
</div>
<div v-if="topicStore.activeSummary.sources.length > 0" class="sources-section"> <div class="summary-text summary-text--markdown" v-html="renderedSummary" @click="handleSummaryClick"></div>
<div ref="sourcesSection" v-if="topicStore.activeSummary.sources.length > 0" class="sources-section">
<button class="sources-toggle" @click="showSources = !showSources"> <button class="sources-toggle" @click="showSources = !showSources">
Sources ({{ topicStore.activeSummary.sources.length }}) Sources ({{ topicStore.activeSummary.sources.length }})
<span>{{ showSources ? '▲' : '▼' }}</span> <span>{{ showSources ? '▲' : '▼' }}</span>
@@ -66,35 +101,115 @@
v-for="(source, idx) in topicStore.activeSummary.sources" v-for="(source, idx) in topicStore.activeSummary.sources"
:key="idx" :key="idx"
class="source-chip" class="source-chip"
:class="{ 'source-chip--clickable': source.bookId && source.page }"
@click="source.bookId && source.page ? handleOpenSource(source.bookId, source.page) : undefined"
> >
<span class="source-icon">📖</span>
<span class="source-book">{{ source.bookTitle }}</span> <span class="source-book">{{ source.bookTitle }}</span>
<span v-if="source.page" class="source-page">p. {{ source.page }}</span> <span v-if="source.page" class="source-page">p.&nbsp;{{ source.page }}</span>
<span v-if="source.bookId && source.page" class="source-open-hint"></span>
</div> </div>
</div> </div>
<BookPagePanel
v-if="readerPanel"
:book-id="readerPanel.bookId"
:page="readerPanel.page"
:book-title="readerPanel.bookTitle"
class="reader-panel"
@close="readerPanel = null"
@navigate="(p) => readerPanel && (readerPanel.page = p)"
/>
</div> </div>
<div v-else class="no-sources"> <div v-else class="no-sources">
No source citations available for this summary. No source citations available for this summary.
</div> </div>
</div> </div>
<!-- Topic Grid -->
<div class="topic-grid">
<TopicCard
v-for="topic in summaryTopics"
:key="topic.id"
:topic="topic"
:is-generating="topicStore.activeSummaryTopicId === topic.id"
:is-selected="selectedTopicId === topic.id"
@generate="handleTopicClick"
/>
</div>
</div><!-- end topics-main -->
</div> </div>
</div> </div>
</template> </template>
<script setup lang="ts"> <script setup lang="ts">
import { ref, onMounted, inject } from 'vue' import { ref, computed, onMounted, inject } from 'vue'
import { marked } from 'marked'
import { RouterLink } from 'vue-router' import { RouterLink } from 'vue-router'
import { useTopicStore } from '@/stores/topicStore' import { useTopicStore, type SavedSummaryItem } from '@/stores/topicStore'
import { useBookStore } from '@/stores/bookStore'
import TopicCard from '@/components/TopicCard.vue' import TopicCard from '@/components/TopicCard.vue'
import BookPagePanel from '@/components/BookPagePanel.vue'
const topicStore = useTopicStore() const topicStore = useTopicStore()
const bookStore = useBookStore()
const showToast = inject<(msg: string, type?: 'error' | 'success') => void>('showToast') const showToast = inject<(msg: string, type?: 'error' | 'success') => void>('showToast')
const showSources = ref(true) const showSources = ref(true)
const summaryError = ref<string | null>(null) const summaryError = ref<string | null>(null)
const isNoBooks = ref(false) const isNoBooks = ref(false)
const sourcesSection = ref<HTMLElement | null>(null)
const selectedTopicId = ref<string | null>(null)
interface ReaderPanel { bookId: string; page: number; bookTitle?: string }
const readerPanel = ref<ReaderPanel | null>(null)
const summaryTopics = computed(() => topicStore.topics.filter(t => t.id !== 'free-form'))
const renderedSummary = computed(() => {
if (!topicStore.activeSummary) return ''
const html = marked.parse(topicStore.activeSummary.summary) as string
return html.replace(/\[S(\d+)\]/g, '<span class="source-ref">[S$1]</span>')
})
function handleSummaryClick(e: MouseEvent) {
if ((e.target as HTMLElement).classList.contains('source-ref')) {
showSources.value = true
sourcesSection.value?.scrollIntoView({ behavior: 'smooth', block: 'start' })
}
}
function handleOpenSource(bookId: string, page: number) {
const book = bookStore.books.find(b => b.id === bookId)
readerPanel.value = { bookId, page, bookTitle: book?.title }
showSources.value = true
}
async function handleTopicClick(topicId: string) {
if (selectedTopicId.value !== topicId) {
selectedTopicId.value = topicId
topicStore.activeSummary = null
summaryError.value = null
await topicStore.fetchSummaries(topicId)
// Auto-load the latest summary if any exist
const list = topicStore.summaryList
if (list.length > 0) {
await topicStore.fetchSummaryDetail(topicId, list[list.length - 1].id)
}
}
}
async function handleLoadSummary(item: SavedSummaryItem) {
if (!selectedTopicId.value) return
summaryError.value = null
await topicStore.fetchSummaryDetail(selectedTopicId.value, item.id)
}
onMounted(async () => { onMounted(async () => {
await topicStore.fetchTopics() await topicStore.fetchTopics()
if (bookStore.books.length === 0) {
await bookStore.fetchBooks()
}
}) })
async function handleGenerate(topicId: string) { async function handleGenerate(topicId: string) {
@@ -109,27 +224,122 @@ async function handleGenerate(topicId: string) {
summaryError.value.toLowerCase().includes('no books') || summaryError.value.toLowerCase().includes('no books') ||
summaryError.value.toLowerCase().includes('knowledge source') summaryError.value.toLowerCase().includes('knowledge source')
showToast?.(summaryError.value, 'error') showToast?.(summaryError.value, 'error')
} else {
// Refresh the history list to include the newly saved summary
await topicStore.fetchSummaries(topicId)
} }
} }
function formatDate(iso: string): string { function formatDate(iso: string): string {
return new Date(iso).toLocaleString() return new Date(iso).toLocaleString()
} }
function formatDateShort(iso: string): string {
return new Date(iso).toLocaleDateString(undefined, { month: 'short', day: 'numeric' })
}
</script> </script>
<style scoped> <style scoped>
.topics-layout { .topics-layout {
display: flex;
gap: 2rem;
}
.topics-main {
flex: 1;
min-width: 0;
display: flex; display: flex;
flex-direction: column; flex-direction: column;
gap: 2rem; gap: 2rem;
} }
.reader-panel {
margin-top: 1rem;
height: 600px;
min-height: 400px;
border-radius: 10px;
box-shadow: 0 2px 8px rgba(0, 0, 0, 0.07);
}
.topic-grid { .topic-grid {
display: grid; display: grid;
grid-template-columns: repeat(auto-fill, minmax(280px, 1fr)); grid-template-columns: repeat(auto-fill, minmax(280px, 1fr));
gap: 1rem; gap: 1rem;
} }
/* History panel */
.history-panel {
border-top: 3px solid #805ad5;
padding: 1rem;
}
.history-header {
display: flex;
align-items: center;
justify-content: space-between;
margin-bottom: 0.75rem;
}
.history-title {
font-size: 0.875rem;
font-weight: 600;
color: #553c9a;
}
.btn-sm {
font-size: 0.8rem;
padding: 0.3rem 0.75rem;
}
.history-loading {
font-size: 0.85rem;
color: #718096;
display: flex;
align-items: center;
}
.history-empty {
font-size: 0.85rem;
color: #a0aec0;
font-style: italic;
}
.history-list {
display: flex;
flex-wrap: wrap;
gap: 0.5rem;
}
.history-chip {
background: #faf5ff;
border: 1px solid #d6bcfa;
border-radius: 6px;
padding: 0.3rem 0.75rem;
font-size: 0.8rem;
font-weight: 500;
color: #553c9a;
cursor: pointer;
transition: background 0.15s, border-color 0.15s;
}
.history-chip:hover {
background: #e9d8fd;
border-color: #b794f4;
}
.history-chip--active {
background: #805ad5;
border-color: #805ad5;
color: #fff;
}
.history-chip-date {
font-weight: 400;
color: inherit;
opacity: 0.75;
}
/* Summary panel */
.summary-panel { .summary-panel {
border-top: 3px solid #3182ce; border-top: 3px solid #3182ce;
} }
@@ -170,6 +380,22 @@ function formatDate(iso: string): string {
color: #1a365d; color: #1a365d;
} }
.summary-meta {
display: flex;
align-items: baseline;
gap: 0.5rem;
}
.summary-number {
font-size: 0.8rem;
font-weight: 600;
color: #805ad5;
background: #faf5ff;
border: 1px solid #d6bcfa;
border-radius: 4px;
padding: 0.1rem 0.4rem;
}
.summary-timestamp { .summary-timestamp {
font-size: 0.8rem; font-size: 0.8rem;
color: #a0aec0; color: #a0aec0;
@@ -179,10 +405,60 @@ function formatDate(iso: string): string {
font-size: 0.95rem; font-size: 0.95rem;
line-height: 1.7; line-height: 1.7;
color: #2d3748; color: #2d3748;
white-space: pre-wrap;
margin-bottom: 1rem; margin-bottom: 1rem;
} }
.summary-text--markdown {
white-space: normal;
}
.summary-text--markdown :deep(h1),
.summary-text--markdown :deep(h2),
.summary-text--markdown :deep(h3),
.summary-text--markdown :deep(h4) {
font-weight: 700;
margin: 0.75rem 0 0.35rem;
line-height: 1.3;
color: #1a202c;
}
.summary-text--markdown :deep(h1) { font-size: 1.15rem; }
.summary-text--markdown :deep(h2) { font-size: 1.05rem; }
.summary-text--markdown :deep(h3) { font-size: 0.975rem; }
.summary-text--markdown :deep(h4) { font-size: 0.925rem; }
.summary-text--markdown :deep(p) { margin: 0.4rem 0; }
.summary-text--markdown :deep(ul),
.summary-text--markdown :deep(ol) {
padding-left: 1.4rem;
margin: 0.4rem 0;
}
.summary-text--markdown :deep(li) { margin: 0.2rem 0; }
.summary-text--markdown :deep(strong) {
font-weight: 700;
color: #1a202c;
}
.summary-text--markdown :deep(em) { font-style: italic; }
.summary-text--markdown :deep(code) {
background: #edf2f7;
border-radius: 3px;
padding: 0.1em 0.35em;
font-size: 0.87em;
font-family: monospace;
}
.summary-text--markdown :deep(blockquote) {
border-left: 3px solid #bee3f8;
margin: 0.5rem 0;
padding: 0.25rem 0.75rem;
color: #4a5568;
}
.sources-section { .sources-section {
border-top: 1px solid #e2e8f0; border-top: 1px solid #e2e8f0;
padding-top: 0.75rem; padding-top: 0.75rem;
@@ -223,6 +499,20 @@ function formatDate(iso: string): string {
font-size: 0.8rem; font-size: 0.8rem;
} }
.source-chip--clickable {
cursor: pointer;
transition: background 0.15s, border-color 0.15s;
}
.source-chip--clickable:hover {
background: #bee3f8;
border-color: #90cdf4;
}
.source-icon {
font-size: 0.8rem;
}
.source-book { .source-book {
color: #2b6cb0; color: #2b6cb0;
font-weight: 500; font-weight: 500;
@@ -232,6 +522,12 @@ function formatDate(iso: string): string {
color: #718096; color: #718096;
} }
.source-open-hint {
font-size: 0.75rem;
color: #3182ce;
margin-left: 0.1rem;
}
.no-sources { .no-sources {
font-size: 0.85rem; font-size: 0.85rem;
color: #a0aec0; color: #a0aec0;
@@ -252,4 +548,31 @@ function formatDate(iso: string): string {
color: #3182ce; color: #3182ce;
text-decoration: underline; text-decoration: underline;
} }
.summary-placeholder {
display: flex;
align-items: center;
justify-content: center;
min-height: 6rem;
border-top-color: #cbd5e0;
}
.summary-placeholder-text {
font-size: 0.95rem;
color: #a0aec0;
font-style: italic;
}
.summary-text--markdown :deep(.source-ref) {
color: #3182ce;
font-weight: 600;
cursor: pointer;
border-radius: 3px;
padding: 0 0.15em;
}
.summary-text--markdown :deep(.source-ref:hover) {
background: #ebf8ff;
text-decoration: underline;
}
</style> </style>
@@ -0,0 +1,34 @@
# Specification Quality Checklist: RAG Retrieval Quality Improvements
**Purpose**: Validate specification completeness and quality before proceeding to planning
**Created**: 2026-04-06
**Feature**: [spec.md](../spec.md)
## Content Quality
- [x] No implementation details (languages, frameworks, APIs)
- [x] Focused on user value and business needs
- [x] Written for non-technical stakeholders
- [x] All mandatory sections completed
## Requirement Completeness
- [x] No [NEEDS CLARIFICATION] markers remain
- [x] Requirements are testable and unambiguous
- [x] Success criteria are measurable
- [x] Success criteria are technology-agnostic (no implementation details)
- [x] All acceptance scenarios are defined
- [x] Edge cases are identified
- [x] Scope is clearly bounded
- [x] Dependencies and assumptions identified
## Feature Readiness
- [x] All functional requirements have clear acceptance criteria
- [x] User scenarios cover primary flows
- [x] Feature meets measurable outcomes defined in Success Criteria
- [x] No implementation details leak into specification
## Notes
- All items pass. Ready to proceed to `/speckit.clarify` or `/speckit.plan`.
@@ -0,0 +1,33 @@
# API Contract: Chat (unchanged endpoints)
**Branch**: `004-rag-retrieval-quality` | **Date**: 2026-04-06
## No endpoint changes
This feature makes no changes to the public API surface. All existing `/api/v1/chat/...` endpoints remain identical in path, method, request body, and response shape.
## Message response — `sources` field addition
The only observable change to callers is that each source entry in `Message.sources` gains an optional `refLabel` field. This is backwards-compatible (additive only).
### Existing contract (unchanged)
```
POST /api/v1/chat/sessions/{sessionId}/messages
Body: { "content": "..." }
Response: Message { id, sessionId, role, content, sources: [...], createdAt }
```
### Source entry schema (additive change)
Before:
```json
{ "type": "TEXT", "bookTitle": "...", "page": 142, "chunkText": "..." }
```
After (new optional field):
```json
{ "type": "TEXT", "refLabel": "S1", "bookTitle": "...", "page": 142, "chunkText": "..." }
```
Frontend consumers that ignore unknown fields are unaffected. `ChatMessage.vue` may optionally use `refLabel` for future inline citation linking.
@@ -0,0 +1,68 @@
# API Contract: Topics — Summary Persistence
**Base path**: `/api/v1/topics`
---
## Existing endpoint (unchanged behaviour, extended response)
### `POST /api/v1/topics/{id}/summary`
Generates a new summary for the topic, **persists it**, and returns it.
**Path param**: `id` — topic id (e.g. `intracranial-aneurysms`)
**Response** `200 OK`:
```json
{
"id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"summaryNumber": 3,
"topicId": "intracranial-aneurysms",
"topicName": "Intracranial Aneurysms",
"summary": "## Key Points\n...",
"sources": [
{ "bookId": "uuid", "bookTitle": "Youmans & Winn", "page": 142 }
],
"generatedAt": "2026-04-07T10:23:00Z"
}
```
**Error responses**:
- `404 Not Found` — topic id does not exist
- `503 Service Unavailable` — no books ready (existing `NoKnowledgeSourceException`)
---
## New endpoints
### `GET /api/v1/topics/{id}/summaries`
Returns the list of saved summaries for a topic (no full text — list metadata only).
**Path param**: `id` — topic id
**Response** `200 OK`:
```json
[
{ "id": "uuid-1", "summaryNumber": 1, "generatedAt": "2026-04-06T08:00:00Z" },
{ "id": "uuid-2", "summaryNumber": 2, "generatedAt": "2026-04-06T09:15:00Z" }
]
```
Ordered ascending by `summaryNumber`. Empty array if no summaries saved yet.
**Error responses**:
- `404 Not Found` — topic id does not exist
---
### `GET /api/v1/topics/{id}/summaries/{summaryId}`
Fetches the full content of a specific saved summary.
**Path params**: `id` — topic id, `summaryId` — summary UUID
**Response** `200 OK`: same shape as `POST /summary` response above.
**Error responses**:
- `404 Not Found` — topic or summary not found
@@ -0,0 +1,91 @@
# Data Model: RAG Retrieval Quality + Topic Summary Persistence
**Branch**: `004-rag-retrieval-quality` | **Date**: 2026-04-07
---
## New persistent entity: TopicSummaryEntity
**Table**: `topic_summary`
**Migration**: `V6__topic_summary.sql`
| Column | Type | Notes |
|--------|------|-------|
| `id` | `UUID` PK | `gen_random_uuid()` default |
| `topic_id` | `VARCHAR(100)` NOT NULL | FK to `topic.id` |
| `summary_number` | `INT` NOT NULL | Sequential per topic (1, 2, 3, …). Set at insert time: `COUNT(*) WHERE topic_id = ? + 1`. |
| `summary` | `TEXT` NOT NULL | Full markdown summary text |
| `sources_json` | `TEXT` NOT NULL | JSON array of `SourceReference` objects (same structure as `TopicSummaryResponse.sources`) |
| `generated_at` | `TIMESTAMPTZ` NOT NULL | UTC timestamp of generation |
**Constraints**: no unique constraint on `summary_number` (sequential, not concurrent-safe for POC). No FK constraint enforced at DB level (topic ids are static seed data).
---
## In-memory objects (new, from RAG quality work)
### ExpandedQuery (value object, not persisted)
Produced by `QueryExpansionService` for each user message.
| Field | Type | Description |
|-------|------|-------------|
| `original` | `String` | The user's literal question |
| `rewritten` | `String` | Clinically rewritten version used for vector search |
---
### LabelledContext (value object, not persisted)
Produced by `ChatService.buildContextPrompt()` to track the mapping from ref-labels to source entities.
| Field | Type | Description |
|-------|------|-------------|
| `sectionLabels` | `Map<String, SectionEntity>` | e.g. `{"S1" → SectionEntity, "S2" → SectionEntity}` |
| `figureLabels` | `Map<String, FigureEntity>` | e.g. `{"F1" → FigureEntity}` |
| `promptText` | `String` | The fully formatted context prompt including `[S1]`, `[F1]` tags |
---
## New API DTOs
### SavedSummaryItem (list view — no full text)
```java
record SavedSummaryItem(UUID id, int summaryNumber, Instant generatedAt) {}
```
Used in `GET /api/v1/topics/{id}/summaries` to show the summary history list without transmitting full text.
### TopicSummaryResponse (existing, extended)
Adds `id` (UUID) and `summaryNumber` (int) fields so the frontend knows which saved record was just created.
---
## Existing entities (unchanged)
| Entity | Table | Change |
|--------|-------|--------|
| `SectionEntity` | `section` | None |
| `FigureEntity` | `figure` | None |
| `Message` | `message` | `sources` field gets `refLabel` key added per entry |
| `ChatSession` | `chat_session` | None |
| `Book` | `book` | None |
| `Topic` | `topic` | None |
---
## Message.sources structure (existing, clarified)
After the RAG quality feature each entry includes `refLabel`:
```json
{
"type": "TEXT",
"refLabel": "S1",
"bookTitle": "Youmans & Winn Neurological Surgery",
"page": 142,
"chunkText": "..."
}
```
+108
View File
@@ -0,0 +1,108 @@
# Implementation Plan: RAG Retrieval Quality + Topic Summary Persistence
**Branch**: `004-rag-retrieval-quality` | **Date**: 2026-04-07 | **Spec**: [spec.md](spec.md)
**Input**: Feature specification from `/specs/004-rag-retrieval-quality/spec.md` + user request: "Save summaries by topic; list previously saved summaries; button to generate new."
## Summary
Improve RAG retrieval quality by (1) expanding user queries via LLM rewrite to bridge vocabulary gaps and (2) validating generated citations against the retrieved context to eliminate hallucinated references. Additionally, persist generated topic summaries to the database so students can revisit past summaries and generate new ones on demand.
## Technical Context
**Language/Version**: Java 21 (backend), TypeScript / Node 20 (frontend)
**Primary Dependencies**: Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API (chat + embeddings), Vue 3.4, Pinia 2.1, Axios 1.7
**Storage**: PostgreSQL (JPA + Flyway), pgvector (`VectorStore`)
**Testing**: JUnit 5, Spring Boot Test, Vitest
**Target Platform**: Linux server
**Project Type**: Web application (backend API + Vue frontend)
**Performance Goals**: RAG query latency remains acceptable (one additional LLM call for query expansion, ~12s overhead)
**Constraints**: KISS — no new architectural layers; no new external services; single deployable backend + frontend
## Constitution Check
| Principle | Status | Notes |
|-----------|--------|-------|
| I — KISS | ✅ Pass | Query expansion = one LLM call. Citation validation = string scan. Summary persistence = one new table + minimal service change. No new layers. |
| II — Easy to Change | ✅ Pass | `QueryExpansionService`, `CitationValidatorService`, `TopicSummaryRepository` are small, focused interfaces. Swappable without touching callers. |
| III — Web-First | ✅ Pass | All new functionality exposed via REST at `/api/v1/...`. Frontend-only UI changes. |
| IV — Documentation as Architecture | ⚠️ Action needed | README must be updated to reflect new `topic_summary` table and the summary history flow. Update in the same PR. |
## Project Structure
### Documentation (this feature)
```text
specs/004-rag-retrieval-quality/
├── plan.md # This file
├── research.md # Phase 0 — decisions on query expansion, citation grounding, summary persistence
├── data-model.md # Phase 1 — topic_summary table, ExpandedQuery, LabelledContext DTOs
├── contracts/
│ ├── chat-api.md # Existing chat API (updated for labelled sources)
│ └── topics-api.md # New/updated topics API (summary list + detail endpoints)
└── tasks.md # Phase 2 output (/speckit.tasks command)
```
### Source Code
```text
backend/
├── src/main/java/com/aiteacher/
│ ├── retrieval/
│ │ ├── QueryExpansionService.java (NEW) — LLM rewrite of user query
│ │ ├── ExpandedQuery.java (NEW) — value object {original, rewritten}
│ │ ├── LabelledContext.java (NEW) — value object {sectionLabels, figureLabels, promptText}
│ │ └── CitationValidatorService.java (NEW) — strip unknown [Sx]/[Fx] refs from answer text
│ └── topic/
│ ├── TopicSummaryEntity.java (NEW) — JPA entity for topic_summary table
│ ├── TopicSummaryRepository.java (NEW) — findByTopicIdOrderBySummaryNumberAsc
│ ├── SavedSummaryItem.java (NEW) — DTO for list view (id, summaryNumber, generatedAt)
│ ├── TopicSummaryResponse.java (MODIFY) — add id: UUID, summaryNumber: int fields
│ ├── TopicSummaryService.java (MODIFY) — persist after generation; add list/get methods
│ └── TopicController.java (MODIFY) — add GET /summaries and GET /summaries/{id}
├── src/main/resources/db/migration/
│ └── V6__topic_summary.sql (NEW) — create topic_summary table
└── chat/
└── ChatService.java (MODIFY) — use QueryExpansionService + CitationValidatorService
frontend/
└── src/
├── stores/
│ └── topicStore.ts (MODIFY) — add fetchSummaries(), fetchSummaryDetail(), summaryList state
└── views/
└── TopicsView.vue (MODIFY) — show summary list when topic selected, "Generate New" button
```
**Structure Decision**: Option 2 (web application). Existing `backend/` + `frontend/` directories. No new top-level directories.
## Complexity Tracking
> No constitution violations introduced.
---
## Phase 0: Research (complete)
See [research.md](research.md). All decisions resolved:
- Query expansion: single LLM rewrite call
- Citation grounding: ref-label tagging + post-generation string scan
- Summary persistence: new `topic_summary` table, sequential numbering at insert time
## Phase 1: Design (complete)
### Data model
See [data-model.md](data-model.md). New `topic_summary` table; two new in-memory value objects; `TopicSummaryResponse` extended with `id` + `summaryNumber`.
### API contracts
See [contracts/topics-api.md](contracts/topics-api.md):
- `POST /api/v1/topics/{id}/summary` — now persists and returns `id` + `summaryNumber`
- `GET /api/v1/topics/{id}/summaries` — list metadata (no full text)
- `GET /api/v1/topics/{id}/summaries/{summaryId}` — full detail
### Frontend behaviour
When a **topic card is clicked** (before generating):
1. Call `GET /api/v1/topics/{id}/summaries`.
2. If summaries exist → show list panel with chips: "Summary #1 · Apr 6", "Summary #2 · Apr 7", … + a "Generate New" button.
3. If no summaries → show "Generate Summary" button (current behaviour).
4. Clicking a chip → call `GET /api/v1/topics/{id}/summaries/{summaryId}` → display the saved summary.
5. Clicking "Generate New" → call `POST /api/v1/topics/{id}/summary` → display new summary, refresh list.
@@ -0,0 +1,71 @@
# Research: RAG Retrieval Quality Improvements
**Branch**: `004-rag-retrieval-quality` | **Date**: 2026-04-06
## Decision 1: Query Expansion Strategy
**Decision**: Single LLM rewrite — ask the model to restate the user's question using clinical/technical terminology before retrieval. Use the rewritten query for vector search instead of (or in addition to) the original.
**Rationale**: The simplest approach that directly addresses vocabulary mismatch. A single extra LLM call rewrites the query into the language of the documentation (clinical terms), so the embedding similarity search has a much better chance of matching. No new dependencies, no index changes.
**Alternatives considered**:
- *HyDE (Hypothetical Document Embeddings)*: Ask the model to write a hypothetical answer, then embed that. More powerful but adds latency and the answer may itself hallucinate clinical content — rejected for POC.
- *Multi-query retrieval*: Generate 35 alternative queries and merge results. Effective but multiplies retrieval calls and deduplication complexity — rejected (KISS).
- *Synonym dictionary*: Pre-built medical thesaurus mapping. No new dependencies but requires maintenance and won't generalise — rejected.
- *Re-ranking*: Run a cross-encoder after retrieval. Addresses ordering, not vocabulary gap — deferred.
**Implementation note**: The rewrite prompt should be a short, focused instruction: "Rewrite the following question using precise medical/surgical terminology as it would appear in a neurosurgery textbook index. Output only the rewritten question." Use the same `ChatClient` bean; no new API client needed.
---
## Decision 2: Citation Grounding Strategy
**Decision**: Tag each retrieved context section with a short ref-label (`[S1]`, `[S2]`, …, `[Fn]` for figures) in the prompt. Instruct the model to cite only using those labels. Post-process the generated answer to detect and strip any citation that does not correspond to a known label.
**Rationale**: The simplest way to make citations verifiable — all valid citation targets are enumerated in the prompt itself. Post-processing is a pure string operation; no second LLM call needed for validation.
**Alternatives considered**:
- *Ask the model to self-check citations*: Second LLM call to verify each claim. More accurate but doubles cost and latency — rejected for POC.
- *Structured output (JSON)*: Return answer as JSON with claim/source pairs. Most precise but requires significant prompt engineering and frontend changes — deferred.
- *No citation enforcement*: Let the model cite freely and show retrieved sources separately. Already the current state — rejected because it doesn't solve citation hallucination.
**Implementation note**:
- Context sections labelled `[S1] Section Title, p.N` through `[Sk]` in `buildContextPrompt()`.
- Figures labelled `[F1]`, `[F2]` etc.
- System prompt updated: "Cite claims using ONLY the reference labels [S1]…[Sk] and [F1]…[Fj] provided in the context. Do not invent page numbers or section titles."
- `CitationValidatorService` scans generated text for `[Sx]` / `[Fx]` patterns, checks each against the known label set, and removes unknown ones.
- `Message.sources` is already a `List<Map<String,Object>>`. No schema change needed — but we store the ref-label alongside each source so the frontend can correlate.
---
## Decision 3: Frontend Source Display
**Decision**: No frontend change needed for MVP. The backend already filters out hallucinated citations via `CitationValidatorService` before saving the message. The `sources` list attached to the message is already the retrieved set. The existing `ChatMessage.vue` source panel continues to show all retrieved sources (which is correct — they were all used as context).
**Rationale**: KISS. The critical correctness fix (no hallucinated citations in answer text) is fully backend-side. The sources panel shows context that was genuinely available to the model — that is accurate and useful. Linking specific claims to specific sources (inline highlighting) is a UX enhancement that can be a follow-on feature.
**Alternatives considered**:
- *Inline citation links*: Highlight `[S1]` in answer text and link to the source card. Better UX but requires markdown parsing and component changes — deferred.
- *Show only cited sources*: Filter `sources` to only those actually cited in the answer. Marginally more accurate but the uncited context is still legitimately retrieved — deferred.
---
## Decision 4: Persisting Topic Summaries
**Decision**: Add a new `topic_summary` table that stores each generated summary. Summaries are numbered sequentially per topic (summary #1, #2, …). The POST endpoint continues to generate and now also persists. A new GET endpoint lists saved summaries for a topic.
**Rationale**: The simplest approach — one new table, one new repository, minimal changes to the existing service. No caching layer, no event system. The sequential number is derived at query time via `ROW_NUMBER` or counted in the repository, keeping the schema minimal.
**Alternatives considered**:
- *Store in frontend state only (session memory)*: Already the current state — summaries are lost on reload. Rejected because the user explicitly wants persistence.
- *Store summary_number as a persisted column*: Avoids a query-time count but risks gaps/duplicates on concurrent writes. For a POC with single-user use, a `SELECT COUNT(*) + 1` approach at insert time is sufficient.
- *Versioning / soft-delete*: Overkill for a POC where the user just wants to re-read old summaries. No delete endpoint needed initially.
**Implementation note**:
- New Flyway migration `V6__topic_summary.sql`.
- `TopicSummaryEntity` (JPA `@Entity` → table `topic_summary`): `id` (UUID), `topicId` (VARCHAR FK), `summaryNumber` (INT), `summary` (TEXT), `sourcesJson` (TEXT — JSON array), `generatedAt` (TIMESTAMP).
- `summaryNumber` set at insert time: `COUNT(*) WHERE topic_id = ?` + 1.
- `TopicSummaryRepository extends JpaRepository<TopicSummaryEntity, UUID>` with a `findByTopicIdOrderBySummaryNumberAsc` query.
- `TopicSummaryService.generateSummary()` saves the result before returning it.
- `TopicController` gains `GET /api/v1/topics/{id}/summaries` returning `List<SavedSummaryResponse>` (id, summaryNumber, generatedAt — no full text, for list efficiency) and `GET /api/v1/topics/{id}/summaries/{summaryId}` for the full detail.
- Frontend: when a topic card is clicked, fetch the summary list. Show "Summary #1", "Summary #2" chips + "Generate New" button. Clicking a chip loads the full summary.
+111
View File
@@ -0,0 +1,111 @@
# Feature Specification: RAG Retrieval Quality Improvements
**Feature Branch**: `004-rag-retrieval-quality`
**Created**: 2026-04-06
**Status**: Draft
**Input**: User description: "I want to enhance the current RAG system, to avoid common pitfalls like: Vocabulary mismatch in retrieval, where the user's language doesn't overlap with the documentation's terminology and Citation errors in generation, where the model cites a chunk that either wasn't retrieved or doesn't support the specific claim being made"
## Overview
The AI Teacher's question-answering system retrieves relevant content from neurosurgery documentation and generates answers. Two reliability problems reduce trust in the system:
1. **Vocabulary mismatch**: A student asking "what happens after cutting the skull?" may use everyday language while the documentation uses clinical terms like "craniotomy" — causing relevant passages to be missed entirely.
2. **Citation hallucination**: The model sometimes references a section or page in its answer that was not actually retrieved or that does not support the specific claim made, misleading students.
This feature improves both the accuracy of what gets retrieved and the integrity of what the model claims as its sources.
## User Scenarios & Testing *(mandatory)*
### User Story 1 - Accurate Retrieval Despite Different Terminology (Priority: P1)
A medical student asks a question using lay or imprecise language. The system bridges the gap between their vocabulary and the technical terminology used in the textbook, returning contextually relevant passages even when there is no word overlap between the question and the document text.
**Why this priority**: Vocabulary mismatch is the most frequent silent failure — the system returns an answer but based on wrong or empty context, so students receive incorrect information without realising it.
**Independent Test**: Ask the system a question using a common synonym or lay term for a concept that appears in the documentation only under a clinical name. Verify that the retrieved passages contain relevant content about that concept.
**Acceptance Scenarios**:
1. **Given** a question uses a lay term (e.g., "brain swelling"), **When** the system retrieves content, **Then** it returns passages discussing the medically indexed equivalent ("cerebral edema") even though that phrase was not in the question.
2. **Given** a question uses an acronym the documentation spells out fully, **When** retrieval runs, **Then** relevant passages are still found.
3. **Given** a highly technical question that perfectly matches documentation language, **When** retrieval runs, **Then** quality does not regress compared to current behavior.
---
### User Story 2 - Grounded Citation in Generated Answers (Priority: P1)
When the model produces an answer and references a source (section, page, or figure), that source must have been part of the retrieved context and must genuinely support the specific claim being cited.
**Why this priority**: A hallucinated citation is worse than no citation — it gives students false confidence that a claim is documented when it is not.
**Independent Test**: Submit a query, capture the retrieved context passed to the model, and verify that every source reference in the generated answer maps to an identifier present in that context.
**Acceptance Scenarios**:
1. **Given** retrieved context contains sections A, B, and C, **When** the model generates an answer citing "Section D", **Then** that citation is either removed or flagged before the answer is shown to the user.
2. **Given** a retrieved passage is used as context, **When** the model cites it, **Then** the cited passage actually contains the information the citation supports.
3. **Given** no relevant context was retrieved for part of a query, **When** the model responds, **Then** it acknowledges uncertainty rather than fabricating a source.
---
### User Story 3 - User Visibility into Retrieval Confidence (Priority: P2)
A student can see which parts of the answer are well-supported by the retrieved material and which parts carry lower confidence, allowing them to judge how much to rely on each claim.
**Why this priority**: Transparency builds appropriate trust — students should know when the system is uncertain rather than presenting all answers with equal authority.
**Independent Test**: Ask a question where only partial relevant content exists in the corpus. Verify the answer visually differentiates well-supported claims from lower-confidence statements.
**Acceptance Scenarios**:
1. **Given** an answer contains a claim backed by a directly retrieved passage, **When** displayed, **Then** the source is shown alongside the claim.
2. **Given** an answer contains a claim not directly covered by retrieved content, **When** displayed, **Then** the system signals lower confidence or absence of a source for that claim.
---
### Edge Cases
- What happens when query enrichment produces terms that retrieve completely unrelated passages?
- How does the system behave when the retrieved context is very short or empty?
- What if two retrieved sections contradict each other — how does citation work in that case?
- What if citation verification removes all citations from an answer (model cited nothing valid)?
- How does the system handle questions in languages other than the documentation language?
## Requirements *(mandatory)*
### Functional Requirements
- **FR-001**: The system MUST enrich user queries to bridge vocabulary gaps before retrieval, using the domain context of the book being queried.
- **FR-002**: The system MUST retrieve passages based on the enriched query, not solely on the literal user input.
- **FR-003**: The system MUST pass only verified, retrieved passage identifiers to the generation step as eligible citation targets.
- **FR-004**: The system MUST validate that every source reference in a generated answer corresponds to a passage present in the retrieved context.
- **FR-005**: The system MUST suppress or flag any citation in the generated answer that refers to a passage not present in the retrieved context.
- **FR-006**: The system MUST surface to the user which retrieved passages were used to support each claim (or indicate absence of supporting source).
- **FR-007**: When no relevant passages are retrieved, the system MUST communicate this clearly rather than generating an unsupported answer.
- **FR-008**: Query enrichment MUST be scoped to the active book to avoid introducing terminology from unrelated domains.
### Key Entities
- **Enriched Query**: The augmented version of the user's original question, including synonyms, alternate phrasings, or domain-aligned terms used for retrieval.
- **Retrieved Context**: The set of passages (sections and figures) returned by retrieval, each identified by a unique source reference, passed to generation.
- **Citation**: A reference in the generated answer to a specific source; must be traceable to a member of the Retrieved Context.
- **Citation Validation Result**: A per-citation judgment of whether the cited source was retrieved and supports the claim.
## Success Criteria *(mandatory)*
### Measurable Outcomes
- **SC-001**: Questions using common lay synonyms for clinical terms retrieve relevant passages at least 80% of the time (verified on a manually curated test set of synonym-query pairs).
- **SC-002**: Zero citations appear in generated answers that reference a source not present in the retrieved context passed to the model.
- **SC-003**: For every generated answer, 100% of displayed citations can be traced to a retrieved passage identifier.
- **SC-004**: Retrieval quality for queries that already match documentation vocabulary does not degrade (baseline score maintained or improved).
- **SC-005**: Users can identify, for each factual claim in an answer, whether a supporting source was found — without needing to ask a follow-up question.
## Assumptions
- The existing retrieval pipeline returns section and figure identifiers that can be used as citation anchors.
- Query enrichment operates at query time (not ingestion time), so no changes to stored embeddings are needed.
- The generation model can be instructed via prompt to restrict citations to a provided list of identifiers.
- Citation validation is performed after generation but before the answer is shown to the user (post-processing step).
- Mobile or offline support is out of scope for this feature.
- Multi-language support (non-English questions against English documentation) is a future concern and not addressed here.
+250
View File
@@ -0,0 +1,250 @@
# Tasks: RAG Retrieval Quality Improvements
**Input**: Design documents from `/specs/004-rag-retrieval-quality/`
**Prerequisites**: plan.md ✅, spec.md ✅, research.md ✅, data-model.md ✅, contracts/ ✅
**Organization**: Tasks are grouped by user story to enable independent implementation and testing of each story.
**Tests**: Not requested in spec — no test tasks generated.
## Format: `[ID] [P?] [Story] Description`
- **[P]**: Can run in parallel (different files, no dependencies)
- **[Story]**: Which user story this task belongs to (US1, US2, US3)
---
## Phase 1: Setup (Shared Infrastructure)
**Purpose**: No new project structure needed — services are added to the existing `retrieval/` package. This phase is a single verification step.
- [x] T001 Verify active branch is `004-rag-retrieval-quality` and `backend/src/main/java/com/aiteacher/retrieval/` exists
---
## Phase 2: Foundational (Blocking Prerequisites)
**Purpose**: Two lightweight value objects shared by both user stories.
**⚠️ CRITICAL**: Both user story phases depend on these records being present.
- [x] T002 Create `ExpandedQuery` record in `backend/src/main/java/com/aiteacher/retrieval/ExpandedQuery.java` with fields `String original` and `String rewritten`
- [x] T003 [P] Create `LabelledContext` record in `backend/src/main/java/com/aiteacher/retrieval/LabelledContext.java` with fields `Map<String, SectionEntity> sectionLabels`, `Map<String, FigureEntity> figureLabels`, and `String promptText`
**Checkpoint**: Foundation ready — US1 and US2 implementation can begin in parallel
---
## Phase 3: User Story 1 — Accurate Retrieval Despite Different Terminology (Priority: P1) 🎯 MVP
**Goal**: Before each retrieval call, rewrite the user's question into clinical terminology so that vector search finds relevant sections even when the user uses lay language.
**Independent Test**: Ask "what happens after cutting the skull?" — verify retrieved sections contain content about craniotomy without that word appearing in the query.
### Implementation for User Story 1
- [x] T004 [US1] Create `QueryExpansionService` in `backend/src/main/java/com/aiteacher/retrieval/QueryExpansionService.java`:
- Constructor-inject `ChatClient`
- Method `expand(String query): ExpandedQuery`
- LLM prompt: *"Rewrite the following question using precise medical/surgical terminology as it would appear in a neurosurgery textbook index. Output only the rewritten question, nothing else. Question: {query}"*
- Return `new ExpandedQuery(query, rewrittenText)`
- Annotate with `@Service`
- [x] T005 [US1] Modify `NeurosurgeryRetriever.retrieve()` in `backend/src/main/java/com/aiteacher/retrieval/NeurosurgeryRetriever.java`:
- Change method signature from `retrieve(String query, UUID bookId)` to `retrieve(String query, UUID bookId)` — no signature change; just use `query` for vector search (already correct; no change needed here unless query is pre-expanded by caller)
- *Note*: expansion is done in ChatService before calling retrieve, so no change to NeurosurgeryRetriever is required
- [x] T006 [US1] Modify `ChatService` in `backend/src/main/java/com/aiteacher/chat/ChatService.java`:
- Constructor-inject `QueryExpansionService`
- In `sendMessage()`, call `queryExpansionService.expand(fullQuestion)` before the retrieval loop
- Pass `expandedQuery.rewritten()` to `retriever.retrieve()` instead of `fullQuestion`
- Keep passing `fullQuestion` (original) to `buildContextPrompt()` so the QUESTION block shown to the model reflects what the user actually asked
**Checkpoint**: User Story 1 fully functional — retrieval now uses clinically rewritten queries
---
## Phase 4: User Story 2 — Grounded Citation in Generated Answers (Priority: P1)
**Goal**: Tag all retrieved sections and figures with short ref-labels (`[S1]`, `[F1]`…) in the prompt, instruct the model to cite only those labels, then post-process the answer to strip any citation referencing a label that was not provided.
**Independent Test**: Trigger a question where only sections S1S3 are retrieved. Verify the generated answer contains no citation outside that set, and the `sources` list in the response carries `refLabel` fields.
### Implementation for User Story 2
- [x] T007 [US2] Create `CitationValidatorService` in `backend/src/main/java/com/aiteacher/retrieval/CitationValidatorService.java`:
- Annotate with `@Service`
- Method `validate(String generatedAnswer, Set<String> validLabels): String`
- Scan `generatedAnswer` for occurrences of `[Sn]` and `[Fn]` patterns using a regex like `\[(S|F)\d+\]`
- Remove (or replace with empty string) any match whose label is not in `validLabels`
- Return the cleaned answer text
- [x] T008 [US2] Modify `ChatService.buildContextPrompt()` in `backend/src/main/java/com/aiteacher/chat/ChatService.java`:
- Change signature to return `LabelledContext` instead of `String`
- Assign sequential labels: sections get `S1`, `S2`, …; figures get `F1`, `F2`, …
- Prefix each section block with its label: `[S1] Section Title, p.N\n{fullText}\n\n`
- Prefix each figure line with its label: `[F1] Fig. X (p.N): caption`
- Populate `sectionLabels` and `figureLabels` maps in the returned `LabelledContext`
- Store the full formatted prompt in `LabelledContext.promptText()`
- [x] T009 [US2] Update system prompt constant in `backend/src/main/java/com/aiteacher/chat/ChatService.java`:
- Replace the citation rule *"Cite sources for each major point (book title and page number from the context)"* with: *"Cite claims using ONLY the reference labels provided in the context (e.g. [S1], [F2]). Do not invent page numbers, section titles, or labels not present in the CONTEXT block."*
- [x] T010 [US2] Wire `CitationValidatorService` into `ChatService.sendMessage()` in `backend/src/main/java/com/aiteacher/chat/ChatService.java`:
- Constructor-inject `CitationValidatorService`
- After the `chatClient.prompt()...call().content()` call, pass `assistantContent` and the label set from `LabelledContext` to `citationValidatorService.validate()`
- Use the validated string as `assistantContent` going forward
- [x] T011 [US2] Modify `buildSources()` in `backend/src/main/java/com/aiteacher/chat/ChatService.java`:
- Accept the `LabelledContext` (or its two maps) as an additional parameter
- Add `"refLabel"` entry to each source map: e.g. `source.put("refLabel", "S1")` for sections, `source.put("refLabel", "F1")` for figures
- Keep all other existing fields unchanged
- [x] T012 [US2] Update `sendMessage()` call chain in `backend/src/main/java/com/aiteacher/chat/ChatService.java` to thread `LabelledContext` through steps T008T011:
- `LabelledContext ctx = buildContextPrompt(fullQuestion, allSections, allFigures)`
- Pass `ctx.promptText()` to the LLM call
- Pass `ctx` label maps to `validate()` and `buildSources()`
**Checkpoint**: User Stories 1 and 2 both fully functional — queries are expanded, citations are grounded
---
## Phase 5: User Story 3 — User Visibility into Retrieval Confidence (Priority: P2)
**Goal**: The answer text contains `[S1]`-style labels (after US2). This phase exposes them in the frontend so users can see which claim maps to which source card.
**Independent Test**: Send a question, receive an answer with inline `[S1]` labels visible in the rendered text, and confirm clicking/hovering the label highlights the corresponding source card.
**Note**: Per research.md, the backend is already complete after US1+US2. US3 is a frontend-only UX enhancement.
### Implementation for User Story 3
- [x] T013 [US3] Modify `ChatMessage.vue` in `frontend/src/components/ChatMessage.vue`:
- Parse the answer text for `[Sn]` and `[Fn]` citation labels using a regex
- Render each label as a styled inline badge (e.g. `<span class="citation-badge">[S1]</span>`)
- When a badge is clicked or hovered, highlight the corresponding source card (match by `source.refLabel`)
- [x] T014 [US3] Update source card rendering in `frontend/src/components/ChatMessage.vue`:
- Add a `data-ref-label` attribute to each source card element so it can be targeted by the citation badge interaction
- Apply a visual highlight style (CSS class) when the card is active
**Checkpoint**: All three user stories functional — full end-to-end quality improvements delivered
---
## Phase 6: User Story 4 — Topic Summary Persistence & History (user-requested)
**Goal**: Every generated topic summary is saved to the database. When a topic is selected the UI shows a numbered history list; the student can view any past summary or generate a new one.
**Independent Test**: Generate a summary for "Intracranial Aneurysms", reload the page, click the topic — verify "Summary #1" appears. Generate again — verify "Summary #2" appears. Click "Summary #1" — verify the original text loads without regeneration.
- [x] T018 Create Flyway migration `backend/src/main/resources/db/migration/V6__topic_summary.sql` — table `topic_summary` with columns: `id UUID PRIMARY KEY DEFAULT gen_random_uuid()`, `topic_id VARCHAR(100) NOT NULL`, `summary_number INT NOT NULL`, `summary TEXT NOT NULL`, `sources_json TEXT NOT NULL`, `generated_at TIMESTAMPTZ NOT NULL`
- [x] T019 [P] [US4] Create `TopicSummaryEntity.java` in `backend/src/main/java/com/aiteacher/topic/TopicSummaryEntity.java` — JPA `@Entity` mapped to table `topic_summary`; fields: `@Id UUID id`, `String topicId`, `int summaryNumber`, `String summary`, `String sourcesJson`, `Instant generatedAt`; no-arg + all-args constructor
- [x] T02X [P] [US4] Create `SavedSummaryItem.java` record in `backend/src/main/java/com/aiteacher/topic/SavedSummaryItem.java` — fields: `UUID id`, `int summaryNumber`, `Instant generatedAt` (list-view DTO, no full text)
- [x] T02X [US4] Create `TopicSummaryRepository.java` in `backend/src/main/java/com/aiteacher/topic/TopicSummaryRepository.java``extends JpaRepository<TopicSummaryEntity, UUID>`; add `List<TopicSummaryEntity> findByTopicIdOrderBySummaryNumberAsc(String topicId)` and `long countByTopicId(String topicId)`
- [x] T02X [US4] Modify `TopicSummaryResponse.java` in `backend/src/main/java/com/aiteacher/topic/TopicSummaryResponse.java` — add fields `UUID id` and `int summaryNumber` to the record components
- [x] T02X [US4] Modify `TopicSummaryService.java` in `backend/src/main/java/com/aiteacher/topic/TopicSummaryService.java` — inject `TopicSummaryRepository` and `ObjectMapper`; at end of `generateSummary()` compute `summaryNumber = (int) repository.countByTopicId(topicId) + 1`, persist a `TopicSummaryEntity` (serialise `sources` list to JSON via `objectMapper.writeValueAsString()`), and include `id` + `summaryNumber` in the returned `TopicSummaryResponse`; add `List<SavedSummaryItem> listSummaries(String topicId)` and `TopicSummaryResponse getSummary(UUID summaryId)` methods
- [x] T02X [US4] Modify `TopicController.java` in `backend/src/main/java/com/aiteacher/topic/TopicController.java` — add `@GetMapping("/{id}/summaries")` returning `List<SavedSummaryItem>` (delegates to `listSummaries`); add `@GetMapping("/{id}/summaries/{summaryId}")` returning `TopicSummaryResponse` (delegates to `getSummary`); both return 404 via `NoSuchElementException` when topic or summary not found
- [x] T02X [US4] Modify `topicStore.ts` in `frontend/src/stores/topicStore.ts` — add state `summaryList: SavedSummaryItem[]`; add `fetchSummaries(topicId)` action calling `GET /api/v1/topics/{topicId}/summaries`; add `fetchSummaryDetail(topicId, summaryId)` action calling `GET /api/v1/topics/{topicId}/summaries/{summaryId}` and setting `activeSummary`; clear `summaryList` when a different topic is selected
- [x] T02X [US4] Modify `TopicsView.vue` in `frontend/src/views/TopicsView.vue` — when a topic card is clicked: (1) call `topicStore.fetchSummaries(topicId)` first; (2) if summaries exist, display a summary history list showing chips "Summary #1 · [date]", "Summary #2 · [date]", … + a "Generate New" button; (3) clicking a chip calls `fetchSummaryDetail()` and renders the saved summary in the existing panel; (4) clicking "Generate New" calls `handleGenerate()` then re-calls `fetchSummaries()` to refresh the list; (5) if no summaries exist, show only the "Generate Summary" button (current behaviour)
**Checkpoint**: Summary persistence fully working end-to-end. US4 independently testable.
---
## Phase 7: Polish & Cross-Cutting Concerns
**Purpose**: Constitution IV compliance and cleanup.
- [x] T027 Update `README.md` Mermaid architecture diagram to add `QueryExpansionService` and `CitationValidatorService` to the chat pipeline flow, and the `topic_summary` table to the data diagram (required by Constitution Principle IV — must be in the same PR)
- [x] T028 [P] Log the expanded query at DEBUG level in `QueryExpansionService` (e.g. `log.debug("Query expanded: '{}' → '{}'", original, rewritten)`) for observability
- [x] T029 [P] Log stripped citation labels at WARN level in `CitationValidatorService` when any labels are removed (e.g. `log.warn("Stripped hallucinated citations: {}", removedLabels)`)
---
## Dependencies & Execution Order
### Phase Dependencies
- **Phase 1 (Setup)**: No dependencies — start immediately
- **Phase 2 (Foundational)**: Depends on Phase 1 — blocks all user story phases
- **Phase 3 (US1)**: Depends on Phase 2 (needs `ExpandedQuery`)
- **Phase 4 (US2)**: Depends on Phase 2 (needs `LabelledContext`); can run in parallel with Phase 3
- **Phase 5 (US3)**: Depends on Phase 4 (needs `refLabel` in sources)
- **Phase 6 (US4)**: No dependency on Phase 2 for the migration (T018); entity/service work (T019+) depends on T018
- **Phase 7 (Polish)**: Depends on all implementation phases complete
### User Story Dependencies
- **User Story 1 (P1)**: Depends on Phase 2 only — no dependency on US2 or US3
- **User Story 2 (P1)**: Depends on Phase 2 only — can run in parallel with US1
- **User Story 3 (P2)**: Depends on US2 (needs `refLabel` in the API response)
- **User Story 4**: Independent of US1US3 — can start immediately after T018 migration
### Within Each User Story
- T004 → T006 (QueryExpansionService must exist before ChatService wiring)
- T007 → T010 → T012 (CitationValidatorService → wire into sendMessage → thread context)
- T008 → T012 (LabelledContext must be built before threading through)
- T013 → T014 (badge rendering before card targeting)
### Parallel Opportunities
- T002 and T003 (Phase 2) can run in parallel — different files
- Phase 3 (US1) and Phase 4 (US2) can run in parallel after Phase 2 — all different files
- T015, T016, T017 (Polish) can run in parallel — different files
---
## Parallel Example: US1 + US2
```
After Phase 2 completes:
Track A (US1):
T004 — Create QueryExpansionService
T005 — (no change to NeurosurgeryRetriever)
T006 — Wire into ChatService
Track B (US2):
T007 — Create CitationValidatorService
T008 — Modify buildContextPrompt() → LabelledContext
T009 — Update system prompt
T010 — Wire CitationValidatorService into sendMessage()
T011 — Add refLabel to buildSources()
T012 — Thread LabelledContext through call chain
Merge point: Both tracks modify ChatService — coordinate T006 and T012
to avoid conflicts (implement T006 first or use feature branches).
```
---
## Implementation Strategy
### MVP First (User Stories 1 + 2 — both P1)
1. Complete Phase 1: Setup (T001)
2. Complete Phase 2: Foundational (T002, T003)
3. Complete Phase 3: US1 — query expansion (T004T006)
4. **VALIDATE**: Ask a lay-language question; confirm relevant clinical passages are retrieved
5. Complete Phase 4: US2 — citation grounding (T007T012)
6. **VALIDATE**: Confirm no `[Sx]` label appears in the answer that wasn't in the retrieved set
7. **STOP and DEMO**: Both P1 stories deliver the core reliability improvements
### Incremental Delivery
1. Phase 1 + 2 → infrastructure ready
2. Phase 3 → vocabulary mismatch fixed → demo-able
3. Phase 4 → citation hallucination fixed → demo-able
4. Phase 5 → citation badges in UI → UX polish
5. Phase 6 → README + logging → PR-ready
---
## Notes
- `ChatService` is modified by both US1 (T006) and US2 (T008T012) — coordinate edits or implement sequentially
- `buildContextPrompt()` changes return type from `String` to `LabelledContext` (T008) — update all callers in the same task
- The system prompt change (T009) is a one-line string edit inside `ChatService`; no separate class needed
- `CitationValidatorService` operates purely on strings — no DB or AI dependency, easy to unit-test manually
- US3 frontend tasks (T013T014) are entirely in `ChatMessage.vue` — no backend change