ai-teacher/specs/002-image-aware-embedding/spec.md

# Feature Specification: Enhanced Embedding with Image Parsing and Metadata

**Feature Branch**: `002-image-aware-embedding`  
**Created**: 2026-04-03  
**Status**: Draft  
**Input**: User description: "I want to enhance the embedding process. I want also parse image from each pages if any and add proper metadata so that it can match the retrieved chunk/vector that match what user are querying."

## User Scenarios & Testing *(mandatory)*

### User Story 1 - Image Content Surfaced in Query Results (Priority: P1)

A neurosurgeon asks a question in the chat (e.g., "Show me the anatomy of the Circle of Willis")
that is best answered by a diagram or figure in an uploaded book. The system retrieves the image
content — its description and surrounding context — and uses it to construct a grounded answer,
citing the page and book where the image appeared.

**Why this priority**: This is the direct, user-visible payoff of the feature. Without it, the
enhancement has no observable benefit. All other stories support this outcome.

**Independent Test**: Upload a book containing a labelled anatomical diagram. Ask a query whose
answer is conveyed by that diagram (not in the surrounding text). Confirm the system returns an
answer that references the diagram's content and cites the correct book and page.

**Acceptance Scenarios**:

1. **Given** a book with an anatomical diagram on page 42, **When** a user asks a question whose
   answer is only depicted in that diagram, **Then** the system returns a response that draws on
   the diagram's content and cites "Page 42, [Book Title]".
2. **Given** a page with both text and an image, **When** the system retrieves that page's content,
   **Then** the image-derived content and the surrounding text are each independently retrievable
   and independently citable.
3. **Given** a query that has no relevant image in any uploaded book, **When** the system searches,
   **Then** it does not fabricate image-derived content and falls back to text-only results (or
   states no relevant content was found).

---

### User Story 2 - All Pages Scanned for Images During Embedding (Priority: P1)

When a book is uploaded and processed, every page is inspected for images. Any image found is
extracted and represented as a searchable content chunk enriched with metadata (page number,
book title, position on page, caption if present). Pages without images are processed as
text-only chunks, unchanged from the existing behaviour.

**Why this priority**: This is the prerequisite for User Story 1. Without systematic per-page
image detection, image content cannot be retrieved.

**Independent Test**: Upload a book whose pages include a mix of text-only and image-containing
pages. After processing completes, verify that chunks exist for each image page and that each
image chunk carries the correct metadata (page number, source book, caption).

**Acceptance Scenarios**:

1. **Given** a book being processed, **When** the embedding pipeline runs, **Then** every page
   is evaluated for images and each detected image generates at least one content chunk.
2. **Given** an image with a caption or label, **When** the chunk is created, **Then** the
   caption or label text is included in the chunk's content and metadata.
3. **Given** a page with multiple images, **When** processing completes, **Then** each image is
   represented as a separate chunk with its own metadata, not merged into a single chunk.
4. **Given** a page with no images, **When** processing completes, **Then** no image chunk is
   created for that page and text processing is unaffected.

---

### User Story 3 - Rich Metadata Enables Precise Source Attribution (Priority: P2)

When the system returns a result based on image content, the user can see exactly where that
image appeared: which book, which page, and what type of content (diagram, table, photograph,
etc.). This gives the user confidence in the source and lets them locate the original image
in their physical or digital copy of the book.

**Why this priority**: Metadata quality directly impacts user trust. Neurosurgeons require
traceable, citable evidence. Richer metadata also improves retrieval accuracy by giving the
search engine more signals to match against a query.

**Independent Test**: Retrieve a result sourced from an image chunk. Inspect the displayed
citation and verify it includes: book title, page number, content type (e.g., "diagram"),
and caption (if present in the original).

**Acceptance Scenarios**:

1. **Given** a retrieved image chunk, **When** the system displays the source citation,
   **Then** the citation includes at minimum: book title, page number, and a content-type
   label (e.g., diagram, table, figure).
2. **Given** an image chunk with a detected caption, **When** the citation is displayed,
   **Then** the caption text is shown alongside the other metadata fields.
3. **Given** a topic summary that draws on both text and image chunks, **When** the user
   inspects citations, **Then** image-sourced and text-sourced claims are distinguishable
   from each other.

---

### Edge Cases

- What happens when an image is too small to contain meaningful content (e.g., a decorative
  bullet icon or a publisher logo)?
- How does the system handle a page that is entirely an image (scanned page with no digital text)?
- What if an image spans multiple pages (e.g., a fold-out diagram)?
- How does the system behave when an image has no caption and its surrounding text provides
  no useful context?
- What happens if image processing fails for a specific page — does it abort the whole book
  or continue with the remaining pages?

## Requirements *(mandatory)*

### Functional Requirements

- **FR-001**: System MUST inspect every page of an uploaded book for the presence of images
  during the embedding process.
- **FR-002**: System MUST extract each detected image and create a dedicated, independently
  searchable content chunk for it.
- **FR-003**: System MUST generate a descriptive textual representation of each extracted
  image so its content is semantically searchable by the retrieval system.
- **FR-004**: System MUST associate the following metadata with every image chunk: book title,
  page number, content type (e.g., diagram, table, figure, photograph), and caption text
  (where present).
- **FR-005**: System MUST include the same base metadata (book title, page number) on text
  chunks so that all retrieved content — image or text — carries consistent, comparable
  source attribution.
- **FR-006**: System MUST treat image chunks as first-class retrievable units: they must be
  ranked and returned alongside text chunks when they are relevant to a user query.
- **FR-007**: System MUST skip images that fall below a minimum meaningful-content threshold
  (e.g., decorative icons, page separators) and MUST NOT create chunks for them.
- **FR-008**: If image processing fails for a specific page, the system MUST log the failure,
  skip that page's image, and continue processing the remaining pages and text content of
  the book.
- **FR-009**: System MUST display image-sourced content citations distinctly from text-sourced
  citations so users can identify when a result originates from a visual element.
- **FR-010**: Processing a book that contains images MUST NOT degrade the accuracy or
  completeness of the existing text-only embedding for that book.

### Key Entities

- **Image Chunk**: A searchable content unit derived from a page image. Attributes: generated
  description, source book title, page number, content type, caption (optional), embedding vector.
- **Text Chunk**: Existing unit; extended to carry explicit metadata: source book title,
  page number, section heading (if detectable), content type ("text").
- **Chunk Metadata**: Structured attributes attached to every chunk regardless of type,
  enabling consistent filtering and citation. Mandatory fields: book title, page number,
  content type. Optional fields: caption, section heading.

## Success Criteria *(mandatory)*

### Measurable Outcomes

- **SC-001**: At least 90% of pages containing images in a test book result in a retrievable
  image chunk after processing completes.
- **SC-002**: A controlled set of 10 queries whose answers are conveyed by diagrams in an
  uploaded book returns at least 7 correct image-sourced answers (70% recall on image queries).
- **SC-003**: Embedding processing time for a book with images increases by no more than 3×
  compared to processing the same book as text-only, for books up to 500 pages.
- **SC-004**: Every retrieved result — text or image — includes a citation that identifies
  at minimum the source book title and page number, with 100% coverage across a test result set.
- **SC-005**: In a user evaluation with 5 representative queries that previously returned
  no useful results (because the answer was only in a diagram), at least 4 now return a
  useful, grounded answer.

## Assumptions

- Books are still uploaded exclusively as PDFs; image parsing applies to PDF pages only.
- The platform already has a working text-only embedding pipeline (from feature 001); this
  feature enhances it without replacing or rewriting the text processing logic.
- Images worth processing are those that occupy a meaningful portion of the page; small
  decorative or structural images (logos, dividers, icons) are excluded based on a size
  threshold determined during implementation.
- The descriptive representation of an image (FR-003) is generated at embedding time, not
  at query time; query latency is not affected by image interpretation.
- The shared global book library model from feature 001 is retained; image chunks from a
  processed book are available to all users immediately upon completion.
- Scanned pages (fully rasterised pages with no digital text layer) are treated as a single
  full-page image; the system attempts to extract content from them but does not guarantee
  the same fidelity as pages with digital text.
- Per-chunk metadata is stored alongside the vector so it can be used for both retrieval
  filtering and source citation display without a separate lookup.
- Books already processed under feature 001 (text-only) are not automatically re-processed;
  re-embedding must be triggered explicitly by the user or an administrator.