5 Commits

Author SHA1 Message Date
Adrien e5d53b4e80 add possibility to disable delete and upload of books 2026-04-06 14:09:17 +02:00
Adrien 5c641f4bcc enhance page parsing using json output and html 2026-04-05 21:55:30 +02:00
Adrien ea1276dc2e adding Marker to parse effectively pdf 2026-04-04 21:30:18 +02:00
Adrien b154e29f2d s3 bucket integration for image storage 2026-04-04 13:26:55 +02:00
Adrien 5acfdd33c1 first implementation - image/drawing integration 2026-04-04 12:56:56 +02:00
59 changed files with 5073 additions and 160 deletions
+12
View File
@@ -1,3 +1,15 @@
# Runtime uploads (extracted figures)
uploads/
# Java build
target/
*.class
*.jar
# Node
node_modules/
dist/
# OS # OS
.DS_Store .DS_Store
Thumbs.db Thumbs.db
+9 -2
View File
@@ -1,8 +1,13 @@
# ai-teacher Development Guidelines # ai-teacher Development Guidelines
Auto-generated from all feature plans. Last updated: 2026-03-31 Auto-generated from all feature plans. Last updated: 2026-04-04
## Active Technologies ## Active Technologies
- Java 25 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API (embeddings + chat), PDFBox (via Spring AI PDF reader dependency) (002-image-aware-embedding)
- PostgreSQL (JPA + Flyway), pgvector (Spring AI `VectorStore`), local file system (extracted images — `/uploads/figures/`) (002-image-aware-embedding)
- Java 25 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API, PDFBox (rendering only), `com.google.cloud:google-cloud-documentai` (~2.40.x) (002-image-aware-embedding)
- PostgreSQL (JPA + Flyway), pgvector (Spring AI VectorStore), S3 / local filesystem (figure images) (002-image-aware-embedding)
- PostgreSQL (JPA + Flyway), pgvector (Spring AI `VectorStore`), S3-compatible (002-image-aware-embedding)
- Java 21 (backend), TypeScript / Node 20 (frontend) (001-neuro-rag-learning) - Java 21 (backend), TypeScript / Node 20 (frontend) (001-neuro-rag-learning)
@@ -22,8 +27,10 @@ npm test && npm run lint
Java 21 (backend), TypeScript / Node 20 (frontend): Follow standard conventions Java 21 (backend), TypeScript / Node 20 (frontend): Follow standard conventions
## Recent Changes ## Recent Changes
- 002-image-aware-embedding: Added Java 25 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API (embeddings +
- 002-image-aware-embedding: Added Java 25 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API, PDFBox (rendering only), `com.google.cloud:google-cloud-documentai` (~2.40.x)
- 002-image-aware-embedding: Added Java 25 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API (embeddings + chat), PDFBox (via Spring AI PDF reader dependency)
- 001-neuro-rag-learning: Added Java 21 (backend), TypeScript / Node 20 (frontend)
<!-- MANUAL ADDITIONS START --> <!-- MANUAL ADDITIONS START -->
<!-- MANUAL ADDITIONS END --> <!-- MANUAL ADDITIONS END -->
+566
View File
@@ -0,0 +1,566 @@
# Marker
Marker converts documents to markdown, JSON, chunks, and HTML quickly and accurately.
- Converts PDF, image, PPTX, DOCX, XLSX, HTML, EPUB files in all languages
- Formats tables, forms, equations, inline math, links, references, and code blocks
- Extracts and saves images
- Removes headers/footers/other artifacts
- Extensible with your own formatting and logic
- Does structured extraction, given a JSON schema (beta)
- Optionally boost accuracy with LLMs (and your own prompt)
- Works on GPU, CPU, or MPS
For our managed API or on-prem document intelligence solution, check out [our platform here](https://datalab.to?utm_source=gh-marker).
## Performance
<img src="data/images/overall.png" width="800px"/>
Marker benchmarks favorably compared to cloud services like Llamaparse and Mathpix, as well as other open source tools.
The above results are running single PDF pages serially. Marker is significantly faster when running in batch mode, with a projected throughput of 25 pages/second on an H100.
See [below](#benchmarks) for detailed speed and accuracy benchmarks, and instructions on how to run your own benchmarks.
## Hybrid Mode
For the highest accuracy, pass the `--use_llm` flag to use an LLM alongside marker. This will do things like merge tables across pages, handle inline math, format tables properly, and extract values from forms. It can use any gemini or ollama model. By default, it uses `gemini-2.0-flash`. See [below](#llm-services) for details.
Here is a table benchmark comparing marker, gemini flash alone, and marker with use_llm:
<img src="data/images/table.png" width="400px"/>
As you can see, the use_llm mode offers higher accuracy than marker or gemini alone.
## Examples
| PDF | File type | Markdown | JSON |
|-----|-----------|------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------|
| [Think Python](https://greenteapress.com/thinkpython/thinkpython.pdf) | Textbook | [View](https://github.com/VikParuchuri/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md) | [View](https://github.com/VikParuchuri/marker/blob/master/data/examples/json/thinkpython.json) |
| [Switch Transformers](https://arxiv.org/pdf/2101.03961.pdf) | arXiv paper | [View](https://github.com/VikParuchuri/marker/blob/master/data/examples/markdown/switch_transformers/switch_trans.md) | [View](https://github.com/VikParuchuri/marker/blob/master/data/examples/json/switch_trans.json) |
| [Multi-column CNN](https://arxiv.org/pdf/1804.07821.pdf) | arXiv paper | [View](https://github.com/VikParuchuri/marker/blob/master/data/examples/markdown/multicolcnn/multicolcnn.md) | [View](https://github.com/VikParuchuri/marker/blob/master/data/examples/json/multicolcnn.json) |
# Commercial usage
Our model weights use a modified AI Pubs Open Rail-M license (free for research, personal use, and startups under $2M funding/revenue) and our code is GPL. For broader commercial licensing or to remove GPL requirements, visit our pricing page [here](https://www.datalab.to/pricing?utm_source=gh-marker).
# Hosted API & On-prem
There's a [hosted API](https://www.datalab.to?utm_source=gh-marker) and [painless on-prem solution](https://www.datalab.to/blog/self-serve-on-prem-licensing) for marker - it's free to sign up, and we'll throw in credits for you to test it out.
The API:
- Supports PDF, image, PPT, PPTX, DOC, DOCX, XLS, XLSX, HTML, EPUB files
- Is 1/4th the price of leading cloud-based competitors
- Fast - ~15s for a 250 page PDF
- Supports LLM mode
- High uptime (99.99%)
# Community
[Discord](https://discord.gg//KuZwXNGnfH) is where we discuss future development.
# Installation
You'll need python 3.10+ and [PyTorch](https://pytorch.org/get-started/locally/).
Install with:
```shell
pip install marker-pdf
```
If you want to use marker on documents other than PDFs, you will need to install additional dependencies with:
```shell
pip install marker-pdf[full]
```
# Usage
First, some configuration:
- Your torch device will be automatically detected, but you can override this. For example, `TORCH_DEVICE=cuda`.
- Some PDFs, even digital ones, have bad text in them. Set `--force_ocr` to force OCR on all lines, or the `strip_existing_ocr` to keep all digital text, and strip out any existing OCR text.
- If you care about inline math, set `force_ocr` to convert inline math to LaTeX.
## Interactive App
I've included a streamlit app that lets you interactively try marker with some basic options. Run it with:
```shell
pip install streamlit streamlit-ace
marker_gui
```
## Convert a single file
```shell
marker_single /path/to/file.pdf
```
You can pass in PDFs or images.
Options:
- `--page_range TEXT`: Specify which pages to process. Accepts comma-separated page numbers and ranges. Example: `--page_range "0,5-10,20"` will process pages 0, 5 through 10, and page 20.
- `--output_format [markdown|json|html|chunks]`: Specify the format for the output results.
- `--output_dir PATH`: Directory where output files will be saved. Defaults to the value specified in settings.OUTPUT_DIR.
- `--paginate_output`: Paginates the output, using `\n\n{PAGE_NUMBER}` followed by `-` * 48, then `\n\n`
- `--use_llm`: Uses an LLM to improve accuracy. You will need to configure the LLM backend - see [below](#llm-services).
- `--force_ocr`: Force OCR processing on the entire document, even for pages that might contain extractable text. This will also format inline math properly.
- `--block_correction_prompt`: if LLM mode is active, an optional prompt that will be used to correct the output of marker. This is useful for custom formatting or logic that you want to apply to the output.
- `--strip_existing_ocr`: Remove all existing OCR text in the document and re-OCR with surya.
- `--redo_inline_math`: If you want the absolute highest quality inline math conversion, use this along with `--use_llm`.
- `--disable_image_extraction`: Don't extract images from the PDF. If you also specify `--use_llm`, then images will be replaced with a description.
- `--debug`: Enable debug mode for additional logging and diagnostic information.
- `--processors TEXT`: Override the default processors by providing their full module paths, separated by commas. Example: `--processors "module1.processor1,module2.processor2"`
- `--config_json PATH`: Path to a JSON configuration file containing additional settings.
- `config --help`: List all available builders, processors, and converters, and their associated configuration. These values can be used to build a JSON configuration file for additional tweaking of marker defaults.
- `--converter_cls`: One of `marker.converters.pdf.PdfConverter` (default) or `marker.converters.table.TableConverter`. The `PdfConverter` will convert the whole PDF, the `TableConverter` will only extract and convert tables.
- `--llm_service`: Which llm service to use if `--use_llm` is passed. This defaults to `marker.services.gemini.GoogleGeminiService`.
- `--help`: see all of the flags that can be passed into marker. (it supports many more options then are listed above)
The list of supported languages for surya OCR is [here](https://github.com/VikParuchuri/surya/blob/master/surya/recognition/languages.py). If you don't need OCR, marker can work with any language.
## Convert multiple files
```shell
marker /path/to/input/folder
```
- `marker` supports all the same options from `marker_single` above.
- `--workers` is the number of conversion workers to run simultaneously. This is automatically set by default, but you can increase it to increase throughput, at the cost of more CPU/GPU usage. Marker will use 5GB of VRAM per worker at the peak, and 3.5GB average.
## Convert multiple files on multiple GPUs
```shell
NUM_DEVICES=4 NUM_WORKERS=15 marker_chunk_convert ../pdf_in ../md_out
```
- `NUM_DEVICES` is the number of GPUs to use. Should be `2` or greater.
- `NUM_WORKERS` is the number of parallel processes to run on each GPU.
## Use from python
See the `PdfConverter` class at `marker/converters/pdf.py` function for additional arguments that can be passed.
```python
from marker.converters.pdf import PdfConverter
from marker.models import create_model_dict
from marker.output import text_from_rendered
converter = PdfConverter(
artifact_dict=create_model_dict(),
)
rendered = converter("FILEPATH")
text, _, images = text_from_rendered(rendered)
```
`rendered` will be a pydantic basemodel with different properties depending on the output type requested. With markdown output (default), you'll have the properties `markdown`, `metadata`, and `images`. For json output, you'll have `children`, `block_type`, and `metadata`.
### Custom configuration
You can pass configuration using the `ConfigParser`. To see all available options, do `marker_single --help`.
```python
from marker.converters.pdf import PdfConverter
from marker.models import create_model_dict
from marker.config.parser import ConfigParser
config = {
"output_format": "json",
"ADDITIONAL_KEY": "VALUE"
}
config_parser = ConfigParser(config)
converter = PdfConverter(
config=config_parser.generate_config_dict(),
artifact_dict=create_model_dict(),
processor_list=config_parser.get_processors(),
renderer=config_parser.get_renderer(),
llm_service=config_parser.get_llm_service()
)
rendered = converter("FILEPATH")
```
### Extract blocks
Each document consists of one or more pages. Pages contain blocks, which can themselves contain other blocks. It's possible to programmatically manipulate these blocks.
Here's an example of extracting all forms from a document:
```python
from marker.converters.pdf import PdfConverter
from marker.models import create_model_dict
from marker.schema import BlockTypes
converter = PdfConverter(
artifact_dict=create_model_dict(),
)
document = converter.build_document("FILEPATH")
forms = document.contained_blocks((BlockTypes.Form,))
```
Look at the processors for more examples of extracting and manipulating blocks.
## Other converters
You can also use other converters that define different conversion pipelines:
### Extract tables
The `TableConverter` will only convert and extract tables:
```python
from marker.converters.table import TableConverter
from marker.models import create_model_dict
from marker.output import text_from_rendered
converter = TableConverter(
artifact_dict=create_model_dict(),
)
rendered = converter("FILEPATH")
text, _, images = text_from_rendered(rendered)
```
This takes all the same configuration as the PdfConverter. You can specify the configuration `force_layout_block=Table` to avoid layout detection and instead assume every page is a table. Set `output_format=json` to also get cell bounding boxes.
You can also run this via the CLI with
```shell
marker_single FILENAME --use_llm --force_layout_block Table --converter_cls marker.converters.table.TableConverter --output_format json
```
### OCR Only
If you only want to run OCR, you can also do that through the `OCRConverter`. Set `--keep_chars` to keep individual characters and bounding boxes.
```python
from marker.converters.ocr import OCRConverter
from marker.models import create_model_dict
converter = OCRConverter(
artifact_dict=create_model_dict(),
)
rendered = converter("FILEPATH")
```
This takes all the same configuration as the PdfConverter.
You can also run this via the CLI with
```shell
marker_single FILENAME --converter_cls marker.converters.ocr.OCRConverter
```
### Structured Extraction (beta)
You can run structured extraction via the `ExtractionConverter`. This requires an llm service to be setup first (see [here](#llm-services) for details). You'll get a JSON output with the extracted values.
```python
from marker.converters.extraction import ExtractionConverter
from marker.models import create_model_dict
from marker.config.parser import ConfigParser
from pydantic import BaseModel
class Links(BaseModel):
links: list[str]
schema = Links.model_json_schema()
config_parser = ConfigParser({
"page_schema": schema
})
converter = ExtractionConverter(
artifact_dict=create_model_dict(),
config=config_parser.generate_config_dict(),
llm_service=config_parser.get_llm_service(),
)
rendered = converter("FILEPATH")
```
Rendered will have an `original_markdown` field. If you pass this back in next time you run the converter, as the `existing_markdown` config key, you can skip re-parsing the document.
# Output Formats
## Markdown
Markdown output will include:
- image links (images will be saved in the same folder)
- formatted tables
- embedded LaTeX equations (fenced with `$$`)
- Code is fenced with triple backticks
- Superscripts for footnotes
## HTML
HTML output is similar to markdown output:
- Images are included via `img` tags
- equations are fenced with `<math>` tags
- code is in `pre` tags
## JSON
JSON output will be organized in a tree-like structure, with the leaf nodes being blocks. Examples of leaf nodes are a single list item, a paragraph of text, or an image.
The output will be a list, with each list item representing a page. Each page is considered a block in the internal marker schema. There are different types of blocks to represent different elements.
Pages have the keys:
- `id` - unique id for the block.
- `block_type` - the type of block. The possible block types can be seen in `marker/schema/__init__.py`. As of this writing, they are ["Line", "Span", "FigureGroup", "TableGroup", "ListGroup", "PictureGroup", "Page", "Caption", "Code", "Figure", "Footnote", "Form", "Equation", "Handwriting", "TextInlineMath", "ListItem", "PageFooter", "PageHeader", "Picture", "SectionHeader", "Table", "Text", "TableOfContents", "Document"]
- `html` - the HTML for the page. Note that this will have recursive references to children. The `content-ref` tags must be replaced with the child content if you want the full html. You can see an example of this at `marker/output.py:json_to_html`. That function will take in a single block from the json output, and turn it into HTML.
- `polygon` - the 4-corner polygon of the page, in (x1,y1), (x2,y2), (x3, y3), (x4, y4) format. (x1,y1) is the top left, and coordinates go clockwise.
- `children` - the child blocks.
The child blocks have two additional keys:
- `section_hierarchy` - indicates the sections that the block is part of. `1` indicates an h1 tag, `2` an h2, and so on.
- `images` - base64 encoded images. The key will be the block id, and the data will be the encoded image.
Note that child blocks of pages can have their own children as well (a tree structure).
```json
{
"id": "/page/10/Page/366",
"block_type": "Page",
"html": "<content-ref src='/page/10/SectionHeader/0'></content-ref><content-ref src='/page/10/SectionHeader/1'></content-ref><content-ref src='/page/10/Text/2'></content-ref><content-ref src='/page/10/Text/3'></content-ref><content-ref src='/page/10/Figure/4'></content-ref><content-ref src='/page/10/SectionHeader/5'></content-ref><content-ref src='/page/10/SectionHeader/6'></content-ref><content-ref src='/page/10/TextInlineMath/7'></content-ref><content-ref src='/page/10/TextInlineMath/8'></content-ref><content-ref src='/page/10/Table/9'></content-ref><content-ref src='/page/10/SectionHeader/10'></content-ref><content-ref src='/page/10/Text/11'></content-ref>",
"polygon": [[0.0, 0.0], [612.0, 0.0], [612.0, 792.0], [0.0, 792.0]],
"children": [
{
"id": "/page/10/SectionHeader/0",
"block_type": "SectionHeader",
"html": "<h1>Supplementary Material for <i>Subspace Adversarial Training</i> </h1>",
"polygon": [
[217.845703125, 80.630859375], [374.73046875, 80.630859375],
[374.73046875, 107.0],
[217.845703125, 107.0]
],
"children": null,
"section_hierarchy": {
"1": "/page/10/SectionHeader/1"
},
"images": {}
},
...
]
}
```
## Chunks
Chunks format is similar to JSON, but flattens everything into a single list instead of a tree. Only the top level blocks from each page show up. It also has the full HTML of each block inside, so you don't need to crawl the tree to reconstruct it. This enable flexible and easy chunking for RAG.
## Metadata
All output formats will return a metadata dictionary, with the following fields:
```json
{
"table_of_contents": [
{
"title": "Introduction",
"heading_level": 1,
"page_id": 0,
"polygon": [...]
}
], // computed PDF table of contents
"page_stats": [
{
"page_id": 0,
"text_extraction_method": "pdftext",
"block_counts": [("Span", 200), ...]
},
...
]
}
```
# LLM Services
When running with the `--use_llm` flag, you have a choice of services you can use:
- `Gemini` - this will use the Gemini developer API by default. You'll need to pass `--gemini_api_key` to configuration.
- `Google Vertex` - this will use vertex, which can be more reliable. You'll need to pass `--vertex_project_id`. To use it, set `--llm_service=marker.services.vertex.GoogleVertexService`.
- `Ollama` - this will use local models. You can configure `--ollama_base_url` and `--ollama_model`. To use it, set `--llm_service=marker.services.ollama.OllamaService`.
- `Claude` - this will use the anthropic API. You can configure `--claude_api_key`, and `--claude_model_name`. To use it, set `--llm_service=marker.services.claude.ClaudeService`.
- `OpenAI` - this supports any openai-like endpoint. You can configure `--openai_api_key`, `--openai_model`, and `--openai_base_url`. To use it, set `--llm_service=marker.services.openai.OpenAIService`.
- `Azure OpenAI` - this uses the Azure OpenAI service. You can configure `--azure_endpoint`, `--azure_api_key`, and `--deployment_name`. To use it, set `--llm_service=marker.services.azure_openai.AzureOpenAIService`.
These services may have additional optional configuration as well - you can see it by viewing the classes.
# Internals
Marker is easy to extend. The core units of marker are:
- `Providers`, at `marker/providers`. These provide information from a source file, like a PDF.
- `Builders`, at `marker/builders`. These generate the initial document blocks and fill in text, using info from the providers.
- `Processors`, at `marker/processors`. These process specific blocks, for example the table formatter is a processor.
- `Renderers`, at `marker/renderers`. These use the blocks to render output.
- `Schema`, at `marker/schema`. The classes for all the block types.
- `Converters`, at `marker/converters`. They run the whole end to end pipeline.
To customize processing behavior, override the `processors`. To add new output formats, write a new `renderer`. For additional input formats, write a new `provider.`
Processors and renderers can be directly passed into the base `PDFConverter`, so you can specify your own custom processing easily.
## API server
There is a very simple API server you can run like this:
```shell
pip install -U uvicorn fastapi python-multipart
marker_server --port 8001
```
This will start a fastapi server that you can access at `localhost:8001`. You can go to `localhost:8001/docs` to see the endpoint options.
You can send requests like this:
```
import requests
import json
post_data = {
'filepath': 'FILEPATH',
# Add other params here
}
requests.post("http://localhost:8001/marker", data=json.dumps(post_data)).json()
```
Note that this is not a very robust API, and is only intended for small-scale use. If you want to use this server, but want a more robust conversion option, you can use the hosted [Datalab API](https://www.datalab.to/plans).
# Troubleshooting
There are some settings that you may find useful if things aren't working the way you expect:
- If you have issues with accuracy, try setting `--use_llm` to use an LLM to improve quality. You must set `GOOGLE_API_KEY` to a Gemini API key for this to work.
- Make sure to set `force_ocr` if you see garbled text - this will re-OCR the document.
- `TORCH_DEVICE` - set this to force marker to use a given torch device for inference.
- If you're getting out of memory errors, decrease worker count. You can also try splitting up long PDFs into multiple files.
## Debugging
Pass the `debug` option to activate debug mode. This will save images of each page with detected layout and text, as well as output a json file with additional bounding box information.
# Benchmarks
## Overall PDF Conversion
We created a [benchmark set](https://huggingface.co/datasets/datalab-to/marker_benchmark) by extracting single PDF pages from common crawl. We scored based on a heuristic that aligns text with ground truth text segments, and an LLM as a judge scoring method.
| Method | Avg Time | Heuristic Score | LLM Score |
|------------|----------|-----------------|-----------|
| marker | 2.83837 | 95.6709 | 4.23916 |
| llamaparse | 23.348 | 84.2442 | 3.97619 |
| mathpix | 6.36223 | 86.4281 | 4.15626 |
| docling | 3.69949 | 86.7073 | 3.70429 |
Benchmarks were run on an H100 for markjer and docling - llamaparse and mathpix used their cloud services. We can also look at it by document type:
<img src="data/images/per_doc.png" width="1000px"/>
| Document Type | Marker heuristic | Marker LLM | Llamaparse Heuristic | Llamaparse LLM | Mathpix Heuristic | Mathpix LLM | Docling Heuristic | Docling LLM |
|----------------------|------------------|------------|----------------------|----------------|-------------------|-------------|-------------------|-------------|
| Scientific paper | 96.6737 | 4.34899 | 87.1651 | 3.96421 | 91.2267 | 4.46861 | 92.135 | 3.72422 |
| Book page | 97.1846 | 4.16168 | 90.9532 | 4.07186 | 93.8886 | 4.35329 | 90.0556 | 3.64671 |
| Other | 95.1632 | 4.25076 | 81.1385 | 4.01835 | 79.6231 | 4.00306 | 83.8223 | 3.76147 |
| Form | 88.0147 | 3.84663 | 66.3081 | 3.68712 | 64.7512 | 3.33129 | 68.3857 | 3.40491 |
| Presentation | 95.1562 | 4.13669 | 81.2261 | 4 | 83.6737 | 3.95683 | 84.8405 | 3.86331 |
| Financial document | 95.3697 | 4.39106 | 82.5812 | 4.16111 | 81.3115 | 4.05556 | 86.3882 | 3.8 |
| Letter | 98.4021 | 4.5 | 93.4477 | 4.28125 | 96.0383 | 4.45312 | 92.0952 | 4.09375 |
| Engineering document | 93.9244 | 4.04412 | 77.4854 | 3.72059 | 80.3319 | 3.88235 | 79.6807 | 3.42647 |
| Legal document | 96.689 | 4.27759 | 86.9769 | 3.87584 | 91.601 | 4.20805 | 87.8383 | 3.65552 |
| Newspaper page | 98.8733 | 4.25806 | 84.7492 | 3.90323 | 96.9963 | 4.45161 | 92.6496 | 3.51613 |
| Magazine page | 98.2145 | 4.38776 | 87.2902 | 3.97959 | 93.5934 | 4.16327 | 93.0892 | 4.02041 |
## Throughput
We benchmarked throughput using a [single long PDF](https://www.greenteapress.com/thinkpython/thinkpython.pdf).
| Method | Time per page | Time per document | VRAM used |
|---------|---------------|-------------------|---------- |
| marker | 0.18 | 43.42 | 3.17GB |
The projected throughput is 122 pages per second on an H100 - we can run 22 individual processes given the VRAM used.
## Table Conversion
Marker can extract tables from PDFs using `marker.converters.table.TableConverter`. The table extraction performance is measured by comparing the extracted HTML representation of tables against the original HTML representations using the test split of [FinTabNet](https://developer.ibm.com/exchanges/data/all/fintabnet/). The HTML representations are compared using a tree edit distance based metric to judge both structure and content. Marker detects and identifies the structure of all tables in a PDF page and achieves these scores:
| Method | Avg score | Total tables |
|------------------|-----------|--------------|
| marker | 0.816 | 99 |
| marker w/use_llm | 0.907 | 99 |
| gemini | 0.829 | 99 |
The `--use_llm` flag can significantly improve table recognition performance, as you can see.
We filter out tables that we cannot align with the ground truth, since fintabnet and our layout model have slightly different detection methods (this results in some tables being split/merged).
## Running your own benchmarks
You can benchmark the performance of marker on your machine. Install marker manually with:
```shell
git clone https://github.com/VikParuchuri/marker.git
poetry install
```
### Overall PDF Conversion
Download the benchmark data [here](https://drive.google.com/file/d/1ZSeWDo2g1y0BRLT7KnbmytV2bjWARWba/view?usp=sharing) and unzip. Then run the overall benchmark like this:
```shell
python benchmarks/overall.py --methods marker --scores heuristic,llm
```
Options:
- `--use_llm` use an llm to improve the marker results.
- `--max_rows` how many rows to process for the benchmark.
- `--methods` can be `llamaparse`, `mathpix`, `docling`, `marker`. Comma separated.
- `--scores` which scoring functions to use, can be `llm`, `heuristic`. Comma separated.
### Table Conversion
The processed FinTabNet dataset is hosted [here](https://huggingface.co/datasets/datalab-to/fintabnet-test) and is automatically downloaded. Run the benchmark with:
```shell
python benchmarks/table/table.py --max_rows 100
```
Options:
- `--use_llm` uses an llm with marker to improve accuracy.
- `--use_gemini` also benchmarks gemini 2.0 flash.
# How it works
Marker is a pipeline of deep learning models:
- Extract text, OCR if necessary (heuristics, [surya](https://github.com/VikParuchuri/surya))
- Detect page layout and find reading order ([surya](https://github.com/VikParuchuri/surya))
- Clean and format each block (heuristics, [texify](https://github.com/VikParuchuri/texify), [surya](https://github.com/VikParuchuri/surya))
- Optionally use an LLM to improve quality
- Combine blocks and postprocess complete text
It only uses models where necessary, which improves speed and accuracy.
# Limitations
PDF is a tricky format, so marker will not always work perfectly. Here are some known limitations that are on the roadmap to address:
- Very complex layouts, with nested tables and forms, may not work
- Forms may not be rendered well
Note: Passing the `--use_llm` and `--force_ocr` flags will mostly solve these issues.
# Usage and Deployment Examples
You can always run `marker` locally, but if you wanted to expose it as an API, we have a few options:
- Our platform API which is powered by `marker` and `surya` and is easy to test out - it's free to sign up, and we'll include credits, [try it out here](https://datalab.to)
- Our painless on-prem solution for commercial use, which you can [read about here](https://www.datalab.to/blog/self-serve-on-prem-licensing) and gives you privacy guarantees with high throughput inference optimizations.
- [Deployment example with Modal](./examples/README_MODAL.md) that shows you how to deploy and access `marker` through a web endpoint using [`Modal`](https://modal.com). Modal is an AI compute platform that enables developers to deploy and scale models on GPUs in minutes.
+120 -4
View File
@@ -11,13 +11,115 @@ graph TD
User["Neurosurgeon (Browser)"] User["Neurosurgeon (Browser)"]
FE["Frontend\nVue.js 3 / Vite\n:5173"] FE["Frontend\nVue.js 3 / Vite\n:5173"]
BE["Backend\nSpring Boot 4 / Spring AI\n:8080"] BE["Backend\nSpring Boot 4 / Spring AI\n:8080"]
DB["PostgreSQL + pgvector\n(provided)"] DB["PostgreSQL + pgvector\n(source of truth)"]
LLM["LLM Provider\n(OpenAI / configurable)"] FS["File Store\nuploads/ (local disk)\nExtracted figure PNGs"]
LLM["LLM Provider\n(OpenAI)\nEmbeddings + Chat + Vision"]
User -->|HTTP| FE User -->|HTTP| FE
FE -->|REST /api/v1/...| BE FE -->|REST /api/v1/...| BE
BE -->|JDBC / pgvector| DB BE -->|"JDBC — books, chapters,\nsections, figures, refs"| DB
BE -->|Embedding + Chat API| LLM BE -->|"pgvector — text chunks\n+ figure caption vectors"| DB
BE -->|"PNG read/write\n(figure extraction)"| FS
FE -->|"GET /api/v1/figures/**\n(static file serving)"| BE
BE -->|"Embedding + Chat\n+ Vision (image description)"| LLM
subgraph "Embedding Pipeline (per PDF upload)"
EP1["Parse pages → SectionEntity"]
EP2["Extract images → FigureEntity"]
EP3["Vision describe → embed caption"]
EP4["Chunk text → embed chunks"]
EP5["Link chunks ↔ figures"]
EP1 --> EP2
EP1 --> EP4
EP2 --> EP3
EP4 --> EP5
EP3 --> EP5
end
subgraph "Retrieval Pipeline (per chat query)"
RP1["Text chunk search (topK=5)"]
RP2["Figure caption search (topK=3)"]
RP3["Expand chunks → full section text"]
RP4["Fetch linked figures (chunk_figure_ref)"]
RP5["Merge + deduplicate figures"]
RP6["Build LLM prompt + call"]
RP1 --> RP3
RP1 --> RP4
RP2 --> RP5
RP4 --> RP5
RP3 --> RP6
RP5 --> RP6
end
```
## Marker API Response Structure
The PDF parsing pipeline calls a local [Marker](https://github.com/VikParuchuri/marker) server (`POST /marker/upload`).
### Top-level envelope
```json
{
"format": "json",
"output": "<JSON-encoded string>"
}
```
`output` is a **JSON-encoded string** (not a nested object) and must be parsed a second time to get the document tree.
### Parsed `output` shape
```
{
"children": [ <Page block>, ... ]
}
```
### Block types
Every block shares these fields:
| Field | Type | Notes |
|------------------|-------------------|--------------------------------------------|
| `id` | string | e.g. `/page/0/Picture/2` |
| `block_type` | string | see table below |
| `html` | string | rendered HTML; may contain `<content-ref>` |
| `bbox` | `[x0,y0,x1,y1]` | bounding box in page coordinates |
| `children` | array or null | nested blocks |
| `images` | object or null | base64 PNG map (leaf image blocks only) |
| `section_hierarchy` | object | heading ancestry |
#### Known `block_type` values
| block_type | Category | Notes |
|------------------|----------|-------------------------------------------------------|
| `Page` | structure | Top-level; direct children are the page content |
| `SectionHeader` | text | Section / chapter heading |
| `Text` | text | |
| `TextInlineMath` | text | |
| `ListItem` | text | |
| `Table` | text | |
| `Code` | text | |
| `Equation` | text | |
| `Footnote` | text | |
| `Caption` | text | Usually a child of a `*Group` block |
| `PageHeader` | text | |
| `PageFooter` | text | |
| `Handwriting` | text | |
| `Picture` | image | Leaf block; `images` map holds base64 PNG keyed by ID |
| `Figure` | image | Leaf block; same as `Picture` |
| `PictureGroup` | container | Wraps one `Picture` + one `Caption` child |
| `FigureGroup` | container | Wraps one `Figure` + one `Caption` child |
### Image extraction
Images are only present on **leaf** image blocks (`Picture`, `Figure`).
Group blocks (`PictureGroup`, `FigureGroup`) have `images: null` — the base64 PNG lives on the child leaf block.
```
PictureGroup
├── Picture ← images: { "/page/0/Picture/2": "<base64 PNG>" }
└── Caption ← html: "<p>Figure 1 — ...</p>"
``` ```
## Stack ## Stack
@@ -49,6 +151,8 @@ npm run dev
### Environment Variables ### Environment Variables
#### Backend
| Variable | Required | Description | | Variable | Required | Description |
|----------|----------|-------------| |----------|----------|-------------|
| `OPENAI_API_KEY` | Yes | OpenAI API key for embeddings and chat | | `OPENAI_API_KEY` | Yes | OpenAI API key for embeddings and chat |
@@ -56,3 +160,15 @@ npm run dev
| `DB_URL` | Yes | JDBC URL, e.g. `jdbc:postgresql://localhost:5432/aiteacher` | | `DB_URL` | Yes | JDBC URL, e.g. `jdbc:postgresql://localhost:5432/aiteacher` |
| `DB_USERNAME` | Yes | Database username | | `DB_USERNAME` | Yes | Database username |
| `DB_PASSWORD` | Yes | Database password | | `DB_PASSWORD` | Yes | Database password |
| `FIGURE_STORAGE_PATH` | No | Base path for uploaded PDFs and extracted figures (default: `./uploads`) |
| `UPLOAD_ENABLED` | No | Set to `false` to disable the book upload endpoint (default: `true`) |
| `DELETE_ENABLED` | No | Set to `false` to disable the book delete endpoint (default: `true`) |
#### Frontend
| Variable | Required | Description |
|----------|----------|-------------|
| `VITE_API_URL` | No | Backend API base URL (default: `/api/v1`) |
| `VITE_APP_PASSWORD` | Yes | Shared password for HTTP Basic auth (must match `APP_PASSWORD`) |
| `VITE_UPLOAD_ENABLED` | No | Set to `false` to hide the upload UI (default: `true`) |
| `VITE_DELETE_ENABLED` | No | Set to `false` to hide the delete button (default: `true`) |
+21 -1
View File
@@ -32,6 +32,13 @@
<type>pom</type> <type>pom</type>
<scope>import</scope> <scope>import</scope>
</dependency> </dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>bom</artifactId>
<version>2.30.14</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies> </dependencies>
</dependencyManagement> </dependencyManagement>
@@ -95,12 +102,25 @@
<artifactId>spring-ai-advisors-vector-store</artifactId> <artifactId>spring-ai-advisors-vector-store</artifactId>
</dependency> </dependency>
<!-- Spring AI — PDF document reader --> <!-- Spring AI — PDF document reader (includes PDFBox transitively) -->
<dependency> <dependency>
<groupId>org.springframework.ai</groupId> <groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-pdf-document-reader</artifactId> <artifactId>spring-ai-pdf-document-reader</artifactId>
</dependency> </dependency>
<!-- PDFBox — page rendering and cropping for figure extraction -->
<dependency>
<groupId>org.apache.pdfbox</groupId>
<artifactId>pdfbox</artifactId>
<version>3.0.3</version>
</dependency>
<!-- AWS SDK v2 — S3 figure storage -->
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3</artifactId>
</dependency>
<!-- Jackson (JSON) --> <!-- Jackson (JSON) -->
<dependency> <dependency>
<groupId>com.fasterxml.jackson.core</groupId> <groupId>com.fasterxml.jackson.core</groupId>
@@ -1,6 +1,11 @@
package com.aiteacher.book; package com.aiteacher.book;
import com.aiteacher.document.FigureEntity;
import com.aiteacher.document.FigureRepository;
import com.aiteacher.document.MarkdownStorageService;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.http.HttpStatus; import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity; import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*; import org.springframework.web.bind.annotation.*;
import org.springframework.web.multipart.MultipartFile; import org.springframework.web.multipart.MultipartFile;
@@ -15,13 +20,25 @@ import java.util.UUID;
public class BookController { public class BookController {
private final BookService bookService; private final BookService bookService;
private final FigureRepository figureRepository;
private final MarkdownStorageService markdownStorageService;
public BookController(BookService bookService) { @Value("${app.features.upload-enabled:true}")
private boolean uploadEnabled;
@Value("${app.features.delete-enabled:true}")
private boolean deleteEnabled;
public BookController(BookService bookService, FigureRepository figureRepository,
MarkdownStorageService markdownStorageService) {
this.bookService = bookService; this.bookService = bookService;
this.figureRepository = figureRepository;
this.markdownStorageService = markdownStorageService;
} }
@PostMapping(consumes = "multipart/form-data") @PostMapping(consumes = "multipart/form-data")
public ResponseEntity<?> upload(@RequestParam("file") MultipartFile file) throws IOException { public ResponseEntity<?> upload(@RequestParam("file") MultipartFile file) throws IOException {
if (!uploadEnabled) return ResponseEntity.status(HttpStatus.METHOD_NOT_ALLOWED).build();
Book book = bookService.upload(file); Book book = bookService.upload(file);
return ResponseEntity.status(HttpStatus.ACCEPTED).body(toSummaryResponse(book)); return ResponseEntity.status(HttpStatus.ACCEPTED).body(toSummaryResponse(book));
} }
@@ -42,10 +59,52 @@ public class BookController {
@DeleteMapping("/{id}") @DeleteMapping("/{id}")
public ResponseEntity<Void> delete(@PathVariable UUID id) { public ResponseEntity<Void> delete(@PathVariable UUID id) {
if (!deleteEnabled) return ResponseEntity.status(HttpStatus.METHOD_NOT_ALLOWED).build();
bookService.delete(id); bookService.delete(id);
return ResponseEntity.noContent().build(); return ResponseEntity.noContent().build();
} }
@PostMapping("/{id}/reembed")
public ResponseEntity<Map<String, Object>> reembed(@PathVariable UUID id) {
Book book = bookService.reembed(id);
return ResponseEntity.accepted().body(Map.of(
"bookId", book.getId(),
"status", BookStatus.PROCESSING.name()
));
}
@GetMapping(value = "/{id}/pages/{pageNumber}/html", produces = MediaType.TEXT_HTML_VALUE)
public ResponseEntity<String> getPageHtml(@PathVariable UUID id,
@PathVariable int pageNumber) {
bookService.getById(id); // 404 if not found
try {
return ResponseEntity.ok(markdownStorageService.getText(id, pageNumber));
} catch (Exception e) {
return ResponseEntity.notFound().build();
}
}
@GetMapping("/{id}/figures")
public ResponseEntity<List<FigureResponse>> figures(@PathVariable UUID id) {
bookService.getById(id); // 404 if not found
List<FigureResponse> responses = figureRepository.findAllByBookId(id)
.stream()
.map(f -> toFigureResponse(id, f))
.toList();
return ResponseEntity.ok(responses);
}
private FigureResponse toFigureResponse(UUID bookId, FigureEntity f) {
String filename = f.getImagePath().substring(f.getImagePath().lastIndexOf('/') + 1);
String imageUrl = "/api/v1/figures/" + bookId + "/" + filename;
return new FigureResponse(
f.getId(), f.getLabel(), f.getCaption(),
f.getFigureType().name(), f.getPage(), imageUrl,
f.getSectionId(),
null // section title not eagerly loaded here
);
}
private Map<String, Object> toSummaryResponse(Book book) { private Map<String, Object> toSummaryResponse(Book book) {
return Map.of( return Map.of(
"id", book.getId(), "id", book.getId(),
@@ -1,41 +1,82 @@
package com.aiteacher.book; package com.aiteacher.book;
import com.aiteacher.document.*;
import com.aiteacher.figure.FigureStorageService;
import org.slf4j.Logger; import org.slf4j.Logger;
import org.slf4j.LoggerFactory; import org.slf4j.LoggerFactory;
import org.springframework.ai.document.Document; import org.springframework.ai.document.Document;
import org.springframework.ai.reader.pdf.PagePdfDocumentReader;
import org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig;
import org.springframework.ai.vectorstore.VectorStore; import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.ai.vectorstore.filter.FilterExpressionBuilder; import org.springframework.ai.vectorstore.filter.FilterExpressionBuilder;
import org.springframework.core.io.FileSystemResource; import org.springframework.beans.factory.annotation.Value;
import org.springframework.scheduling.annotation.Async; import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.nio.file.Path; import java.nio.file.Path;
import java.util.List; import java.time.Instant;
import java.util.UUID; import java.util.*;
import java.util.regex.Pattern;
@Service @Service
public class BookEmbeddingService { public class BookEmbeddingService {
private static final Logger log = LoggerFactory.getLogger(BookEmbeddingService.class); private static final Logger log = LoggerFactory.getLogger(BookEmbeddingService.class);
// Pattern to detect diagram/figure captions
private static final Pattern CAPTION_PATTERN =
Pattern.compile("^(Figure|Fig\\.|Table|Diagram)\\s+[\\d.]+", Pattern.CASE_INSENSITIVE);
private final VectorStore vectorStore; private final VectorStore vectorStore;
private final BookRepository bookRepository; private final BookRepository bookRepository;
private final MarkerPageParser markerPageParser;
private final FigureExtractionService figureExtractionService;
private final VisionDescriptionService visionDescriptionService;
private final TextChunkingService textChunkingService;
private final ChunkFigureRefService chunkFigureRefService;
private final SectionRepository sectionRepository;
private final ChapterRepository chapterRepository;
private final FigureRepository figureRepository;
private final ChunkFigureRefRepository chunkFigureRefRepository;
private final FigureStorageService figureStorageService;
private final MarkdownStorageService markdownStorageService;
public BookEmbeddingService(VectorStore vectorStore, BookRepository bookRepository) { @Value("${app.embedding.batch-size:50}")
private int embeddingBatchSize;
@Value("${app.embedding.batch-delay-ms:1000}")
private long embeddingBatchDelayMs;
@Value("${app.embedding.skip-embedding:false}")
private boolean skipEmbedding;
public BookEmbeddingService(
VectorStore vectorStore,
BookRepository bookRepository,
MarkerPageParser markerPageParser,
FigureExtractionService figureExtractionService,
VisionDescriptionService visionDescriptionService,
TextChunkingService textChunkingService,
ChunkFigureRefService chunkFigureRefService,
SectionRepository sectionRepository,
ChapterRepository chapterRepository,
FigureRepository figureRepository,
ChunkFigureRefRepository chunkFigureRefRepository,
FigureStorageService figureStorageService,
MarkdownStorageService markdownStorageService) {
this.vectorStore = vectorStore; this.vectorStore = vectorStore;
this.bookRepository = bookRepository; this.bookRepository = bookRepository;
this.markerPageParser = markerPageParser;
this.figureExtractionService = figureExtractionService;
this.visionDescriptionService = visionDescriptionService;
this.textChunkingService = textChunkingService;
this.chunkFigureRefService = chunkFigureRefService;
this.sectionRepository = sectionRepository;
this.chapterRepository = chapterRepository;
this.figureRepository = figureRepository;
this.chunkFigureRefRepository = chunkFigureRefRepository;
this.figureStorageService = figureStorageService;
this.markdownStorageService = markdownStorageService;
} }
@Async @Async
public void embedBook(UUID bookId, String bookTitle, Path pdfPath) { public void embedBook(UUID bookId, String bookTitle, Path pdfPath) {
log.info("Starting embedding for book {} ({})", bookId, bookTitle); log.info("Starting Marker-powered embedding for book {} ({})", bookId, bookTitle);
Book book = bookRepository.findById(bookId).orElse(null); Book book = bookRepository.findById(bookId).orElse(null);
if (book == null) { if (book == null) {
@@ -47,29 +88,87 @@ public class BookEmbeddingService {
book.setStatus(BookStatus.PROCESSING); book.setStatus(BookStatus.PROCESSING);
bookRepository.save(book); bookRepository.save(book);
PagePdfDocumentReader reader = new PagePdfDocumentReader( String chapterId = bookId + "-ch1";
new FileSystemResource(pdfPath.toFile()), ChapterEntity chapter = new ChapterEntity(chapterId, bookId, 1, bookTitle, 1);
PdfDocumentReaderConfig.builder() chapterRepository.save(chapter);
.withPagesPerDocument(1)
.build()
);
List<Document> pages = reader.get(); // Step 1: Parse with Marker — JSON (structured) + Markdown (per-page) in parallel
int pageCount = pages.size(); ParsedBook parsed = markerPageParser.parse(pdfPath);
// Enrich metadata and tag diagram captions List<PageResult> pageResults = parsed.pages();
List<Document> enriched = pages.stream()
.map(doc -> enrichDocument(doc, bookId.toString(), bookTitle))
.toList();
vectorStore.add(enriched); // Step 2: Build SectionEntity per page and persist
List<SectionEntity> sections = buildAndSaveSections(bookId, bookTitle, chapterId, pageResults);
// Step 3: Chunk and embed text
List<Document> allChunks = new ArrayList<>();
for (SectionEntity section : sections) {
allChunks.addAll(textChunkingService.chunk(section, bookTitle));
}
if (skipEmbedding) {
log.info("skip-embedding=true — skipping text embedding for book {}", bookId);
} else {
embedInBatches(allChunks, bookId);
log.info("Embedded {} text chunks for book {}", allChunks.size(), bookId);
}
// Step 4: Decode pre-cropped figures from Marker output
FigureExtractionService.ExtractionResult extraction =
figureExtractionService.extract(bookId, chapterId, pageResults);
List<FigureEntity> figures = extraction.figures();
// Step 4b: Save per-page HTML to S3, replacing Marker image src with API URLs
parsed.htmlByPage().forEach((pageNumber, html) -> {
String resolved = resolveImageSrcs(html, bookId, extraction.blockIdToFigureId());
markdownStorageService.save(bookId, pageNumber, resolved);
});
log.info("Saved {} HTML pages to S3 for book {}", parsed.htmlByPage().size(), bookId);
// Step 5: Vision analysis (description + visible text) → embed figure chunks
for (FigureEntity figure : figures) {
byte[] imageBytes = figureStorageService.getBytes(figure.getImagePath());
VisionDescriptionService.ImageAnalysis analysis =
visionDescriptionService.analyze(imageBytes, figure.getCaption());
if (figure.getCaption() == null || figure.getCaption().isBlank()) {
figure.setCaption(analysis.description());
figureRepository.save(figure);
}
// Embedding content: description + caption + visible image text
String embeddingContent = analysis.description()
+ (figure.getCaption() != null ? "\n" + figure.getCaption() : "")
+ (analysis.imageText().isEmpty() ? "" : "\n" + analysis.imageText());
String embeddingId = UUID.randomUUID().toString();
if (!skipEmbedding) {
Document figureDoc = new Document(embeddingId, embeddingContent,
buildFigureMetadata(figure, bookTitle, embeddingId, analysis.imageText()));
vectorStore.add(List.of(figureDoc));
figure.setCaptionEmbeddingId(UUID.fromString(embeddingId));
}
figureRepository.save(figure);
}
log.info("Embedded {} figure chunks for book {}", figures.size(), bookId);
// Step 6: Link text chunks to figures via in-text references
for (SectionEntity section : sections) {
List<Document> sectionChunks = allChunks.stream()
.filter(d -> section.getId().equals(d.getMetadata().get("section_id")))
.toList();
List<FigureEntity> sectionFigures = figures.stream()
.filter(f -> section.getId().equals(f.getSectionId()))
.toList();
chunkFigureRefService.linkChunksToFigures(sectionChunks, sectionFigures, section.getPageStart());
}
book.setStatus(BookStatus.READY); book.setStatus(BookStatus.READY);
book.setPageCount(pageCount); book.setPageCount(sections.size());
book.setProcessedAt(java.time.Instant.now()); book.setProcessedAt(Instant.now());
bookRepository.save(book); bookRepository.save(book);
log.info("Finished embedding book {} — {} pages", bookId, pageCount); log.info("Finished embedding book {} — {} pages, {} figures",
bookId, sections.size(), figures.size());
} catch (Exception ex) { } catch (Exception ex) {
log.error("Failed to embed book {}", bookId, ex); log.error("Failed to embed book {}", bookId, ex);
@@ -79,40 +178,101 @@ public class BookEmbeddingService {
} }
} }
private Document enrichDocument(Document doc, String bookId, String bookTitle) { @Transactional
String content = doc.getText();
String chunkType = detectChunkType(content);
doc.getMetadata().put("book_id", bookId);
doc.getMetadata().put("book_title", bookTitle);
doc.getMetadata().put("chunk_type", chunkType);
return doc;
}
private String detectChunkType(String content) {
if (content != null) {
for (String line : content.split("\\r?\\n")) {
if (CAPTION_PATTERN.matcher(line.trim()).find()) {
return "diagram";
}
}
}
return "text";
}
public void deleteBookChunks(UUID bookId) { public void deleteBookChunks(UUID bookId) {
log.info("Deleting vector chunks for book {}", bookId); log.info("Deleting all data for book {}", bookId);
try { try {
List<String> figureIds = figureRepository.findAllByBookId(bookId)
.stream().map(FigureEntity::getId).toList();
if (!figureIds.isEmpty()) {
chunkFigureRefRepository.deleteByFigureIdIn(figureIds);
}
figureRepository.deleteAllByBookId(bookId);
figureStorageService.deleteAll(bookId);
markdownStorageService.deleteAll(bookId);
sectionRepository.deleteAllByBookId(bookId);
chapterRepository.deleteAllByBookId(bookId);
FilterExpressionBuilder b = new FilterExpressionBuilder(); FilterExpressionBuilder b = new FilterExpressionBuilder();
vectorStore.delete(b.eq("book_id", bookId.toString()).build()); vectorStore.delete(b.eq("book_id", bookId.toString()).build());
} catch (Exception ex) { } catch (Exception ex) {
log.warn("Could not delete vector chunks for book {}: {}", bookId, ex.getMessage()); log.warn("Error during cleanup for book {}: {}", bookId, ex.getMessage());
} }
} }
private String truncate(String message, int maxLength) { // --- Private helpers ---
if (message == null) return null;
return message.length() <= maxLength ? message : message.substring(0, maxLength); private List<SectionEntity> buildAndSaveSections(UUID bookId, String bookTitle,
String chapterId,
List<PageResult> pageResults) {
List<SectionEntity> sections = new ArrayList<>();
for (PageResult page : pageResults) {
if (page.orderedText().isBlank()) continue;
String sectionId = bookId + "-p" + page.pageNumber();
String title = page.headingTitle() != null ? page.headingTitle() : "Page " + page.pageNumber();
SectionEntity section = new SectionEntity(
sectionId, chapterId, bookId,
String.valueOf(page.pageNumber()),
title,
page.pageNumber(), page.pageNumber(),
page.orderedText());
sections.add(sectionRepository.save(section));
}
return sections;
}
private void embedInBatches(List<Document> docs, UUID bookId) {
int total = docs.size();
for (int i = 0; i < total; i += embeddingBatchSize) {
List<Document> batch = docs.subList(i, Math.min(i + embeddingBatchSize, total));
vectorStore.add(batch);
log.debug("Embedded batch {}/{} for book {}",
i / embeddingBatchSize + 1, (total - 1) / embeddingBatchSize + 1, bookId);
if (i + embeddingBatchSize < total) {
try { Thread.sleep(embeddingBatchDelayMs); }
catch (InterruptedException e) { Thread.currentThread().interrupt(); }
}
}
}
private Map<String, Object> buildFigureMetadata(FigureEntity figure, String bookTitle,
String embeddingId, String imageText) {
Map<String, Object> m = new HashMap<>();
m.put("type", "FIGURE");
m.put("book_id", figure.getBookId().toString());
m.put("book_title", bookTitle);
m.put("chapter_id", figure.getChapterId() != null ? figure.getChapterId() : "");
m.put("section_id", figure.getSectionId() != null ? figure.getSectionId() : "");
m.put("figure_id", figure.getId());
m.put("figure_type", figure.getFigureType().name());
m.put("image_path", figure.getImagePath());
m.put("label", figure.getLabel() != null ? figure.getLabel() : "");
m.put("page", figure.getPage());
m.put("embedding_id", embeddingId);
m.put("image_text", imageText); // verbatim text visible inside the image
return m;
}
/**
* Replaces Marker's {@code src='{blockId}'} image attributes with resolved API URLs.
* Block IDs look like {@code /page/0/Figure/2}.
*/
private String resolveImageSrcs(String html, UUID bookId, Map<String, String> blockIdToFigureId) {
for (Map.Entry<String, String> entry : blockIdToFigureId.entrySet()) {
String blockId = entry.getKey();
String figureId = entry.getValue();
String apiUrl = "/api/v1/figures/" + bookId + "/" + figureId + ".png";
// Marker emits both single and double-quoted src attributes
html = html.replace("src='" + blockId + "'", "src='" + apiUrl + "'");
html = html.replace("src=\"" + blockId + "\"", "src=\"" + apiUrl + "\"");
}
return html;
}
private String truncate(String msg, int max) {
if (msg == null) return null;
return msg.length() <= max ? msg : msg.substring(0, max);
} }
} }
@@ -1,11 +1,13 @@
package com.aiteacher.book; package com.aiteacher.book;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import org.springframework.web.multipart.MultipartFile; import org.springframework.web.multipart.MultipartFile;
import java.io.IOException; import java.io.IOException;
import java.nio.file.Files; import java.nio.file.Files;
import java.nio.file.Path; import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.List; import java.util.List;
import java.util.NoSuchElementException; import java.util.NoSuchElementException;
import java.util.UUID; import java.util.UUID;
@@ -15,10 +17,15 @@ public class BookService {
private final BookRepository bookRepository; private final BookRepository bookRepository;
private final BookEmbeddingService bookEmbeddingService; private final BookEmbeddingService bookEmbeddingService;
private final Path bookStoragePath;
public BookService(BookRepository bookRepository, BookEmbeddingService bookEmbeddingService) { public BookService(
BookRepository bookRepository,
BookEmbeddingService bookEmbeddingService,
@Value("${app.figure-storage.base-path:./uploads}") String basePath) {
this.bookRepository = bookRepository; this.bookRepository = bookRepository;
this.bookEmbeddingService = bookEmbeddingService; this.bookEmbeddingService = bookEmbeddingService;
this.bookStoragePath = Paths.get(basePath).toAbsolutePath().normalize().resolve("books");
} }
public Book upload(MultipartFile file) throws IOException { public Book upload(MultipartFile file) throws IOException {
@@ -28,20 +35,35 @@ public class BookService {
} }
String title = deriveTitle(originalFilename); String title = deriveTitle(originalFilename);
Book book = new Book(title, originalFilename, file.getSize()); Book book = new Book(title, originalFilename, file.getSize());
book = bookRepository.save(book); book = bookRepository.save(book);
// Write to a temp file so the async task can read it // Persist PDF in a stable location for potential re-embedding
Path tempFile = Files.createTempFile("aiteacher-", "-" + book.getId() + ".pdf"); Files.createDirectories(bookStoragePath);
file.transferTo(tempFile.toFile()); Path pdfPath = bookStoragePath.resolve(book.getId() + ".pdf");
file.transferTo(pdfPath.toFile());
UUID bookId = book.getId(); UUID bookId = book.getId();
Path pdfPath = tempFile; bookEmbeddingService.embedBook(bookId, title, pdfPath);
String bookTitle = title; return book;
}
bookEmbeddingService.embedBook(bookId, bookTitle, pdfPath); public Book reembed(UUID id) {
Book book = bookRepository.findById(id)
.orElseThrow(() -> new NoSuchElementException("Book not found."));
if (book.getStatus() == BookStatus.PROCESSING) {
throw new IllegalStateException("Book is already being processed.");
}
Path pdfPath = bookStoragePath.resolve(id + ".pdf");
if (!Files.exists(pdfPath)) {
throw new IllegalStateException(
"Original PDF not found. Please re-upload the book before re-embedding.");
}
bookEmbeddingService.deleteBookChunks(id);
bookEmbeddingService.embedBook(id, book.getTitle(), pdfPath);
return book; return book;
} }
@@ -63,14 +85,21 @@ public class BookService {
} }
bookEmbeddingService.deleteBookChunks(id); bookEmbeddingService.deleteBookChunks(id);
// Delete the stored PDF
Path pdfPath = bookStoragePath.resolve(id + ".pdf");
try {
Files.deleteIfExists(pdfPath);
} catch (IOException ex) {
// Non-fatal — log only
}
bookRepository.deleteById(id); bookRepository.deleteById(id);
} }
private String deriveTitle(String filename) { private String deriveTitle(String filename) {
// Strip .pdf extension and replace separators with spaces
String name = filename.replaceAll("(?i)\\.pdf$", ""); String name = filename.replaceAll("(?i)\\.pdf$", "");
name = name.replaceAll("[-_]", " "); name = name.replaceAll("[-_]", " ");
// Capitalise first letter
if (!name.isEmpty()) { if (!name.isEmpty()) {
name = Character.toUpperCase(name.charAt(0)) + name.substring(1); name = Character.toUpperCase(name.charAt(0)) + name.substring(1);
} }
@@ -0,0 +1,12 @@
package com.aiteacher.book;
public record FigureResponse(
String figureId,
String label,
String caption,
String figureType,
int page,
String imageUrl,
String sectionId,
String sectionTitle
) {}
@@ -3,22 +3,16 @@ package com.aiteacher.chat;
import com.aiteacher.book.BookRepository; import com.aiteacher.book.BookRepository;
import com.aiteacher.book.BookStatus; import com.aiteacher.book.BookStatus;
import com.aiteacher.book.NoKnowledgeSourceException; import com.aiteacher.book.NoKnowledgeSourceException;
import com.aiteacher.document.FigureEntity;
import com.aiteacher.document.SectionEntity;
import com.aiteacher.retrieval.NeurosurgeryRetriever;
import com.aiteacher.retrieval.RetrievalResult;
import org.slf4j.Logger; import org.slf4j.Logger;
import org.slf4j.LoggerFactory; import org.slf4j.LoggerFactory;
import org.springframework.ai.chat.client.ChatClient; import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.document.Document;
import org.springframework.ai.vectorstore.SearchRequest;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import java.util.ArrayList; import java.util.*;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.NoSuchElementException;
import java.util.UUID;
@Service @Service
public class ChatService { public class ChatService {
@@ -35,26 +29,28 @@ public class ChatService {
- Build answers from what is present: procedures, conditions, techniques, and descriptions all contribute; combine them into a rich, structured response - Build answers from what is present: procedures, conditions, techniques, and descriptions all contribute; combine them into a rich, structured response
- Use clear structure: headings, bullet points, or numbered steps where appropriate to maximize clarity - Use clear structure: headings, bullet points, or numbered steps where appropriate to maximize clarity
- Only say you cannot answer if the context is entirely unrelated to the question - Only say you cannot answer if the context is entirely unrelated to the question
- Cite sources for each major point (book title and page number from the context metadata) - Cite sources for each major point (book title and page number from the context)
- When referencing diagrams or figures, cite them as [Fig. X, p.N]
- Maintain continuity with the conversation history - Maintain continuity with the conversation history
- Never fabricate clinical information not present in the context - Never fabricate clinical information not present in the context
"""; """;
private final ChatClient chatClient; private final ChatClient chatClient;
private final VectorStore vectorStore;
private final BookRepository bookRepository; private final BookRepository bookRepository;
private final ChatSessionRepository sessionRepository; private final ChatSessionRepository sessionRepository;
private final MessageRepository messageRepository; private final MessageRepository messageRepository;
private final NeurosurgeryRetriever retriever;
public ChatService(ChatClient chatClient, VectorStore vectorStore, public ChatService(ChatClient chatClient,
BookRepository bookRepository, BookRepository bookRepository,
ChatSessionRepository sessionRepository, ChatSessionRepository sessionRepository,
MessageRepository messageRepository) { MessageRepository messageRepository,
NeurosurgeryRetriever retriever) {
this.chatClient = chatClient; this.chatClient = chatClient;
this.vectorStore = vectorStore;
this.bookRepository = bookRepository; this.bookRepository = bookRepository;
this.sessionRepository = sessionRepository; this.sessionRepository = sessionRepository;
this.messageRepository = messageRepository; this.messageRepository = messageRepository;
this.retriever = retriever;
} }
public ChatSession createSession(String topicId) { public ChatSession createSession(String topicId) {
@@ -73,7 +69,11 @@ public class ChatService {
ChatSession session = sessionRepository.findById(sessionId) ChatSession session = sessionRepository.findById(sessionId)
.orElseThrow(() -> new NoSuchElementException("Session not found.")); .orElseThrow(() -> new NoSuchElementException("Session not found."));
if (!bookRepository.existsByStatus(BookStatus.READY)) { List<com.aiteacher.book.Book> readyBooks = bookRepository.findAll().stream()
.filter(b -> b.getStatus() == BookStatus.READY)
.toList();
if (readyBooks.isEmpty()) {
throw new NoKnowledgeSourceException("No books are available as knowledge sources."); throw new NoKnowledgeSourceException("No books are available as knowledge sources.");
} }
@@ -81,27 +81,31 @@ public class ChatService {
Message userMessage = new Message(sessionId, MessageRole.USER, userContent); Message userMessage = new Message(sessionId, MessageRole.USER, userContent);
messageRepository.save(userMessage); messageRepository.save(userMessage);
// Build conversation history for context // Build full question with conversation history
List<Message> history = messageRepository.findBySessionIdOrderByCreatedAtAsc(sessionId); List<Message> history = messageRepository.findBySessionIdOrderByCreatedAtAsc(sessionId);
// Build the prompt with full conversation history as context
String fullQuestion = buildQuestionWithHistory(history, userContent, session.getTopicId()); String fullQuestion = buildQuestionWithHistory(history, userContent, session.getTopicId());
var qaAdvisor = QuestionAnswerAdvisor.builder(vectorStore) // Retrieve context from all ready books (aggregate across books)
.searchRequest(SearchRequest.builder().similarityThreshold(0.5d).topK(6).build()) List<SectionEntity> allSections = new ArrayList<>();
.build(); List<FigureEntity> allFigures = new ArrayList<>();
for (com.aiteacher.book.Book book : readyBooks) {
RetrievalResult result = retriever.retrieve(fullQuestion, book.getId());
allSections.addAll(result.parentSections());
allFigures.addAll(result.figures());
}
ChatResponse response = chatClient.prompt() // Build LLM prompt with section full texts and figure references
.advisors(qaAdvisor) String contextPrompt = buildContextPrompt(fullQuestion, allSections, allFigures);
String assistantContent = chatClient.prompt()
.system(SYSTEM_PROMPT) .system(SYSTEM_PROMPT)
.user(fullQuestion) .user(contextPrompt)
.call() .call()
.chatResponse(); .content();
String assistantContent = response.getResult().getOutput().getText(); // Build sources list with TEXT and FIGURE entries
List<Map<String, Object>> sources = extractSources(response); List<Map<String, Object>> sources = buildSources(allSections, allFigures);
// Persist assistant message
Message assistantMessage = new Message(sessionId, MessageRole.ASSISTANT, assistantContent); Message assistantMessage = new Message(sessionId, MessageRole.ASSISTANT, assistantContent);
assistantMessage.setSources(sources); assistantMessage.setSources(sources);
return messageRepository.save(assistantMessage); return messageRepository.save(assistantMessage);
@@ -118,24 +122,95 @@ public class ChatService {
sessionRepository.deleteById(sessionId); sessionRepository.deleteById(sessionId);
} }
// -------------------------------------------------------------------------
// Private helpers
// -------------------------------------------------------------------------
private String buildContextPrompt(String question,
List<SectionEntity> sections,
List<FigureEntity> figures) {
StringBuilder sb = new StringBuilder();
if (!sections.isEmpty()) {
sb.append("CONTEXT:\n\n");
for (SectionEntity section : sections) {
sb.append("[").append(section.getTitle())
.append(", p.").append(section.getPageStart()).append("]\n");
sb.append(section.getFullText()).append("\n\n");
}
}
if (!figures.isEmpty()) {
sb.append("AVAILABLE FIGURES:\n");
for (FigureEntity figure : figures) {
sb.append("- ").append(figure.getLabel() != null ? figure.getLabel() : "Figure")
.append(" (p.").append(figure.getPage()).append("): ")
.append(figure.getCaption() != null ? figure.getCaption() : "")
.append("\n");
}
sb.append("\nWhen referencing diagrams, cite them as [Fig. X, p.N].\n\n");
}
sb.append("QUESTION:\n").append(question);
return sb.toString();
}
private List<Map<String, Object>> buildSources(List<SectionEntity> sections,
List<FigureEntity> figures) {
List<Map<String, Object>> sources = new ArrayList<>();
for (SectionEntity section : sections) {
Map<String, Object> source = new LinkedHashMap<>();
source.put("type", "TEXT");
source.put("bookTitle", deriveTitleFromSection(section));
source.put("page", section.getPageStart());
source.put("chunkText", truncate(section.getFullText(), 500));
sources.add(source);
}
for (FigureEntity figure : figures) {
Map<String, Object> source = new LinkedHashMap<>();
source.put("type", "FIGURE");
source.put("bookTitle", bookRepository.findById(figure.getBookId())
.map(com.aiteacher.book.Book::getTitle).orElse("Book"));
source.put("page", figure.getPage());
source.put("figureId", figure.getId());
source.put("label", figure.getLabel() != null ? figure.getLabel() : "");
source.put("caption", figure.getCaption() != null ? figure.getCaption() : "");
source.put("figureType", figure.getFigureType().name());
// imageUrl assembled from relative path: figures/{bookId}/{filename}
String filename = figure.getImagePath().substring(
figure.getImagePath().lastIndexOf('/') + 1);
source.put("imageUrl", "/api/v1/figures/" + figure.getBookId() + "/" + filename);
sources.add(source);
}
return sources;
}
private String deriveTitleFromSection(SectionEntity section) {
if (section == null) return "Book";
return bookRepository.findById(section.getBookId())
.map(com.aiteacher.book.Book::getTitle)
.orElse("Book");
}
private String buildQuestionWithHistory(List<Message> history, String currentQuestion, private String buildQuestionWithHistory(List<Message> history, String currentQuestion,
String topicId) { String topicId) {
boolean hasTopic = topicId != null && !topicId.equals("free-form"); boolean hasTopic = topicId != null && !topicId.equals("free-form");
if (history.size() <= 1) { if (history.size() <= 1) {
return hasTopic return hasTopic
? String.format("[Context: This is a question about the neurosurgery topic '%s']\n%s", ? String.format("[Context: question about neurosurgery topic '%s']\n%s",
topicId, currentQuestion) topicId, currentQuestion)
: currentQuestion; : currentQuestion;
} }
StringBuilder sb = new StringBuilder(); StringBuilder sb = new StringBuilder();
if (hasTopic) { if (hasTopic) {
sb.append(String.format("[Context: This conversation is about the neurosurgery topic '%s']\n\n", sb.append(String.format("[Context: conversation about '%s']\n\n", topicId));
topicId));
} }
sb.append("Previous conversation:\n"); sb.append("Previous conversation:\n");
// Include all messages except the last (which is the current user message just saved)
for (int i = 0; i < history.size() - 1; i++) { for (int i = 0; i < history.size() - 1; i++) {
Message msg = history.get(i); Message msg = history.get(i);
sb.append(msg.getRole().name()).append(": ").append(msg.getContent()).append("\n"); sb.append(msg.getRole().name()).append(": ").append(msg.getContent()).append("\n");
@@ -144,30 +219,8 @@ public class ChatService {
return sb.toString(); return sb.toString();
} }
private List<Map<String, Object>> extractSources(ChatResponse response) { private String truncate(String text, int maxChars) {
List<Map<String, Object>> sources = new ArrayList<>(); if (text == null) return "";
return text.length() <= maxChars ? text : text.substring(0, maxChars) + "";
if (response.getMetadata() != null) {
Object retrieved = response.getMetadata().get(QuestionAnswerAdvisor.RETRIEVED_DOCUMENTS);
if (retrieved instanceof List<?> docs) {
for (Object docObj : docs) {
if (docObj instanceof Document doc) {
Map<String, Object> metadata = doc.getMetadata();
String bookTitle = (String) metadata.get("book_title");
Object pageObj = metadata.get("page_number");
Integer page = pageObj instanceof Number n ? n.intValue() : null;
if (bookTitle != null) {
Map<String, Object> source = new HashMap<>();
source.put("bookTitle", bookTitle);
source.put("page", page);
source.put("chunkText", doc.getText());
sources.add(source);
}
}
}
}
}
return sources;
} }
} }
@@ -0,0 +1,37 @@
package com.aiteacher.config;
import com.aiteacher.figure.FigureStorageService;
import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.server.ResponseStatusException;
import jakarta.servlet.http.HttpServletResponse;
import java.io.IOException;
/**
* Serves figure images by redirecting to a presigned S3 URL.
* The key stored in DB is the full S3 object key, e.g. "figures/{bookId}/{figureId}.png".
*/
@RestController
@RequestMapping("/api/v1/figures")
public class FigureStorageConfig {
private final FigureStorageService figureStorageService;
public FigureStorageConfig(FigureStorageService figureStorageService) {
this.figureStorageService = figureStorageService;
}
@GetMapping("/{bookId}/{filename}")
public void serve(@PathVariable String bookId,
@PathVariable String filename,
HttpServletResponse response) throws IOException {
String key = "figures/" + bookId + "/" + filename;
try {
String url = figureStorageService.presignedUrl(key);
response.sendRedirect(url);
} catch (Exception ex) {
throw new ResponseStatusException(HttpStatus.NOT_FOUND, "Figure not found: " + key);
}
}
}
@@ -0,0 +1,30 @@
package com.aiteacher.config;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.client.JdkClientHttpRequestFactory;
import org.springframework.web.client.RestClient;
import java.net.http.HttpClient;
@Configuration
public class MarkerConfig {
@Value("${app.marker.base-url:http://localhost:8000}")
private String markerBaseUrl;
@Bean
RestClient markerRestClient() {
// Use the JDK HTTP client with no timeout — Marker conversions can take several minutes.
HttpClient httpClient = HttpClient.newBuilder()
.build();
JdkClientHttpRequestFactory factory = new JdkClientHttpRequestFactory(httpClient);
// No read timeout set: JDK HTTP client defaults to no deadline.
return RestClient.builder()
.baseUrl(markerBaseUrl)
.requestFactory(factory)
.build();
}
}
@@ -20,7 +20,9 @@ public class SecurityConfig {
@Bean @Bean
public SecurityFilterChain filterChain(HttpSecurity http) throws Exception { public SecurityFilterChain filterChain(HttpSecurity http) throws Exception {
http http
.authorizeHttpRequests(auth -> auth.anyRequest().authenticated()) .authorizeHttpRequests(auth -> auth
.requestMatchers("/api/v1/figures/**").permitAll()
.anyRequest().authenticated())
.httpBasic(Customizer.withDefaults()) .httpBasic(Customizer.withDefaults())
.csrf(AbstractHttpConfigurer::disable); .csrf(AbstractHttpConfigurer::disable);
return http.build(); return http.build();
@@ -0,0 +1,47 @@
package com.aiteacher.document;
import jakarta.persistence.*;
import java.time.Instant;
import java.util.UUID;
@Entity
@Table(name = "chapter")
public class ChapterEntity {
@Id
@Column(name = "id", length = 200)
private String id;
@Column(name = "book_id", nullable = false)
private UUID bookId;
@Column(name = "number", nullable = false)
private int number;
@Column(name = "title", length = 500)
private String title;
@Column(name = "page_start")
private Integer pageStart;
@Column(name = "created_at", nullable = false)
private Instant createdAt;
public ChapterEntity() {}
public ChapterEntity(String id, UUID bookId, int number, String title, Integer pageStart) {
this.id = id;
this.bookId = bookId;
this.number = number;
this.title = title;
this.pageStart = pageStart;
this.createdAt = Instant.now();
}
public String getId() { return id; }
public UUID getBookId() { return bookId; }
public int getNumber() { return number; }
public String getTitle() { return title; }
public Integer getPageStart() { return pageStart; }
public Instant getCreatedAt() { return createdAt; }
}
@@ -0,0 +1,9 @@
package com.aiteacher.document;
import org.springframework.data.jpa.repository.JpaRepository;
import java.util.UUID;
public interface ChapterRepository extends JpaRepository<ChapterEntity, String> {
void deleteAllByBookId(UUID bookId);
}
@@ -0,0 +1,58 @@
package com.aiteacher.document;
import jakarta.persistence.*;
import java.io.Serializable;
import java.util.Objects;
import java.util.UUID;
@Entity
@Table(name = "chunk_figure_ref")
@IdClass(ChunkFigureRefEntity.PK.class)
public class ChunkFigureRefEntity {
@Id
@Column(name = "chunk_id", nullable = false)
private UUID chunkId;
@Id
@Column(name = "figure_id", nullable = false, length = 200)
private String figureId;
@Column(name = "mention_page")
private Integer mentionPage;
public ChunkFigureRefEntity() {}
public ChunkFigureRefEntity(UUID chunkId, String figureId, Integer mentionPage) {
this.chunkId = chunkId;
this.figureId = figureId;
this.mentionPage = mentionPage;
}
public UUID getChunkId() { return chunkId; }
public String getFigureId() { return figureId; }
public Integer getMentionPage() { return mentionPage; }
public static class PK implements Serializable {
private UUID chunkId;
private String figureId;
public PK() {}
public PK(UUID chunkId, String figureId) {
this.chunkId = chunkId;
this.figureId = figureId;
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (!(o instanceof PK pk)) return false;
return Objects.equals(chunkId, pk.chunkId) && Objects.equals(figureId, pk.figureId);
}
@Override
public int hashCode() {
return Objects.hash(chunkId, figureId);
}
}
}
@@ -0,0 +1,18 @@
package com.aiteacher.document;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.Query;
import org.springframework.data.repository.query.Param;
import java.util.List;
import java.util.UUID;
public interface ChunkFigureRefRepository extends JpaRepository<ChunkFigureRefEntity, ChunkFigureRefEntity.PK> {
@Query("SELECT r FROM ChunkFigureRefEntity r WHERE r.chunkId IN :chunkIds")
List<ChunkFigureRefEntity> findByChunkIdIn(@Param("chunkIds") List<UUID> chunkIds);
@Query("DELETE FROM ChunkFigureRefEntity r WHERE r.figureId IN :figureIds")
@org.springframework.data.jpa.repository.Modifying
void deleteByFigureIdIn(@Param("figureIds") List<String> figureIds);
}
@@ -0,0 +1,62 @@
package com.aiteacher.document;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.ai.document.Document;
import org.springframework.stereotype.Service;
import java.util.List;
import java.util.UUID;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
/**
* Scans chunk text for "Fig. X" and "Figure X" references and persists
* ChunkFigureRefEntity rows linking that chunk to its referenced figures.
*/
@Service
public class ChunkFigureRefService {
private static final Logger log = LoggerFactory.getLogger(ChunkFigureRefService.class);
// Matches: "Fig. 12-4", "Fig. 12.4", "Fig 12", "Figure 12-4", etc.
private static final Pattern REF_PATTERN =
Pattern.compile("(?i)\\b(Fig\\.?|Figure)\\s+(\\d+[\\-.\\d]*)");
private final ChunkFigureRefRepository refRepository;
public ChunkFigureRefService(ChunkFigureRefRepository refRepository) {
this.refRepository = refRepository;
}
/**
* For each text chunk, finds figure references and persists ChunkFigureRefEntity rows.
*/
public void linkChunksToFigures(List<Document> chunks, List<FigureEntity> bookFigures,
int pageNum) {
if (bookFigures.isEmpty()) return;
for (Document chunk : chunks) {
String chunkIdStr = chunk.getId();
UUID chunkId;
try {
chunkId = UUID.fromString(chunkIdStr);
} catch (IllegalArgumentException ex) {
log.warn("Chunk has non-UUID id: {}", chunkIdStr);
continue;
}
Matcher m = REF_PATTERN.matcher(chunk.getText());
while (m.find()) {
String refNum = m.group(2).trim();
// Find matching figure by label suffix
for (FigureEntity figure : bookFigures) {
if (figure.getLabel() != null && figure.getLabel().endsWith(refNum)) {
refRepository.save(new ChunkFigureRefEntity(chunkId, figure.getId(), pageNum));
break;
}
}
}
}
}
}
@@ -0,0 +1,82 @@
package com.aiteacher.document;
import jakarta.persistence.*;
import java.time.Instant;
import java.util.UUID;
@Entity
@Table(name = "figure")
public class FigureEntity {
@Id
@Column(name = "id", length = 200)
private String id;
@Column(name = "book_id", nullable = false)
private UUID bookId;
@Column(name = "section_id", length = 200)
private String sectionId;
@Column(name = "chapter_id", length = 200)
private String chapterId;
@Column(name = "label", length = 100)
private String label;
@Column(name = "caption", columnDefinition = "TEXT")
private String caption;
@Enumerated(EnumType.STRING)
@Column(name = "figure_type", nullable = false, length = 50)
private FigureType figureType;
@Column(name = "page", nullable = false)
private int page;
@Column(name = "image_path", nullable = false, length = 1000)
private String imagePath;
@Column(name = "caption_embedding_id")
private UUID captionEmbeddingId;
@Column(name = "created_at", nullable = false)
private Instant createdAt;
public FigureEntity() {}
public FigureEntity(String id, UUID bookId, String sectionId, String chapterId,
String label, String caption, FigureType figureType,
int page, String imagePath) {
this.id = id;
this.bookId = bookId;
this.sectionId = sectionId;
this.chapterId = chapterId;
this.label = label;
this.caption = caption;
this.figureType = figureType;
this.page = page;
this.imagePath = imagePath;
this.createdAt = Instant.now();
}
public String getId() { return id; }
public UUID getBookId() { return bookId; }
public String getSectionId() { return sectionId; }
public String getChapterId() { return chapterId; }
public String getLabel() { return label; }
public String getCaption() { return caption; }
public FigureType getFigureType() { return figureType; }
public int getPage() { return page; }
public String getImagePath() { return imagePath; }
public UUID getCaptionEmbeddingId() { return captionEmbeddingId; }
public Instant getCreatedAt() { return createdAt; }
public void setCaptionEmbeddingId(UUID captionEmbeddingId) {
this.captionEmbeddingId = captionEmbeddingId;
}
public void setCaption(String caption) {
this.caption = caption;
}
}
@@ -0,0 +1,151 @@
package com.aiteacher.document;
import com.aiteacher.figure.FigureStorageService;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import javax.imageio.ImageIO;
import java.awt.image.BufferedImage;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
/**
* Extracts figure images from {@link PageResult.FigureData} entries produced by
* {@link MarkerPageParser}.
*
* <p>Marker returns pre-cropped PNG bytes for each detected figure, so no PDFBox
* page rendering or bounding-box cropping is needed. This service:
* <ol>
* <li>Decodes the PNG bytes to check dimensions (skip images below min size)</li>
* <li>Classifies the figure type from caption and surrounding text keywords</li>
* <li>Persists the image via {@link FigureStorageService}</li>
* <li>Persists a {@link FigureEntity} to the database</li>
* </ol>
*/
@Service
public class FigureExtractionService {
private static final Logger log = LoggerFactory.getLogger(FigureExtractionService.class);
private static final Pattern LABEL_PATTERN =
Pattern.compile("(?i)Fig\\.?\\s*(\\d+[\\-.\\d]*)");
private final FigureStorageService storageService;
private final FigureRepository figureRepository;
private final int minImageSizePx;
public FigureExtractionService(
FigureStorageService storageService,
FigureRepository figureRepository,
@Value("${app.figure-storage.min-image-size-px:100}") int minImageSizePx) {
this.storageService = storageService;
this.figureRepository = figureRepository;
this.minImageSizePx = minImageSizePx;
}
/** Holds the extraction output: persisted figures and a Marker blockId → DB figureId map. */
public record ExtractionResult(List<FigureEntity> figures, Map<String, String> blockIdToFigureId) {}
/**
* Extracts and persists figures for all pages described by {@code pageResults}.
*
* @param bookId owning book
* @param chapterId chapter bucket for these sections
* @param pageResults Marker parse output — each entry's {@code figures} list
* carries pre-cropped PNG bytes for that page
* @return {@link ExtractionResult} with persisted figures and blockId→figureId map
* (used to resolve markdown image placeholders)
*/
public ExtractionResult extract(UUID bookId, String chapterId,
List<PageResult> pageResults) {
List<FigureEntity> figures = new ArrayList<>();
Map<String, String> blockIdToFigureId = new HashMap<>();
int figureCounter = 0;
for (PageResult page : pageResults) {
if (page.figures().isEmpty()) continue;
for (PageResult.FigureData figureData : page.figures()) {
try {
BufferedImage image = decodeImage(figureData.imageBytes());
if (image == null) {
log.debug("Could not decode image on page {} of book {} (block {})",
page.pageNumber(), bookId, figureData.blockId());
continue;
}
if (image.getWidth() < minImageSizePx || image.getHeight() < minImageSizePx) {
log.debug("Skipping small figure on page {} ({}×{})",
page.pageNumber(), image.getWidth(), image.getHeight());
continue;
}
figureCounter++;
String figureId = bookId + "-fig-" + page.pageNumber() + "-" + figureCounter;
String caption = figureData.nearestCaption();
String label = detectLabel(caption, figureCounter);
FigureType type = classifyType(caption, page.orderedText());
String sectionId = bookId + "-p" + page.pageNumber();
String imagePath = storageService.save(bookId, figureId, image);
FigureEntity figure = new FigureEntity(
figureId, bookId, sectionId, chapterId,
label, caption, type, page.pageNumber(), imagePath);
figures.add(figureRepository.save(figure));
blockIdToFigureId.put(figureData.blockId(), figureId);
} catch (Exception ex) {
log.warn("Failed to extract figure on page {} of book {}: {}",
page.pageNumber(), bookId, ex.getMessage());
}
}
}
log.info("Extracted {} figures for book {}", figures.size(), bookId);
return new ExtractionResult(figures, blockIdToFigureId);
}
// --- Private helpers ---
private BufferedImage decodeImage(byte[] imageBytes) {
if (imageBytes == null || imageBytes.length == 0) return null;
try {
return ImageIO.read(new ByteArrayInputStream(imageBytes));
} catch (IOException ex) {
return null;
}
}
private String detectLabel(String caption, int counter) {
if (caption != null) {
Matcher m = LABEL_PATTERN.matcher(caption);
if (m.find()) return "Fig. " + m.group(1).trim();
}
return "Fig. " + counter;
}
private FigureType classifyType(String caption, String pageText) {
String combined = ((caption != null ? caption : "") + " " +
(pageText != null ? pageText : "")).toLowerCase();
if (combined.contains("mri") || combined.contains("ct ") || combined.contains("magnetic")
|| combined.contains("tomography")) return FigureType.MRI_CT_SCAN;
if (combined.contains("intraoperative") || combined.contains("intra-op"))
return FigureType.INTRAOPERATIVE_IMAGE;
if (caption != null && caption.toLowerCase().startsWith("table"))
return FigureType.TABLE;
if (combined.contains("chart") || combined.contains("histogram") || combined.contains("graph"))
return FigureType.CHART;
if (combined.contains("photograph") || combined.contains("photo"))
return FigureType.SURGICAL_PHOTOGRAPH;
return FigureType.ANATOMICAL_DIAGRAM;
}
}
@@ -0,0 +1,11 @@
package com.aiteacher.document;
import org.springframework.data.jpa.repository.JpaRepository;
import java.util.List;
import java.util.UUID;
public interface FigureRepository extends JpaRepository<FigureEntity, String> {
List<FigureEntity> findAllByBookId(UUID bookId);
void deleteAllByBookId(UUID bookId);
}
@@ -0,0 +1,10 @@
package com.aiteacher.document;
public enum FigureType {
ANATOMICAL_DIAGRAM,
SURGICAL_PHOTOGRAPH,
MRI_CT_SCAN,
TABLE,
CHART,
INTRAOPERATIVE_IMAGE
}
@@ -0,0 +1,14 @@
package com.aiteacher.document;
import java.util.UUID;
public interface MarkdownStorageService {
/** Uploads the markdown content and returns the S3 key. */
String save(UUID bookId, int pageNumber, String markdown);
/** Downloads and returns the markdown content for the given book and page. */
String getText(UUID bookId, int pageNumber);
/** Deletes all markdown files for the given book. */
void deleteAll(UUID bookId);
}
@@ -0,0 +1,287 @@
package com.aiteacher.document;
import tools.jackson.databind.JsonNode;
import tools.jackson.databind.ObjectMapper;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.core.io.FileSystemResource;
import org.springframework.http.MediaType;
import org.springframework.stereotype.Service;
import org.springframework.util.LinkedMultiValueMap;
import org.springframework.util.MultiValueMap;
import org.springframework.web.client.RestClient;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.*;
/**
* Parses a PDF with a single call to the Marker server using {@code output_format=json}.
*
* <p>The JSON response contains an {@code output} field that is itself a JSON string with a
* tree structure: the root has a {@code children} array where each item is a {@code Page} block.
* Each block carries an {@code html} field with {@code <content-ref src='blockId'>} placeholders
* that reference its {@code children} by ID.
*
* <p>{@link #jsonToHtml} mirrors the Marker Python {@code json_to_html} utility: it walks the
* tree recursively and resolves every {@code content-ref} with the rendered HTML of the
* referenced child block.
*
* <p>Returns a {@link ParsedBook} with:
* <ul>
* <li>{@code pages} — one {@link PageResult} per non-empty page (drives embeddings)</li>
* <li>{@code htmlByPage} — full resolved HTML per page (saved to S3 for the reader)</li>
* </ul>
*/
@Service
public class MarkerPageParser {
private static final Logger log = LoggerFactory.getLogger(MarkerPageParser.class);
private static final Set<String> TEXT_BLOCK_TYPES = Set.of(
"Text", "TextInlineMath", "ListItem", "Table", "TableOfContents", "Code", "Equation",
"Footnote", "Caption", "PageHeader", "PageFooter", "Handwriting"
);
private static final Set<String> FIGURE_BLOCK_TYPES = Set.of("Figure", "Picture", "FigureGroup", "PictureGroup");
private static final ObjectMapper MAPPER = new ObjectMapper();
private final RestClient restClient;
public MarkerPageParser(@Qualifier("markerRestClient") RestClient restClient) {
this.restClient = restClient;
}
public ParsedBook parse(Path pdfPath) {
log.info("Submitting {} to Marker (json)", pdfPath.getFileName());
MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
body.add("file", new FileSystemResource(pdfPath));
body.add("output_format", "json");
JsonNode response = restClient.post()
.uri("/marker/upload")
.contentType(MediaType.MULTIPART_FORM_DATA)
.body(body)
.retrieve()
.body(JsonNode.class);
try {
Files.writeString(Path.of("/tmp/marker-response-json.json"), response.toPrettyString());
} catch (IOException e) {
log.warn("Could not save Marker response to /tmp/marker-response-json.json", e);
}
List<JsonNode> pageNodes = extractPages(response);
if (pageNodes.isEmpty()) {
log.warn("Marker returned no pages for {}", pdfPath.getFileName());
return new ParsedBook(List.of(), Map.of());
}
log.info("Marker returned {} pages for {}", pageNodes.size(), pdfPath.getFileName());
List<PageResult> pages = new ArrayList<>();
Map<Integer, String> htmlByPage = new LinkedHashMap<>();
for (int i = 0; i < pageNodes.size(); i++) {
JsonNode pageNode = pageNodes.get(i);
int pageNumber = i + 1; // 1-based
PageResult result = buildPageResult(pageNode, pageNumber);
String html = jsonToHtml(pageNode);
if (!result.orderedText().isBlank() || !result.figures().isEmpty()) {
pages.add(result);
htmlByPage.put(pageNumber, html);
}
}
log.info("Marker produced {} non-empty pages from {}", pages.size(), pdfPath.getFileName());
return new ParsedBook(pages, htmlByPage);
}
// ── Page extraction ───────────────────────────────────────────────────────
/**
* Parses the {@code output} JSON string and returns the list of page nodes
* (the top-level {@code children} of the document root).
*/
private List<JsonNode> extractPages(JsonNode response) {
if (response == null) return List.of();
JsonNode outputNode = response.path("output");
if (outputNode.isMissingNode()) {
log.warn("Marker response has no 'output' field");
return List.of();
}
try {
JsonNode root = MAPPER.readTree(outputNode.stringValue());
JsonNode children = root.path("children");
if (children.isMissingNode() || !children.isArray()) {
log.warn("Marker output root has no 'children' array");
return List.of();
}
List<JsonNode> result = new ArrayList<>();
children.forEach(result::add);
return result;
} catch (Exception e) {
log.warn("Could not parse Marker 'output' string as JSON: {}", e.getMessage());
return List.of();
}
}
// ── HTML rendering ────────────────────────────────────────────────────────
/**
* Java equivalent of the Marker Python {@code json_to_html} utility.
*
* <p>Algorithm:
* <ol>
* <li>If the block has no children, return its {@code html} as-is (leaf node).</li>
* <li>Otherwise recursively render each child, then replace every
* {@code <content-ref src='childId'>} placeholder in the block's own {@code html}
* with the rendered child HTML.</li>
* </ol>
*/
String jsonToHtml(JsonNode block) {
String html = str(block.path("html"));
// If the block carries image data, inject <img> data-URI tags.
// Marker stores base64 image bytes in block.images keyed by block ID.
// Picture/Figure leaf blocks have empty html, so this is the only way to
// get the image into the rendered output.
JsonNode images = block.path("images");
if (!images.isMissingNode() && !images.isNull() && !images.isEmpty()) {
StringBuilder imgTags = new StringBuilder();
images.properties().forEach(entry -> {
String base64 = str(entry.getValue());
if (!base64.isEmpty()) {
String mime = detectImageMime(base64);
imgTags.append("<img src=\"data:").append(mime)
.append(";base64,").append(base64).append("\">");
}
});
if (!imgTags.isEmpty()) {
html = html + imgTags;
}
}
JsonNode children = block.path("children");
if (children.isMissingNode() || children.isNull() || !children.isArray() || children.isEmpty()) {
return html; // leaf node
}
// Build id → rendered-html map for all direct children
Map<String, String> childHtml = new LinkedHashMap<>();
for (JsonNode child : children) {
String id = str(child.path("id"));
childHtml.put(id, jsonToHtml(child));
}
// Replace every <content-ref src='id'></content-ref> with the child's HTML
for (Map.Entry<String, String> entry : childHtml.entrySet()) {
String ref = "<content-ref src='" + entry.getKey() + "'></content-ref>";
html = html.replace(ref, entry.getValue());
}
return html;
}
// ── PageResult (text + figures for embeddings) ────────────────────────────
private PageResult buildPageResult(JsonNode pageBlock, int pageNumber) {
StringBuilder text = new StringBuilder();
String[] headingTitle = {null};
List<PageResult.FigureData> figures = new ArrayList<>();
walkBlock(pageBlock, text, headingTitle, figures);
return new PageResult(pageNumber, text.toString().strip(), headingTitle[0], figures);
}
/** Recursively walks the block tree, collecting text and figures in reading order. */
private void walkBlock(JsonNode block, StringBuilder text, String[] headingTitle,
List<PageResult.FigureData> figures) {
String type = str(block.path("block_type"));
if ("SectionHeader".equals(type)) {
String heading = stripHtml(str(block.path("html"))).strip();
if (!heading.isEmpty() && headingTitle[0] == null) headingTitle[0] = heading;
appendText(text, heading);
} else if (TEXT_BLOCK_TYPES.contains(type)) {
appendText(text, stripHtml(str(block.path("html"))));
} else if (FIGURE_BLOCK_TYPES.contains(type)) {
String caption = findCaption(block);
extractFigures(block, caption, figures);
}
// Recurse into children (content-ref ordering is implicit via tree order)
JsonNode children = block.path("children");
if (!children.isMissingNode() && !children.isNull() && children.isArray()) {
for (JsonNode child : children) {
walkBlock(child, text, headingTitle, figures);
}
}
}
/** Finds the first Caption child inside a figure block, if any. */
private String findCaption(JsonNode figureBlock) {
JsonNode children = figureBlock.path("children");
if (children.isMissingNode() || !children.isArray()) return null;
for (JsonNode child : children) {
if ("Caption".equals(str(child.path("block_type")))) {
String caption = stripHtml(str(child.path("html"))).strip();
return caption.isEmpty() ? null : caption;
}
}
return null;
}
private void extractFigures(JsonNode block, String caption, List<PageResult.FigureData> out) {
JsonNode images = block.path("images");
if (images.isMissingNode() || images.isEmpty()) return;
images.properties().forEach(entry -> {
String blockId = entry.getKey();
String base64 = str(entry.getValue());
if (base64.isEmpty()) return;
try {
byte[] bytes = Base64.getDecoder().decode(base64);
out.add(new PageResult.FigureData(bytes, caption, blockId));
} catch (IllegalArgumentException ex) {
log.warn("Could not decode base64 image for block {}: {}", blockId, ex.getMessage());
}
});
}
// ── Utilities ─────────────────────────────────────────────────────────────
private void appendText(StringBuilder sb, String text) {
if (text == null) return;
String stripped = text.strip();
if (stripped.isEmpty()) return;
if (sb.length() > 0) sb.append("\n\n");
sb.append(stripped);
}
private String stripHtml(String html) {
if (html == null || html.isEmpty()) return "";
return html.replaceAll("<[^>]*>", "").replaceAll("\\s{2,}", " ").strip();
}
/** Detects MIME type from the first characters of a base64-encoded image. */
private static String detectImageMime(String base64) {
if (base64.startsWith("/9j/")) return "image/jpeg";
if (base64.startsWith("iVBOR")) return "image/png";
if (base64.startsWith("R0lGO")) return "image/gif";
if (base64.startsWith("UklGR")) return "image/webp";
return "image/png"; // safe fallback
}
/** Null-safe string extraction from a JsonNode (Jackson 3: stringValue() returns null for non-strings). */
private static String str(JsonNode node) {
String v = node.stringValue();
return v != null ? v : "";
}
}
@@ -0,0 +1,25 @@
package com.aiteacher.document;
import java.util.List;
/**
* Internal DTO produced by MarkerPageParser for one PDF page.
* Decouples the Marker HTTP API from downstream services.
*/
public record PageResult(
int pageNumber, // 1-based, derived from Marker page block index
String orderedText, // full page text in correct reading order (blocks joined by \n\n)
String headingTitle, // first SectionHeader block on page, or null
List<FigureData> figures // extracted figure images (may be empty)
) {
/**
* A figure extracted from the page.
* Image bytes are PNG data decoded from the Marker JSON {@code images} map.
*/
public record FigureData(
byte[] imageBytes, // PNG image data (base64-decoded from Marker response)
String nearestCaption, // text of the adjacent Caption block, or null
String blockId // Marker block ID (e.g. "/page/0/Figure/2") for traceability
) {}
}
@@ -0,0 +1,16 @@
package com.aiteacher.document;
import java.util.List;
import java.util.Map;
/**
* Result of a full Marker parse: structured page data (from JSON) plus
* native per-page markdown (from the separate Markdown API call).
*
* @param pages one entry per non-empty page, derived from the chunks response
* @param htmlByPage concatenated block HTML keyed by 1-based page number
*/
public record ParsedBook(
List<PageResult> pages,
Map<Integer, String> htmlByPage
) {}
@@ -0,0 +1,114 @@
package com.aiteacher.document;
import org.apache.pdfbox.Loader;
import org.apache.pdfbox.pdmodel.PDDocument;
import org.apache.pdfbox.pdmodel.PDPage;
import org.apache.pdfbox.pdmodel.common.PDRectangle;
import org.apache.pdfbox.text.PDFTextStripperByArea;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.awt.Rectangle;
import java.io.IOException;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
import java.util.UUID;
/**
* Parses a PDF into page-level SectionEntity records stored in Postgres.
* Uses column-aware extraction via PDFTextStripperByArea: for two-column pages,
* left column is extracted first then right, preserving correct reading order.
* Text is also normalized (collapsed whitespace) before storage.
*/
@Service
public class PdfStructureParser {
private static final Logger log = LoggerFactory.getLogger(PdfStructureParser.class);
// Right column is considered empty (single-column page) if it has < 20% of left column's content
private static final double TWO_COLUMN_THRESHOLD = 0.2;
private final ChapterRepository chapterRepository;
private final SectionRepository sectionRepository;
public PdfStructureParser(ChapterRepository chapterRepository,
SectionRepository sectionRepository) {
this.chapterRepository = chapterRepository;
this.sectionRepository = sectionRepository;
}
@Transactional
public List<SectionEntity> parse(UUID bookId, String bookTitle, Path pdfPath) {
log.info("Parsing PDF structure for book {}", bookId);
String chapterId = bookId + "-ch1";
ChapterEntity chapter = new ChapterEntity(chapterId, bookId, 1, bookTitle, 1);
chapterRepository.save(chapter);
List<SectionEntity> sections = new ArrayList<>();
try (PDDocument doc = Loader.loadPDF(pdfPath.toFile())) {
List<PDPage> pages = new ArrayList<>();
doc.getPages().forEach(pages::add);
for (int i = 0; i < 25; i++) {
int pageNum = i + 1;
String text = normalizeWhitespace(extractPageText(pages.get(i)));
if (text.isBlank()) continue;
String sectionId = bookId + "-p" + pageNum;
SectionEntity section = new SectionEntity(
sectionId, chapterId, bookId,
String.valueOf(pageNum),
"Page " + pageNum,
pageNum, pageNum,
text
);
sections.add(sectionRepository.save(section));
}
} catch (IOException e) {
throw new RuntimeException("Failed to parse PDF for book " + bookId, e);
}
log.info("Parsed {} sections for book {}", sections.size(), bookId);
return sections;
}
/**
* Extracts text from a single page using column-aware region extraction.
* Splits the page at the horizontal midpoint. If the right region has fewer
* than 20% of the characters of the left region, treats the page as single-column.
*/
private String extractPageText(PDPage page) throws IOException {
PDRectangle mediaBox = page.getMediaBox();
int width = (int) mediaBox.getWidth();
int height = (int) mediaBox.getHeight();
int mid = width / 2;
PDFTextStripperByArea stripper = new PDFTextStripperByArea();
stripper.setSortByPosition(true);
stripper.addRegion("left", new Rectangle(0, 0, mid, height));
stripper.addRegion("right", new Rectangle(mid, 0, width - mid, height));
stripper.extractRegions(page);
String left = stripper.getTextForRegion("left").strip();
String right = stripper.getTextForRegion("right").strip();
if (right.length() < left.length() * TWO_COLUMN_THRESHOLD) {
// Single-column page — left holds all (or nearly all) content
return left.isEmpty() ? right : left;
}
return left + "\n\n" + right;
}
/** Collapses multi-space/tab runs and excessive blank lines. */
private String normalizeWhitespace(String text) {
return text
.replaceAll("[ \t]{2,}", " ")
.replaceAll("\n{3,}", "\n\n")
.trim();
}
}
@@ -0,0 +1,97 @@
package com.aiteacher.document;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.S3Configuration;
import software.amazon.awssdk.services.s3.model.*;
import java.net.URI;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.UUID;
@Service
public class S3MarkdownStorageService implements MarkdownStorageService {
private static final Logger log = LoggerFactory.getLogger(S3MarkdownStorageService.class);
private final S3Client s3;
private final String bucket;
public S3MarkdownStorageService(
@Value("${app.figure-storage.endpoint}") String endpoint,
@Value("${app.figure-storage.region}") String region,
@Value("${app.figure-storage.bucket}") String bucket,
@Value("${app.figure-storage.access-key-id}") String accessKeyId,
@Value("${app.figure-storage.secret-access-key}") String secretKey) {
this.bucket = bucket;
URI endpointUri = URI.create(endpoint);
StaticCredentialsProvider credentials = StaticCredentialsProvider.create(
AwsBasicCredentials.create(accessKeyId, secretKey));
Region awsRegion = Region.of(region);
S3Configuration s3Config = S3Configuration.builder().pathStyleAccessEnabled(true).build();
this.s3 = S3Client.builder()
.endpointOverride(endpointUri)
.region(awsRegion)
.credentialsProvider(credentials)
.serviceConfiguration(s3Config)
.build();
}
@Override
public String save(UUID bookId, int pageNumber, String markdown) {
String key = key(bookId, pageNumber);
byte[] bytes = markdown.getBytes(StandardCharsets.UTF_8);
s3.putObject(
PutObjectRequest.builder().bucket(bucket).key(key)
.contentType("text/html; charset=utf-8")
.contentLength((long) bytes.length).build(),
RequestBody.fromBytes(bytes));
return key;
}
@Override
public String getText(UUID bookId, int pageNumber) {
byte[] bytes = s3.getObjectAsBytes(
GetObjectRequest.builder().bucket(bucket).key(key(bookId, pageNumber)).build()
).asByteArray();
return new String(bytes, StandardCharsets.UTF_8);
}
@Override
public void deleteAll(UUID bookId) {
String prefix = "html/" + bookId + "/";
try {
List<ObjectIdentifier> toDelete = new ArrayList<>();
s3.listObjectsV2Paginator(ListObjectsV2Request.builder()
.bucket(bucket).prefix(prefix).build()).stream()
.flatMap(page -> page.contents().stream())
.map(S3Object::key)
.map(k -> ObjectIdentifier.builder().key(k).build())
.forEach(toDelete::add);
if (toDelete.isEmpty()) return;
s3.deleteObjects(DeleteObjectsRequest.builder()
.bucket(bucket)
.delete(Delete.builder().objects(toDelete).build())
.build());
log.info("Deleted {} markdown files from S3 for book {}", toDelete.size(), bookId);
} catch (S3Exception ex) {
log.warn("Could not fully delete markdown for book {} from S3: {}", bookId, ex.getMessage());
}
}
private static String key(UUID bookId, int pageNumber) {
return "html/" + bookId + "/page-" + pageNumber + ".html";
}
}
@@ -0,0 +1,63 @@
package com.aiteacher.document;
import jakarta.persistence.*;
import java.time.Instant;
import java.util.UUID;
@Entity
@Table(name = "section")
public class SectionEntity {
@Id
@Column(name = "id", length = 200)
private String id;
@Column(name = "chapter_id", nullable = false, length = 200)
private String chapterId;
@Column(name = "book_id", nullable = false)
private UUID bookId;
@Column(name = "number", length = 50)
private String number;
@Column(name = "title", length = 500)
private String title;
@Column(name = "page_start", nullable = false)
private int pageStart;
@Column(name = "page_end", nullable = false)
private int pageEnd;
@Column(name = "full_text", nullable = false, columnDefinition = "TEXT")
private String fullText;
@Column(name = "created_at", nullable = false)
private Instant createdAt;
public SectionEntity() {}
public SectionEntity(String id, String chapterId, UUID bookId, String number,
String title, int pageStart, int pageEnd, String fullText) {
this.id = id;
this.chapterId = chapterId;
this.bookId = bookId;
this.number = number;
this.title = title;
this.pageStart = pageStart;
this.pageEnd = pageEnd;
this.fullText = fullText;
this.createdAt = Instant.now();
}
public String getId() { return id; }
public String getChapterId() { return chapterId; }
public UUID getBookId() { return bookId; }
public String getNumber() { return number; }
public String getTitle() { return title; }
public int getPageStart() { return pageStart; }
public int getPageEnd() { return pageEnd; }
public String getFullText() { return fullText; }
public Instant getCreatedAt() { return createdAt; }
}
@@ -0,0 +1,11 @@
package com.aiteacher.document;
import org.springframework.data.jpa.repository.JpaRepository;
import java.util.List;
import java.util.UUID;
public interface SectionRepository extends JpaRepository<SectionEntity, String> {
List<SectionEntity> findAllByBookId(UUID bookId);
void deleteAllByBookId(UUID bookId);
}
@@ -0,0 +1,103 @@
package com.aiteacher.document;
import org.springframework.ai.document.Document;
import org.springframework.stereotype.Service;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.UUID;
/**
* Splits a SectionEntity's full text into overlapping chunks for vector embedding.
* Target size: ~1800 characters (~450 tokens); overlap: 200 characters.
*/
@Service
public class TextChunkingService {
private static final int TARGET_CHARS = 1800;
private static final int OVERLAP_CHARS = 200;
public List<Document> chunk(SectionEntity section, String bookTitle) {
String text = section.getFullText();
if (text == null || text.isBlank()) return List.of();
List<String> windows = split(text);
List<Document> documents = new ArrayList<>();
for (int i = 0; i < windows.size(); i++) {
String chunkId = UUID.randomUUID().toString();
Map<String, Object> metadata = buildMetadata(section, bookTitle, i, windows.size(), chunkId);
documents.add(new Document(chunkId, windows.get(i), metadata));
}
return documents;
}
private List<String> split(String text) {
List<String> windows = new ArrayList<>();
int start = 0;
while (start < text.length()) {
int hardEnd = Math.min(start + TARGET_CHARS, text.length());
if (hardEnd == text.length()) {
String last = text.substring(start).strip();
if (!last.isEmpty()) windows.add(last);
break;
}
int splitAt = findSplitPoint(text, start, hardEnd);
String chunk = text.substring(start, splitAt).strip();
if (!chunk.isEmpty()) windows.add(chunk);
// Overlap: back up from split point, align to a word start
int overlapStart = Math.max(start + 1, splitAt - OVERLAP_CHARS);
while (overlapStart < splitAt && text.charAt(overlapStart) != ' ') overlapStart++;
start = overlapStart < splitAt ? overlapStart + 1 : splitAt;
}
return windows;
}
/**
* Finds the best split point at or before hardEnd, preferring (in order):
* paragraph boundary, sentence boundary, word boundary, hard cut.
*/
private int findSplitPoint(String text, int start, int hardEnd) {
int lookback = Math.min(400, (hardEnd - start) / 2);
// 1. Paragraph boundary
int paraIdx = text.lastIndexOf("\n\n", hardEnd);
if (paraIdx > hardEnd - lookback && paraIdx > start) return paraIdx + 2;
// 2. Sentence boundary (. ! ?) followed by space or newline
for (int i = hardEnd - 1; i > hardEnd - lookback && i > start; i--) {
char c = text.charAt(i);
if ((c == '.' || c == '!' || c == '?') && i + 1 < text.length()) {
char next = text.charAt(i + 1);
if (next == ' ' || next == '\n') return i + 1;
}
}
// 3. Word boundary
for (int i = hardEnd - 1; i > hardEnd - 100 && i > start; i--) {
if (text.charAt(i) == ' ') return i + 1;
}
// 4. Hard cut
return hardEnd;
}
private Map<String, Object> buildMetadata(SectionEntity section, String bookTitle,
int index, int total, String chunkId) {
Map<String, Object> m = new HashMap<>();
m.put("type", "TEXT");
m.put("book_id", section.getBookId().toString());
m.put("book_title", bookTitle);
m.put("chapter_id", section.getChapterId());
m.put("section_id", section.getId());
m.put("section_title", section.getTitle() != null ? section.getTitle() : "");
m.put("page_start", section.getPageStart());
m.put("page_end", section.getPageEnd());
m.put("chunk_index", index);
m.put("total_chunks", total);
m.put("chunk_id", chunkId);
return m;
}
}
@@ -0,0 +1,91 @@
package com.aiteacher.document;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.core.io.ByteArrayResource;
import org.springframework.stereotype.Service;
import org.springframework.util.MimeTypeUtils;
/**
* Analyses an extracted figure image using the OpenAI vision model.
*
* <p>Returns an {@link ImageAnalysis} record containing:
* <ul>
* <li>{@code description} — 2-3 sentence clinical description of the image</li>
* <li>{@code imageText} — all visible text, labels, and annotations copied verbatim
* from the image (empty string when none present)</li>
* </ul>
*
* <p>Both fields are stored: {@code description} drives the embedding; {@code imageText}
* is added to chunk metadata so queries can match exact labels (e.g., "Circle of Willis").
*/
@Service
public class VisionDescriptionService {
private static final Logger log = LoggerFactory.getLogger(VisionDescriptionService.class);
private static final String PROMPT = """
You are a neurosurgery educator analysing a medical image.
Respond in EXACTLY this format — no other text, no markdown:
DESCRIPTION: <2-3 sentence clinical description focusing on anatomical structures, surgical landmarks, and clinical significance>
IMAGE_TEXT: <all visible text, labels, measurements, and annotations copied verbatim, comma-separated; write NONE if no text visible>
""";
private final ChatClient chatClient;
public VisionDescriptionService(ChatClient chatClient) {
this.chatClient = chatClient;
}
/**
* Holds the structured output of a vision model call on one figure image.
*
* @param description clinical description of the image content
* @param imageText verbatim text visible inside the image; empty string if none
*/
public record ImageAnalysis(String description, String imageText) {}
/**
* Analyses the image bytes and returns an {@link ImageAnalysis}.
* Falls back gracefully: if the vision call fails, the caption is used as description
* and imageText is left empty.
*
* @param imageBytes PNG bytes of the extracted figure
* @param captionFallback caption detected from surrounding text, may be null
*/
public ImageAnalysis analyze(byte[] imageBytes, String captionFallback) {
try {
String raw = chatClient.prompt()
.user(u -> u
.text(PROMPT)
.media(MimeTypeUtils.IMAGE_PNG, new ByteArrayResource(imageBytes)))
.call()
.content();
return parse(raw, captionFallback);
} catch (Exception ex) {
log.warn("Vision analysis failed: {} — using caption as fallback", ex.getMessage());
return new ImageAnalysis(
captionFallback != null ? captionFallback : "Figure",
"");
}
}
private ImageAnalysis parse(String raw, String captionFallback) {
String description = captionFallback != null ? captionFallback : "Figure";
String imageText = "";
if (raw != null) {
for (String line : raw.split("\n")) {
if (line.startsWith("DESCRIPTION:")) {
String val = line.substring("DESCRIPTION:".length()).strip();
if (!val.isEmpty()) description = val;
} else if (line.startsWith("IMAGE_TEXT:")) {
String val = line.substring("IMAGE_TEXT:".length()).strip();
if (!val.isEmpty() && !"NONE".equalsIgnoreCase(val)) imageText = val;
}
}
}
return new ImageAnalysis(description, imageText);
}
}
@@ -0,0 +1,27 @@
package com.aiteacher.figure;
import java.awt.image.BufferedImage;
import java.util.UUID;
public interface FigureStorageService {
/**
* Saves an extracted image to S3 and returns the object key stored in the database.
*/
String save(UUID bookId, String figureId, BufferedImage image);
/**
* Downloads the image bytes for the given S3 object key.
*/
byte[] getBytes(String key);
/**
* Returns a presigned GET URL valid for 1 hour for the given S3 object key.
*/
String presignedUrl(String key);
/**
* Deletes all figure objects for the given book.
*/
void deleteAll(UUID bookId);
}
@@ -0,0 +1,132 @@
package com.aiteacher.figure;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.S3Configuration;
import software.amazon.awssdk.services.s3.model.*;
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
import software.amazon.awssdk.services.s3.presigner.model.GetObjectPresignRequest;
import software.amazon.awssdk.services.s3.model.S3Object;
import javax.imageio.ImageIO;
import java.awt.image.BufferedImage;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.net.URI;
import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import java.util.UUID;
@Service
public class S3FigureStorageService implements FigureStorageService {
private static final Logger log = LoggerFactory.getLogger(S3FigureStorageService.class);
private final S3Client s3;
private final S3Presigner presigner;
private final String bucket;
public S3FigureStorageService(
@Value("${app.figure-storage.endpoint}") String endpoint,
@Value("${app.figure-storage.region}") String region,
@Value("${app.figure-storage.bucket}") String bucket,
@Value("${app.figure-storage.access-key-id}") String accessKeyId,
@Value("${app.figure-storage.secret-access-key}") String secretKey) {
this.bucket = bucket;
URI endpointUri = URI.create(endpoint);
StaticCredentialsProvider credentials = StaticCredentialsProvider.create(
AwsBasicCredentials.create(accessKeyId, secretKey));
Region awsRegion = Region.of(region);
S3Configuration s3Config = S3Configuration.builder()
.pathStyleAccessEnabled(true)
.build();
this.s3 = S3Client.builder()
.endpointOverride(endpointUri)
.region(awsRegion)
.credentialsProvider(credentials)
.serviceConfiguration(s3Config)
.build();
this.presigner = S3Presigner.builder()
.endpointOverride(endpointUri)
.region(awsRegion)
.credentialsProvider(credentials)
.serviceConfiguration(s3Config)
.build();
}
@Override
public String save(UUID bookId, String figureId, BufferedImage image) {
String key = "figures/" + bookId + "/" + figureId + ".png";
try {
ByteArrayOutputStream out = new ByteArrayOutputStream();
ImageIO.write(image, "PNG", out);
byte[] bytes = out.toByteArray();
s3.putObject(
PutObjectRequest.builder().bucket(bucket).key(key)
.contentType("image/png").contentLength((long) bytes.length).build(),
RequestBody.fromBytes(bytes));
return key;
} catch (IOException ex) {
throw new RuntimeException("Failed to encode figure " + figureId, ex);
} catch (S3Exception ex) {
throw new RuntimeException("Failed to upload figure " + figureId + " to S3", ex);
}
}
@Override
public byte[] getBytes(String key) {
try {
return s3.getObjectAsBytes(
GetObjectRequest.builder().bucket(bucket).key(key).build()).asByteArray();
} catch (S3Exception ex) {
throw new RuntimeException("Failed to download figure from S3: " + key, ex);
}
}
@Override
public String presignedUrl(String key) {
GetObjectPresignRequest request = GetObjectPresignRequest.builder()
.signatureDuration(Duration.ofHours(1))
.getObjectRequest(r -> r.bucket(bucket).key(key))
.build();
return presigner.presignGetObject(request).url().toString();
}
@Override
public void deleteAll(UUID bookId) {
String prefix = "figures/" + bookId + "/";
try {
List<ObjectIdentifier> toDelete = new ArrayList<>();
ListObjectsV2Request listRequest = ListObjectsV2Request.builder()
.bucket(bucket).prefix(prefix).build();
s3.listObjectsV2Paginator(listRequest).stream()
.flatMap(page -> page.contents().stream())
.map(S3Object::key)
.map(k -> ObjectIdentifier.builder().key(k).build())
.forEach(toDelete::add);
if (toDelete.isEmpty()) return;
s3.deleteObjects(DeleteObjectsRequest.builder()
.bucket(bucket)
.delete(Delete.builder().objects(toDelete).build())
.build());
log.info("Deleted {} figures from S3 for book {}", toDelete.size(), bookId);
} catch (S3Exception ex) {
log.warn("Could not fully delete figures for book {} from S3: {}", bookId, ex.getMessage());
}
}
}
@@ -0,0 +1,111 @@
package com.aiteacher.retrieval;
import com.aiteacher.document.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.ai.document.Document;
import org.springframework.ai.vectorstore.SearchRequest;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.ai.vectorstore.filter.FilterExpressionBuilder;
import org.springframework.stereotype.Service;
import java.util.*;
/**
* Dual-modality retriever: searches text chunks and figure captions independently,
* then expands text hits to their parent sections and merges linked figures.
*/
@Service
public class NeurosurgeryRetriever {
private static final Logger log = LoggerFactory.getLogger(NeurosurgeryRetriever.class);
private static final int TEXT_TOP_K = 5;
private static final int FIGURE_TOP_K = 3;
private final VectorStore vectorStore;
private final SectionRepository sectionRepository;
private final FigureRepository figureRepository;
private final ChunkFigureRefRepository chunkFigureRefRepository;
public NeurosurgeryRetriever(VectorStore vectorStore,
SectionRepository sectionRepository,
FigureRepository figureRepository,
ChunkFigureRefRepository chunkFigureRefRepository) {
this.vectorStore = vectorStore;
this.sectionRepository = sectionRepository;
this.figureRepository = figureRepository;
this.chunkFigureRefRepository = chunkFigureRefRepository;
}
public RetrievalResult retrieve(String query, UUID bookId) {
FilterExpressionBuilder b = new FilterExpressionBuilder();
// 1. Text chunk search
List<Document> textHits = vectorStore.similaritySearch(
SearchRequest.builder()
.query(query)
.topK(TEXT_TOP_K)
.filterExpression(b.and(
b.eq("type", "TEXT"),
b.eq("book_id", bookId.toString())
).build())
.build()
);
// 2. Figure caption search (independent topK)
List<Document> figureHits = vectorStore.similaritySearch(
SearchRequest.builder()
.query(query)
.topK(FIGURE_TOP_K)
.filterExpression(b.and(
b.eq("type", "FIGURE"),
b.eq("book_id", bookId.toString())
).build())
.build()
);
// 3. Expand text chunks to parent sections from Postgres
List<String> sectionIds = textHits.stream()
.map(d -> (String) d.getMetadata().get("section_id"))
.filter(Objects::nonNull)
.distinct()
.toList();
List<SectionEntity> sections = sectionIds.isEmpty()
? List.of()
: sectionRepository.findAllById(sectionIds);
// 4. Fetch figures explicitly linked to retrieved chunks
List<UUID> chunkIds = textHits.stream()
.map(d -> {
try { return UUID.fromString(d.getId()); }
catch (Exception e) { return null; }
})
.filter(Objects::nonNull)
.toList();
List<String> linkedFigureIds = chunkIds.isEmpty()
? List.of()
: chunkFigureRefRepository.findByChunkIdIn(chunkIds)
.stream().map(ChunkFigureRefEntity::getFigureId).distinct().toList();
List<FigureEntity> linkedFigures = linkedFigureIds.isEmpty()
? List.of()
: figureRepository.findAllById(linkedFigureIds);
// 5. Collect figures from semantic figure search
List<String> semanticFigureIds = figureHits.stream()
.map(d -> (String) d.getMetadata().get("figure_id"))
.filter(Objects::nonNull)
.toList();
List<FigureEntity> semanticFigures = semanticFigureIds.isEmpty()
? List.of()
: figureRepository.findAllById(semanticFigureIds);
// 6. Merge and deduplicate figures by figureId (linked figures take precedence)
Map<String, FigureEntity> merged = new LinkedHashMap<>();
linkedFigures.forEach(f -> merged.put(f.getId(), f));
semanticFigures.forEach(f -> merged.putIfAbsent(f.getId(), f));
log.debug("Retrieved {} sections, {} figures for query", sections.size(), merged.size());
return new RetrievalResult(sections, new ArrayList<>(merged.values()));
}
}
@@ -0,0 +1,11 @@
package com.aiteacher.retrieval;
import com.aiteacher.document.FigureEntity;
import com.aiteacher.document.SectionEntity;
import java.util.List;
public record RetrievalResult(
List<SectionEntity> parentSections,
List<FigureEntity> figures
) {}
@@ -47,6 +47,26 @@ spring:
max-size: 8 max-size: 8
queue-capacity: 50 queue-capacity: 50
logging:
level:
"[org.apache.pdfbox]": ERROR
app: app:
features:
upload-enabled: ${UPLOAD_ENABLED:true}
delete-enabled: ${DELETE_ENABLED:true}
auth: auth:
password: ${APP_PASSWORD:changeme} password: ${APP_PASSWORD:changeme}
figure-storage:
endpoint: https://s3.immich-ad.ovh
region: garage
bucket: ${S3_BUCKET:aiteacher}
access-key-id: ${S3_ACCESS_KEY_ID}
secret-access-key: ${S3_SECRET_ACCESS_KEY}
min-image-size-px: 100
embedding:
batch-size: 20
batch-delay-ms: 2000
skip-embedding: true
marker:
base-url: ${MARKER_BASE_URL:http://192.168.1.105:8000}
@@ -0,0 +1,28 @@
-- ============================================================
-- V4: Document hierarchy — chapter and section tables
-- Supports parent-child retrieval pattern for RAG precision.
-- ============================================================
CREATE TABLE IF NOT EXISTS chapter (
id VARCHAR(200) PRIMARY KEY,
book_id UUID NOT NULL REFERENCES book(id) ON DELETE CASCADE,
number INT NOT NULL DEFAULT 1,
title VARCHAR(500),
page_start INT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS section (
id VARCHAR(200) PRIMARY KEY,
chapter_id VARCHAR(200) NOT NULL REFERENCES chapter(id) ON DELETE CASCADE,
book_id UUID NOT NULL REFERENCES book(id) ON DELETE CASCADE,
number VARCHAR(50),
title VARCHAR(500),
page_start INT NOT NULL,
page_end INT NOT NULL,
full_text TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_section_book ON section(book_id);
CREATE INDEX IF NOT EXISTS idx_section_chapter ON section(chapter_id);
@@ -0,0 +1,29 @@
-- ============================================================
-- V5: Figures and chunk-to-figure reference table
-- figure: metadata + file path for each extracted image
-- chunk_figure_ref: links vector-store chunks to figures
-- ============================================================
CREATE TABLE IF NOT EXISTS figure (
id VARCHAR(200) PRIMARY KEY,
book_id UUID NOT NULL REFERENCES book(id) ON DELETE CASCADE,
section_id VARCHAR(200) REFERENCES section(id) ON DELETE SET NULL,
chapter_id VARCHAR(200) REFERENCES chapter(id) ON DELETE SET NULL,
label VARCHAR(100),
caption TEXT,
figure_type VARCHAR(50) NOT NULL,
page INT NOT NULL,
image_path VARCHAR(1000) NOT NULL,
caption_embedding_id UUID,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS chunk_figure_ref (
chunk_id UUID NOT NULL,
figure_id VARCHAR(200) NOT NULL REFERENCES figure(id) ON DELETE CASCADE,
mention_page INT,
PRIMARY KEY (chunk_id, figure_id)
);
CREATE INDEX IF NOT EXISTS idx_figure_book ON figure(book_id);
CREATE INDEX IF NOT EXISTS idx_cfr_chunk ON chunk_figure_ref(chunk_id);
+6
View File
@@ -5,3 +5,9 @@ VITE_API_URL=/api/v1
# Shared password for HTTP Basic auth (must match APP_PASSWORD on the backend). # Shared password for HTTP Basic auth (must match APP_PASSWORD on the backend).
VITE_APP_PASSWORD=changeme VITE_APP_PASSWORD=changeme
# Set to 'false' to hide the upload UI (frontend). Also set UPLOAD_ENABLED=false on the backend to block the endpoint.
VITE_UPLOAD_ENABLED=true
# Set to 'false' to hide the delete button (frontend). Also set DELETE_ENABLED=false on the backend to block the endpoint.
VITE_DELETE_ENABLED=true
+5 -2
View File
@@ -64,11 +64,11 @@ body {
Ubuntu, Cantarell, 'Fira Sans', 'Droid Sans', 'Helvetica Neue', sans-serif; Ubuntu, Cantarell, 'Fira Sans', 'Droid Sans', 'Helvetica Neue', sans-serif;
background: #f0f4f8; background: #f0f4f8;
color: #2d3748; color: #2d3748;
min-height: 100vh; height: 100vh;
} }
#app { #app {
min-height: 100vh; height: 100vh;
display: flex; display: flex;
flex-direction: column; flex-direction: column;
} }
@@ -133,6 +133,9 @@ body {
.main-content { .main-content {
flex: 1; flex: 1;
min-height: 0;
display: flex;
flex-direction: column;
padding: 2rem; padding: 2rem;
max-width: 1200px; max-width: 1200px;
margin: 0 auto; margin: 0 auto;
+10
View File
@@ -33,7 +33,15 @@
</div> </div>
<div class="book-actions"> <div class="book-actions">
<router-link
v-if="book.status === 'READY'"
:to="{ name: 'book-reader', params: { id: book.id } }"
class="btn btn-secondary"
>
Read
</router-link>
<button <button
v-if="deleteEnabled"
class="btn btn-danger" class="btn btn-danger"
:disabled="book.status === 'PROCESSING' || deleting" :disabled="book.status === 'PROCESSING' || deleting"
@click="$emit('delete', book.id)" @click="$emit('delete', book.id)"
@@ -52,6 +60,7 @@ import type { Book } from '@/stores/bookStore'
const props = defineProps<{ const props = defineProps<{
book: Book book: Book
deleting?: boolean deleting?: boolean
deleteEnabled?: boolean
}>() }>()
defineEmits<{ defineEmits<{
@@ -181,6 +190,7 @@ function formatDate(iso: string): string {
.book-actions { .book-actions {
display: flex; display: flex;
justify-content: flex-end; justify-content: flex-end;
gap: 0.5rem;
margin-top: 0.25rem; margin-top: 0.25rem;
} }
</style> </style>
+125 -21
View File
@@ -5,22 +5,47 @@
<div v-if="isUser" class="message-content">{{ message.content }}</div> <div v-if="isUser" class="message-content">{{ message.content }}</div>
<div v-else class="message-content message-content--markdown" v-html="renderedContent"></div> <div v-else class="message-content message-content--markdown" v-html="renderedContent"></div>
<!-- Source chips for assistant messages --> <!-- Sources for assistant messages -->
<div v-if="!isUser && message.sources && message.sources.length > 0" class="message-sources"> <div v-if="!isUser && message.sources && message.sources.length > 0" class="message-sources">
<div class="sources-label">Sources:</div> <div class="sources-label">Sources:</div>
<div class="source-list"> <div class="source-list">
<!-- TEXT sources -->
<div <div
v-for="(source, idx) in message.sources" v-for="(source, idx) in textSources"
:key="idx" :key="'text-' + idx"
class="source-item" class="source-item"
> >
<div class="source-chip"> <div class="source-chip source-chip--text">
<span class="source-book-icon">📖</span> <span class="source-icon">📖</span>
<span class="source-book-title">{{ source.bookTitle }}</span> <span class="source-book-title">{{ source.bookTitle }}</span>
<span v-if="source.page" class="source-page">p.&nbsp;{{ source.page }}</span> <span v-if="source.page" class="source-page">p.&nbsp;{{ source.page }}</span>
</div> </div>
<div v-if="source.chunkText" class="source-chunk">{{ source.chunkText }}</div> <div v-if="source.chunkText" class="source-chunk">{{ source.chunkText }}</div>
</div> </div>
<!-- FIGURE sources -->
<div
v-for="(source, idx) in figureSources"
:key="'fig-' + idx"
class="source-item source-item--figure"
>
<div class="source-chip source-chip--figure">
<span class="source-icon">🖼</span>
<span class="source-figure-label">{{ source.label || 'Figure' }}</span>
<span v-if="source.page" class="source-page">p.&nbsp;{{ source.page }}</span>
<span v-if="source.figureType" class="source-figure-type">{{ formatFigureType(source.figureType) }}</span>
</div>
<div v-if="source.caption" class="source-caption">{{ source.caption }}</div>
<div class="source-figure-image">
<img
:src="source.imageUrl"
:alt="source.caption || source.label || 'Figure'"
class="figure-img"
loading="lazy"
@error="onImageError"
/>
</div>
</div>
</div> </div>
</div> </div>
@@ -32,7 +57,7 @@
<script setup lang="ts"> <script setup lang="ts">
import { computed } from 'vue' import { computed } from 'vue'
import { marked } from 'marked' import { marked } from 'marked'
import type { ChatMessage } from '@/stores/chatStore' import type { ChatMessage, ChatSource } from '@/stores/chatStore'
const props = defineProps<{ const props = defineProps<{
message: ChatMessage message: ChatMessage
@@ -41,6 +66,36 @@ const props = defineProps<{
const isUser = computed(() => props.message.role === 'USER') const isUser = computed(() => props.message.role === 'USER')
const renderedContent = computed(() => marked.parse(props.message.content) as string) const renderedContent = computed(() => marked.parse(props.message.content) as string)
const textSources = computed(() =>
(props.message.sources ?? []).filter((s: ChatSource) => s.type === 'TEXT' || !s.type)
)
const figureSources = computed(() =>
(props.message.sources ?? []).filter((s: ChatSource) => s.type === 'FIGURE')
)
function formatFigureType(type: string): string {
const labels: Record<string, string> = {
ANATOMICAL_DIAGRAM: 'Anatomical Diagram',
SURGICAL_PHOTOGRAPH: 'Surgical Photo',
MRI_CT_SCAN: 'MRI / CT',
TABLE: 'Table',
CHART: 'Chart',
INTRAOPERATIVE_IMAGE: 'Intraoperative'
}
return labels[type] ?? type
}
function onImageError(e: Event) {
const img = e.target as HTMLImageElement
img.alt = 'Image unavailable'
img.style.display = 'none'
const wrapper = img.parentElement
if (wrapper) {
wrapper.innerHTML = '<span class="figure-missing">Image unavailable</span>'
}
}
function formatTime(iso: string): string { function formatTime(iso: string): string {
return new Date(iso).toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' }) return new Date(iso).toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' })
} }
@@ -182,6 +237,55 @@ function formatTime(iso: string): string {
gap: 0.25rem; gap: 0.25rem;
} }
.source-item--figure {
gap: 0.4rem;
}
.source-chip {
display: inline-flex;
align-items: center;
gap: 0.25rem;
border-radius: 4px;
padding: 0.2rem 0.5rem;
font-size: 0.78rem;
}
.source-chip--text {
background: #ebf8ff;
border: 1px solid #bee3f8;
}
.source-chip--figure {
background: #f0fff4;
border: 1px solid #9ae6b4;
}
.source-icon {
font-size: 0.8rem;
}
.source-book-title {
color: #2b6cb0;
font-weight: 500;
}
.source-figure-label {
color: #276749;
font-weight: 600;
}
.source-figure-type {
color: #718096;
font-size: 0.72rem;
background: #e2e8f0;
border-radius: 3px;
padding: 0 0.3rem;
}
.source-page {
color: #718096;
}
.source-chunk { .source-chunk {
font-size: 0.78rem; font-size: 0.78rem;
color: #4a5568; color: #4a5568;
@@ -194,28 +298,28 @@ function formatTime(iso: string): string {
line-height: 1.5; line-height: 1.5;
} }
.source-chip { .source-caption {
display: inline-flex;
align-items: center;
gap: 0.25rem;
background: #ebf8ff;
border: 1px solid #bee3f8;
border-radius: 4px;
padding: 0.2rem 0.5rem;
font-size: 0.78rem; font-size: 0.78rem;
color: #4a5568;
font-style: italic;
} }
.source-book-icon { .source-figure-image {
font-size: 0.8rem; max-width: 100%;
} }
.source-book-title { .figure-img {
color: #2b6cb0; max-width: 100%;
font-weight: 500; max-height: 300px;
border-radius: 6px;
border: 1px solid #e2e8f0;
object-fit: contain;
} }
.source-page { .figure-missing {
color: #718096; font-size: 0.78rem;
color: #a0aec0;
font-style: italic;
} }
.message-timestamp { .message-timestamp {
+2
View File
@@ -3,6 +3,8 @@
interface ImportMetaEnv { interface ImportMetaEnv {
readonly VITE_API_URL: string readonly VITE_API_URL: string
readonly VITE_APP_PASSWORD: string readonly VITE_APP_PASSWORD: string
readonly VITE_UPLOAD_ENABLED: string
readonly VITE_DELETE_ENABLED: string
} }
interface ImportMeta { interface ImportMeta {
+6
View File
@@ -2,6 +2,7 @@ import { createRouter, createWebHistory } from 'vue-router'
import UploadView from '@/views/UploadView.vue' import UploadView from '@/views/UploadView.vue'
import TopicsView from '@/views/TopicsView.vue' import TopicsView from '@/views/TopicsView.vue'
import ChatView from '@/views/ChatView.vue' import ChatView from '@/views/ChatView.vue'
import BookReaderView from '@/views/BookReaderView.vue'
const router = createRouter({ const router = createRouter({
history: createWebHistory(import.meta.env.BASE_URL), history: createWebHistory(import.meta.env.BASE_URL),
@@ -20,6 +21,11 @@ const router = createRouter({
path: '/chat', path: '/chat',
name: 'chat', name: 'chat',
component: ChatView component: ChatView
},
{
path: '/books/:id/read',
name: 'book-reader',
component: BookReaderView
} }
] ]
}) })
+15 -1
View File
@@ -2,11 +2,25 @@ import { defineStore } from 'pinia'
import { ref } from 'vue' import { ref } from 'vue'
import { api } from '@/services/api' import { api } from '@/services/api'
export interface ChatSource {
type: 'TEXT' | 'FIGURE'
bookTitle: string
page: number | null
// TEXT-specific
chunkText?: string
// FIGURE-specific
figureId?: string
label?: string
caption?: string
figureType?: string
imageUrl?: string
}
export interface ChatMessage { export interface ChatMessage {
id: string id: string
role: 'USER' | 'ASSISTANT' role: 'USER' | 'ASSISTANT'
content: string content: string
sources: Array<{ bookTitle: string; page: number | null; chunkText?: string }> sources: ChatSource[]
createdAt: string createdAt: string
} }
+325
View File
@@ -0,0 +1,325 @@
<template>
<div class="reader-view">
<!-- Header -->
<div class="reader-header">
<router-link to="/" class="back-link"> Library</router-link>
<div class="reader-title">
<h1 class="book-title">{{ book?.title ?? 'Loading…' }}</h1>
</div>
<div class="page-nav">
<button class="nav-btn" :disabled="currentPage <= 1" @click="goTo(currentPage - 1)">&#8592;</button>
<form class="page-jump" @submit.prevent="onJump">
<input
v-model.number="jumpInput"
type="number"
:min="1"
:max="book?.pageCount ?? 1"
class="page-input"
/>
<span class="page-sep">/ {{ book?.pageCount ?? '…' }}</span>
</form>
<button class="nav-btn" :disabled="!book || currentPage >= book.pageCount!" @click="goTo(currentPage + 1)">&#8594;</button>
</div>
</div>
<!-- Content -->
<div class="reader-body">
<div v-if="loading" class="reader-loading">
<div class="spinner spinner-dark" style="width:28px;height:28px;margin:0 auto 0.75rem;"></div>
<p>Loading page {{ currentPage }}</p>
</div>
<div v-else-if="error" class="reader-error card">
<strong>Could not load page {{ currentPage }}</strong><br />
{{ error }}
</div>
<div v-else class="reader-content card">
<div class="markdown-body" v-html="renderedHtml"></div>
</div>
</div>
</div>
</template>
<script setup lang="ts">
import { ref, watch, onMounted } from 'vue'
import { useRoute } from 'vue-router'
import { api } from '@/services/api'
import { useBookStore } from '@/stores/bookStore'
import type { Book } from '@/stores/bookStore'
const route = useRoute()
const bookStore = useBookStore()
const bookId = route.params.id as string
const book = ref<Book | null>(null)
const currentPage = ref(1)
const jumpInput = ref(1)
const loading = ref(false)
const error = ref<string | null>(null)
const renderedHtml = ref('')
// Blob URLs created this session revoked on next page load
let activeBlobUrls: string[] = []
onMounted(async () => {
book.value = bookStore.books.find(b => b.id === bookId) ?? null
if (!book.value) {
try {
const res = await api.get<Book>(`/books/${bookId}`)
book.value = res.data
} catch {
error.value = 'Book not found.'
return
}
}
await loadPage(1)
})
watch(currentPage, (page) => {
jumpInput.value = page
loadPage(page)
})
async function goTo(page: number) {
if (!book.value) return
const clamped = Math.max(1, Math.min(page, book.value.pageCount ?? 1))
if (clamped !== currentPage.value) {
currentPage.value = clamped
}
}
function onJump() {
goTo(jumpInput.value)
}
async function loadPage(page: number) {
loading.value = true
error.value = null
renderedHtml.value = ''
// Revoke previous blob URLs to free memory
activeBlobUrls.forEach(u => URL.revokeObjectURL(u))
activeBlobUrls = []
try {
const res = await api.get<string>(`/books/${bookId}/pages/${page}/html`, {
headers: { Accept: 'text/html' },
responseType: 'text'
})
let html = await resolveImages(res.data)
renderedHtml.value = html
} catch (e: any) {
error.value = e.message ?? 'Failed to load page.'
} finally {
loading.value = false
}
}
/**
* Finds <img src="/api/v1/figures/..."> in the HTML, fetches each image
* through the authenticated axios instance, and replaces the src with a
* temporary blob URL so the browser can render it without re-authenticating.
*/
async function resolveImages(html: string): Promise<string> {
const srcPattern = /src="(\/api\/v1\/figures\/[^"]+)"/g
const matches = [...html.matchAll(srcPattern)]
if (matches.length === 0) return html
const unique = [...new Set(matches.map(m => m[1]))]
const blobMap: Record<string, string> = {}
await Promise.all(
unique.map(async (src) => {
try {
const res = await api.get(src.replace(/^\/api\/v1/, ''), { responseType: 'blob' })
const blobUrl = URL.createObjectURL(res.data)
activeBlobUrls.push(blobUrl)
blobMap[src] = blobUrl
} catch {
// leave original src browser will attempt (and likely fail silently)
}
})
)
return html.replace(/src="(\/api\/v1\/figures\/[^"]+)"/g, (_, src) =>
blobMap[src] ? `src="${blobMap[src]}"` : `src="${src}"`
)
}
</script>
<style scoped>
.reader-view {
display: flex;
flex-direction: column;
gap: 1rem;
max-width: 860px;
margin: 0 auto;
flex: 1;
min-height: 0;
}
.reader-header {
display: flex;
align-items: center;
gap: 1rem;
flex-wrap: wrap;
}
.back-link {
color: #3182ce;
text-decoration: none;
font-size: 0.9rem;
white-space: nowrap;
}
.back-link:hover { text-decoration: underline; }
.reader-title {
flex: 1;
min-width: 0;
}
.book-title {
font-size: 1.1rem;
font-weight: 600;
color: #1a365d;
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
}
.page-nav {
display: flex;
align-items: center;
gap: 0.5rem;
}
.nav-btn {
width: 2rem;
height: 2rem;
border: 1px solid #cbd5e0;
border-radius: 6px;
background: #fff;
cursor: pointer;
font-size: 1rem;
display: flex;
align-items: center;
justify-content: center;
transition: background 0.15s;
}
.nav-btn:hover:not(:disabled) { background: #ebf8ff; border-color: #3182ce; }
.nav-btn:disabled { opacity: 0.4; cursor: not-allowed; }
.page-jump {
display: flex;
align-items: center;
gap: 0.35rem;
}
.page-input {
width: 3.5rem;
text-align: center;
border: 1px solid #cbd5e0;
border-radius: 6px;
padding: 0.25rem 0.4rem;
font-size: 0.9rem;
color: #2d3748;
}
.page-input:focus { outline: none; border-color: #3182ce; }
.page-sep {
font-size: 0.85rem;
color: #718096;
white-space: nowrap;
}
.reader-body {
flex: 1;
min-height: 0;
display: flex;
flex-direction: column;
}
.reader-loading {
text-align: center;
padding: 3rem;
color: #718096;
}
.reader-error {
padding: 1.25rem;
background: #fff5f5;
border: 1px solid #fed7d7;
color: #742a2a;
border-radius: 8px;
}
.reader-content {
flex: 1;
min-height: 0;
overflow-y: auto;
padding: 2rem;
}
/* Markdown rendering */
.markdown-body {
font-size: 0.95rem;
line-height: 1.75;
color: #2d3748;
}
.markdown-body :deep(h1),
.markdown-body :deep(h2),
.markdown-body :deep(h3) {
color: #1a365d;
font-weight: 600;
margin: 1.5rem 0 0.75rem;
}
.markdown-body :deep(h2) { font-size: 1.15rem; border-bottom: 1px solid #e2e8f0; padding-bottom: 0.4rem; }
.markdown-body :deep(h3) { font-size: 1rem; }
.markdown-body :deep(p) { margin: 0.75rem 0; }
.markdown-body :deep(img) {
max-width: 100%;
border-radius: 6px;
display: block;
margin: 1rem auto;
box-shadow: 0 1px 4px rgba(0,0,0,0.12);
}
.markdown-body :deep(ul),
.markdown-body :deep(ol) {
padding-left: 1.5rem;
margin: 0.75rem 0;
}
.markdown-body :deep(code) {
background: #f7fafc;
border: 1px solid #e2e8f0;
border-radius: 3px;
padding: 0.1em 0.35em;
font-size: 0.88em;
}
.markdown-body :deep(blockquote) {
border-left: 3px solid #3182ce;
padding-left: 1rem;
color: #4a5568;
margin: 0.75rem 0;
}
.markdown-body :deep(table) {
width: 100%;
border-collapse: collapse;
font-size: 0.9em;
margin: 1rem 0;
}
.markdown-body :deep(th),
.markdown-body :deep(td) {
border: 1px solid #e2e8f0;
padding: 0.4rem 0.75rem;
text-align: left;
}
.markdown-body :deep(th) { background: #f7fafc; font-weight: 600; }
</style>
+6 -2
View File
@@ -1,10 +1,10 @@
<template> <template>
<div class="upload-view"> <div class="upload-view">
<h1 class="page-title">Book Library</h1> <h1 class="page-title">Book Library</h1>
<p class="page-subtitle">Upload medical textbooks (PDF) to build the knowledge base.</p> <p v-if="uploadEnabled" class="page-subtitle">Upload medical textbooks (PDF) to build the knowledge base.</p>
<!-- Upload Section --> <!-- Upload Section -->
<div class="upload-section card"> <div v-if="uploadEnabled" class="upload-section card">
<h2 class="section-title">Upload a Book</h2> <h2 class="section-title">Upload a Book</h2>
<div <div
@@ -87,6 +87,7 @@
:key="book.id" :key="book.id"
:book="book" :book="book"
:deleting="deletingId === book.id" :deleting="deletingId === book.id"
:delete-enabled="deleteEnabled"
@delete="handleDelete" @delete="handleDelete"
/> />
</div> </div>
@@ -99,6 +100,9 @@ import { ref, onMounted, onUnmounted, inject } from 'vue'
import { useBookStore } from '@/stores/bookStore' import { useBookStore } from '@/stores/bookStore'
import BookCard from '@/components/BookCard.vue' import BookCard from '@/components/BookCard.vue'
const uploadEnabled = import.meta.env.VITE_UPLOAD_ENABLED !== 'false'
const deleteEnabled = import.meta.env.VITE_DELETE_ENABLED !== 'false'
const bookStore = useBookStore() const bookStore = useBookStore()
const showToast = inject<(msg: string, type?: 'error' | 'success') => void>('showToast') const showToast = inject<(msg: string, type?: 'error' | 'success') => void>('showToast')
@@ -0,0 +1,73 @@
# Embedding & Retrieval Pipeline Checklist: Enhanced Embedding with Image Parsing and Metadata
**Purpose**: Author self-review of embedding pipeline and retrieval requirements quality — validates completeness, clarity, and measurability before implementation tasks are written
**Created**: 2026-04-03
**Feature**: [spec.md](../spec.md) | [research.md](../research.md) | [data-model.md](../data-model.md)
**Focus**: A (Embedding pipeline) + B (Retrieval & ranking) | Depth: Standard | Audience: Author
---
## Requirement Completeness — Embedding Pipeline
- [X] CHK001 - Is the definition of "inspect every page" complete — does the spec cover pages that have no extractable content layer (fully scanned/rasterised pages)? Yes [Completeness, Spec §FR-001, Assumption §6]
- [X] CHK002 - Does FR-002 define what "independently searchable" means in practice — specifically, is it clear that image chunks must be retrievable without a co-located text chunk? [Clarity, Spec §FR-002] - No image should be retrieved along linked text.
- [X] CHK003 - Is the minimum acceptable quality of the "descriptive textual representation" (FR-003) specified — e.g., must it include structural relationships, labelled regions, or clinical terms — or is any non-empty description sufficient? [Clarity, Spec §FR-003, Gap] - any non-empty description sufficient. Text just below the image should have the correct clinical term.
- [C] CHK004 - Are the caption-detection rules defined at spec level — specifically, what pattern or signal determines that a piece of text is a caption vs. body text adjacent to an image? [Clarity, Spec §FR-004, Gap] - We assume a text starting with Fig. follewed by number is a text description of a give image.
- [X] CHK005 - Does FR-004 specify what metadata is stored when a caption is absent — is the caption field omitted, left empty, or populated with a generated substitute? [Completeness, Spec §FR-004] - generated substitute
- [X] CHK006 - Is the "minimum meaningful-content threshold" (FR-007) quantified in the spec, or is it deferred entirely to implementation? The assumption section says "size threshold determined during implementation" — is this intentional and acceptable at the spec level? [Ambiguity, Spec §FR-007, Assumption §3] - Deferred to implementation
- [X] CHK007 - Does FR-008 specify the observable outcome of per-page image failures — specifically, is there a requirement that the book's processing status or error log is accessible to the user or admin after partial failure? [Completeness, Spec §FR-008, Gap] online logs
- [X] CHK008 - Is FR-010 ("MUST NOT degrade accuracy or completeness of text-only embedding") measurable — does the spec define a baseline or acceptance criterion against which degradation can be detected? [Measurability, Spec §FR-010, Gap] no definition
- [X] CHK009 - Are re-embedding requirements complete — does the spec cover what happens to in-progress queries and cached results while a book is being re-embedded? [Coverage, Assumption §8, Gap] - No need to take that into account.
---
## Requirement Completeness — Retrieval & Ranking
- [X] CHK010 - Does FR-006 define how image and text chunks are ranked relative to each other — is ranking unified (single score), or are the two modalities ranked independently with separate topK controls? [Clarity, Spec §FR-006, Gap] - independent separated topK
- [X] CHK011 - Is the relevance threshold for figure retrieval specified — i.e., at what similarity score (or other criterion) should a figure be excluded from results? [Clarity, Spec §FR-006, Gap] not specified
- [X] CHK012 - Are deduplication rules defined for the case where the same figure appears both in the semantic figure search and the chunk-to-figure reference lookup — which representation wins, or are both included? [Completeness, data-model.md §RetrievalResult, Gap] not specified
- [X] CHK013 - Is the requirement for parent section context expansion in the spec — specifically, is there a requirement that the LLM receives the full section text (not just the chunk) when a text chunk is retrieved? [Gap, research.md §Decision 1] - the LLM should receive the full section to have maximum context.
- [X] CHK014 - Does the spec define the required structure of the LLM prompt when both text context and figures are present — or is prompt design left entirely to implementation? [Completeness, Gap] - Left to implementation
- [X] CHK015 - Is SC-002 ("70% recall on image queries") sufficient as a measurability criterion — is the test set composition (10 queries) and evaluation method documented, or does it rely on an undefined manual process? [Measurability, Spec §SC-002] - Manual process.
---
## Scenario Coverage — Edge & Exception Cases
- [X] CHK016 - Does the spec address the scenario where a query is relevant to a book section that has figures but none of those figures rank above the retrieval threshold — is the expected fallback behaviour defined? [Coverage, Edge Case, Gap] - The figure should in this case be retrieved and shon to the user.
- [X] CHK017 - Is the scenario of a figure retrieved in search results but whose image file is missing from the file store covered — what should the system return to the user in that case? [Coverage, Exception Flow, Gap] - missing image error, shown in the front as a broken image link.
- [X] CHK018 - Are requirements defined for multi-image pages where images have conflicting captions or share a single composite caption — which image gets the caption, or is it duplicated? [Coverage, Spec §FR-004, Edge Case] - this case not exist.
---
## Consistency & Alignment
- [X] CHK019 - Are the metadata fields required by FR-004 and FR-005 fully consistent with the metadata schema defined in data-model.md — specifically, do the mandatory fields in the spec match the `type`, `section_id`, and `section_title` fields in the data model? [Consistency, Spec §FR-004, data-model.md §Vector Store Documents] - Left to implementation
- [X] CHK020 - Is SC-003 ("processing time ≤ 3× baseline") consistent with FR-003 — if description generation requires a vision model call per image, is the 3× cap realistic for a 500-page book with dense figures, and is this assumption documented? [Consistency, Spec §SC-003, Assumption §3, Gap] - not documented
- [X] CHK021 - Does the spec's description of citation display (FR-009) align with the `sources` format change documented in contracts/api.md — are the fields the spec says must be "distinct" actually represented distinctly in the API response? [Consistency, Spec §FR-009, contracts/api.md §4] - A section with image-source should be displayed in the front. Text source and image-source are distinct
---
## Notes
- Items marked `[Gap]` indicate requirements that appear absent or deferred; resolve before generating tasks
- Items marked `[Ambiguity]` require a clearer definition in the spec before implementation starts
- Items marked `[Consistency]` should be cross-checked between spec.md, data-model.md, and contracts/api.md
- Mark items `[x]` when resolved; add inline notes with the resolution for traceability
@@ -0,0 +1,34 @@
# Specification Quality Checklist: Enhanced Embedding with Image Parsing and Metadata
**Purpose**: Validate specification completeness and quality before proceeding to planning
**Created**: 2026-04-03
**Feature**: [spec.md](../spec.md)
## Content Quality
- [x] No implementation details (languages, frameworks, APIs)
- [x] Focused on user value and business needs
- [x] Written for non-technical stakeholders
- [x] All mandatory sections completed
## Requirement Completeness
- [x] No [NEEDS CLARIFICATION] markers remain
- [x] Requirements are testable and unambiguous
- [x] Success criteria are measurable
- [x] Success criteria are technology-agnostic (no implementation details)
- [x] All acceptance scenarios are defined
- [x] Edge cases are identified
- [x] Scope is clearly bounded
- [x] Dependencies and assumptions identified
## Feature Readiness
- [x] All functional requirements have clear acceptance criteria
- [x] User scenarios cover primary flows
- [x] Feature meets measurable outcomes defined in Success Criteria
- [x] No implementation details leak into specification
## Notes
- All items pass. Spec is ready for `/speckit.clarify` or `/speckit.plan`.
@@ -0,0 +1,172 @@
# API Contracts: Enhanced Embedding with Image Parsing and Metadata
**Branch**: `002-image-aware-embedding` | **Date**: 2026-04-03
**Base path**: `/api/v1`
**Auth**: HTTP Basic (existing)
---
## New / Changed Endpoints
### 1. Re-embed a book (new)
Triggers a full re-embedding of an already-processed book, replacing all existing chunks and
figures with the new image-aware pipeline output. Safe to call on books previously embedded
by feature 001.
```
POST /api/v1/books/{id}/reembed
```
**Path parameters**
| Parameter | Type | Description |
|-----------|------|-------------|
| `id` | UUID | Book ID |
**Response** `202 Accepted`
```json
{ "bookId": "uuid", "status": "PROCESSING" }
```
**Error responses**
| Status | Condition |
|--------|-----------|
| 404 | Book not found |
| 409 | Book already in PROCESSING state |
---
### 2. Get figures for a book (new)
Returns the list of extracted figures for a book, including their type, caption, and image URL.
Used by the frontend to display a figure gallery or inline figures in chat responses.
```
GET /api/v1/books/{id}/figures
```
**Path parameters**
| Parameter | Type | Description |
|-----------|------|-------------|
| `id` | UUID | Book ID |
**Response** `200 OK`
```json
[
{
"figureId": "youmans-7ed-fig-12-4",
"label": "Fig. 12-4",
"caption": "Coronal cross-section of the cavernous sinus showing cranial nerve relationships",
"figureType": "ANATOMICAL_DIAGRAM",
"page": 184,
"imageUrl": "/api/v1/figures/550e8400-e29b-41d4-a716-446655440000/youmans-7ed-fig-12-4.png",
"sectionId": "youmans-7ed-ch12-s2-3",
"sectionTitle": "Cavernous Sinus"
}
]
```
**Error responses**
| Status | Condition |
|--------|-----------|
| 404 | Book not found |
---
### 3. Serve figure image (new)
Serves the extracted figure image file. Mounted as a static resource from the file store.
```
GET /api/v1/figures/{bookId}/{filename}
```
**Path parameters**
| Parameter | Type | Description |
|-----------|------|-------------|
| `bookId` | UUID | Book ID |
| `filename` | string | Image filename (e.g. `youmans-7ed-fig-12-4.png`) |
**Response** `200 OK` — binary PNG
**Content-Type**: `image/png`
**Error responses**
| Status | Condition |
|--------|-----------|
| 404 | Image file not found |
---
### 4. Chat message response — extended source format (changed)
The existing `POST /api/v1/chat/sessions/{id}/messages` endpoint is unchanged in its request
format. The response `sources` field is extended to include figure references.
**Existing request** (unchanged):
```json
{ "content": "Describe the anatomy of the cavernous sinus" }
```
**Response** `200 OK` — extended `sources`:
```json
{
"id": "uuid",
"role": "ASSISTANT",
"content": "The cavernous sinus is ... [Fig. 12-4, p.184] ...",
"sources": [
{
"type": "TEXT",
"bookTitle": "Youmans and Winn Neurological Surgery, 7th Ed.",
"page": 184,
"chunkText": "The cavernous sinus contains ..."
},
{
"type": "FIGURE",
"bookTitle": "Youmans and Winn Neurological Surgery, 7th Ed.",
"page": 184,
"figureId": "youmans-7ed-fig-12-4",
"label": "Fig. 12-4",
"caption": "Coronal cross-section of the cavernous sinus ...",
"figureType": "ANATOMICAL_DIAGRAM",
"imageUrl": "/api/v1/figures/550e8400-e29b-41d4-a716-446655440000/youmans-7ed-fig-12-4.png"
}
],
"createdAt": "2026-04-03T12:00:00Z"
}
```
**Changed fields in `sources` array**:
| Field | Old | New |
|-------|-----|-----|
| `type` | absent | `"TEXT"` or `"FIGURE"` |
| `figureId` | absent | figure ID string (FIGURE type only) |
| `label` | absent | caption label (FIGURE type only) |
| `caption` | absent | full caption (FIGURE type only) |
| `figureType` | absent | enum name (FIGURE type only) |
| `imageUrl` | absent | image URL (FIGURE type only) |
---
## Unchanged Endpoints
All endpoints from feature 001 remain at their existing paths with no breaking changes:
- `POST /api/v1/books/upload`
- `GET /api/v1/books`
- `DELETE /api/v1/books/{id}`
- `GET /api/v1/topics`
- `GET /api/v1/topics/{id}/summary`
- `POST /api/v1/chat/sessions`
- `GET /api/v1/chat/sessions/{id}/messages`
- `DELETE /api/v1/chat/sessions/{id}`
@@ -0,0 +1,79 @@
# Internal Contract: DocumentAiPageParser → FigureExtractionService
**Branch**: `002-image-aware-embedding` | **Date**: 2026-04-04
**Type**: Internal Java DTO (not an HTTP contract)
---
## Purpose
`PageResult` is the internal data transfer object produced by `DocumentAiPageParser` for each
PDF page. It decouples the Google Document AI SDK types from the rest of the pipeline so that
`PdfStructureParser` can be replaced without cascading changes.
---
## Java Record
```java
package com.aiteacher.document;
import java.util.List;
/**
* Internal DTO produced by DocumentAiPageParser for one PDF page.
* Decouples the Document AI SDK types from downstream services.
*/
public record PageResult(
int pageNumber, // 1-based, matches Document.Page.getPageNumber()
String orderedText, // full page text in correct reading order (blocks joined by \n\n)
String headingTitle, // first HEADING block on page, or null
List<FigureBbox> figures // detected figure regions (may be empty)
) {
/**
* Normalized bounding box for a detected figure region.
* Coordinates are in the [0.0, 1.0] range relative to page dimensions.
*/
public record FigureBbox(
float x, // left edge (normalized)
float y, // top edge (normalized)
float width, // width (normalized)
float height, // height (normalized)
String nearestCaption // text of adjacent paragraph block, or null
) {}
}
```
---
## Production Rules
| Field | Rule |
|-------|------|
| `orderedText` | Concatenation of all `PARAGRAPH` and `HEADING_*` blocks, joined with `\n\n`. Tables are represented as tab-separated text. |
| `headingTitle` | First block whose `blockType` is `HEADING_1` through `HEADING_6`. `null` if no heading detected. |
| `figures` | One entry per `VisualElement` with `type == "figure"` and `confidence ≥ 0.5`. Sorted top-to-bottom by `y`. |
| `nearestCaption` | The `PARAGRAPH` block immediately following the figure bbox (by Y coordinate). May be `null` if no paragraph follows within 10% of page height. |
---
## Mapping from Document AI Proto
```
Document.Page.Block → orderedText (concatenated)
Document.Page.Block (HEADING_*) → headingTitle (first match)
Document.Page.VisualElement → FigureBbox
└─ layout.bounding_poly.normalized_vertices[0] → (x, y) top-left
└─ normalized_vertices[2] → (x+w, y+h) bottom-right
```
---
## Consumers
| Consumer | What It Uses |
|----------|-------------|
| `BookEmbeddingService` | `orderedText``SectionEntity.fullText`; `headingTitle``SectionEntity.title` |
| `FigureExtractionService` | `figures` list → renders page via PDFBox, crops each bbox to `BufferedImage` |
| `TextChunkingService` | Receives `SectionEntity` (indirectly uses `orderedText`) — **unchanged** |
@@ -0,0 +1,84 @@
# Internal Contract: MarkerPageParser → FigureExtractionService / BookEmbeddingService
**Branch**: `002-image-aware-embedding` | **Date**: 2026-04-04
**Type**: Internal Java DTO (not an HTTP contract)
---
## Purpose
`PageResult` is the internal data transfer object produced by `MarkerPageParser` for each
PDF page. It decouples the Marker HTTP API from the rest of the pipeline. Downstream consumers
(`BookEmbeddingService`, `FigureExtractionService`, `TextChunkingService`) are unaware of
Marker and depend only on this DTO.
---
## Java Record
```java
package com.aiteacher.document;
import java.util.List;
/**
* Internal DTO produced by MarkerPageParser for one PDF page.
* Decouples the Marker HTTP API from downstream services.
*/
public record PageResult(
int pageNumber, // 1-based, derived from Marker page block index
String orderedText, // full page text in correct reading order (blocks joined by \n\n)
String headingTitle, // first SectionHeader block on page, or null
List<FigureData> figures // extracted figure images (may be empty)
) {
/**
* A figure extracted from the page.
* Image bytes are PNG data decoded from the Marker JSON `images` map.
*/
public record FigureData(
byte[] imageBytes, // PNG image data (base64-decoded from Marker response)
String nearestCaption, // text of the adjacent Caption block, or null
String blockId // Marker block ID (e.g. "/page/0/Figure/2") for traceability
) {}
}
```
---
## Production Rules
| Field | Rule |
|-------|------|
| `pageNumber` | 1-based index derived from the Marker page block's position in the `children` array (index + 1). |
| `orderedText` | HTML-stripped text from all `Text`, `TextInlineMath`, `SectionHeader`, `ListItem`, and `Table` blocks, joined with `\n\n`. Marker already returns them in reading order. |
| `headingTitle` | Plain text of the first `SectionHeader` block on the page. `null` if no heading detected. |
| `figures` | One `FigureData` per `Figure` or `Picture` block that has a non-empty `images` entry. Blocks with no image data are skipped. |
| `imageBytes` | Base64-decoded bytes from `block.images[blockId]`. Marker returns PNG. |
| `nearestCaption` | Plain text of the first `Caption` block that is a sibling appearing immediately after the figure block. `null` if absent. |
---
## Mapping from Marker JSON
```
Marker JSON → PageResult
Page block ("/page/N/Page/M") → PageResult(pageNumber = N + 1)
SectionHeader child → headingTitle (first match, HTML-stripped)
Text / TextInlineMath children → orderedText (HTML-stripped, joined \n\n)
Figure / Picture child → FigureData
images[blockId] → FigureData.imageBytes (base64-decoded)
next Caption sibling → FigureData.nearestCaption (HTML-stripped)
blockId → FigureData.blockId
```
---
## Consumers
| Consumer | What It Uses |
|----------|-------------|
| `BookEmbeddingService` | `orderedText``SectionEntity.fullText`; `headingTitle``SectionEntity.title` |
| `FigureExtractionService` | `figures` list → decodes `imageBytes`, checks min size, saves to S3 |
| `TextChunkingService` | Receives `SectionEntity` (uses `orderedText` indirectly) — **unchanged** |
@@ -0,0 +1,305 @@
# Data Model: Enhanced Embedding with Image Parsing and Metadata
**Branch**: `002-image-aware-embedding` | **Date**: 2026-04-03
---
## Overview
Three storage tiers work in concert:
```
┌──────────────────────────────────────────────────────────────────┐
│ PDF Upload │
│ │ │
│ ▼ │
│ Parsing Pipeline │
│ │ │ │
│ ▼ ▼ │
│ Postgres (source of truth) pgvector (search index) │
│ - book - vector_store (text chunks) │
│ - chapter - vector_store (figure captions) │
│ - section (+ fullText) File Store (images) │
│ - figure (metadata) - /uploads/figures/{bookId}/*.png │
│ - chunk_figure_refs │
└──────────────────────────────────────────────────────────────────┘
```
---
## Postgres Schema
### Existing tables (unchanged)
- `book` — status, metadata, page count (V1)
- `chat_session`, `message` — conversation (V1)
- `vector_store` — managed by Spring AI pgvector starter (V2)
- `topic` — predefined topics (V3)
### New tables (Flyway V4)
```sql
-- V4: Document hierarchy
CREATE TABLE chapter (
id VARCHAR(200) PRIMARY KEY, -- "{bookId}-ch{N}"
book_id UUID NOT NULL REFERENCES book(id) ON DELETE CASCADE,
number INT NOT NULL,
title VARCHAR(500),
page_start INT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE section (
id VARCHAR(200) PRIMARY KEY, -- "{bookId}-ch{N}-s{X}-{Y}"
chapter_id VARCHAR(200) NOT NULL REFERENCES chapter(id) ON DELETE CASCADE,
book_id UUID NOT NULL REFERENCES book(id) ON DELETE CASCADE,
number VARCHAR(50), -- "2.3" or "12.2.3"
title VARCHAR(500),
page_start INT NOT NULL,
page_end INT NOT NULL,
full_text TEXT NOT NULL, -- NOT in vector store
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX idx_section_book ON section(book_id);
CREATE INDEX idx_section_chapter ON section(chapter_id);
```
### New tables (Flyway V5)
```sql
-- V5: Figures and chunk→figure links
CREATE TABLE figure (
id VARCHAR(200) PRIMARY KEY, -- "{bookId}-fig-{label}"
book_id UUID NOT NULL REFERENCES book(id) ON DELETE CASCADE,
section_id VARCHAR(200) REFERENCES section(id) ON DELETE SET NULL,
chapter_id VARCHAR(200) REFERENCES chapter(id) ON DELETE SET NULL,
label VARCHAR(100), -- "Fig. 12-4"
caption TEXT,
figure_type VARCHAR(50) NOT NULL, -- FigureType enum name
page INT NOT NULL,
image_path VARCHAR(1000) NOT NULL, -- relative path on disk
caption_embedding_id UUID, -- ID in vector_store
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE chunk_figure_ref (
chunk_id UUID NOT NULL, -- vector_store document ID
figure_id VARCHAR(200) NOT NULL REFERENCES figure(id) ON DELETE CASCADE,
mention_page INT,
PRIMARY KEY (chunk_id, figure_id)
);
CREATE INDEX idx_figure_book ON figure(book_id);
CREATE INDEX idx_cfr_chunk ON chunk_figure_ref(chunk_id);
```
---
## Java Domain Records
### Document hierarchy (new package `com.aiteacher.document`)
```java
// Root — in-memory only, not a JPA entity
public record BookNode(
String bookId,
String title,
String isbn,
String edition,
List<String> authors,
List<ChapterNode> chapters
) {}
// Chapter — maps to `chapter` table
public record ChapterNode(
String chapterId,
String bookId,
int number,
String title,
int pageStart,
List<SectionNode> sections
) {}
// Section — maps to `section` table; fullText stays in Postgres
public record SectionNode(
String sectionId,
String chapterId,
String bookId,
String number,
String title,
int pageStart,
int pageEnd,
String fullText,
List<TextChunkNode> chunks,
List<FigureNode> figures
) {}
// Text chunk — embedded into vector_store; references its parent section
public record TextChunkNode(
String chunkId, // UUID → becomes vector_store document ID
String sectionId,
String chapterId,
String bookId,
String text,
int chunkIndex,
int totalChunksInSection,
int pageStart,
int pageEnd,
Map<String, Object> metadata // flattened for Spring AI filtering
) {
public Map<String, Object> toMetadata() {
return Map.of(
"type", "TEXT",
"book_id", bookId,
"chapter_id", chapterId,
"section_id", sectionId,
"section_title", /* from parent SectionNode */,
"page_start", pageStart,
"page_end", pageEnd,
"chunk_index", chunkIndex,
"total_chunks", totalChunksInSection
);
}
}
// Figure — maps to `figure` table; caption embedded into vector_store
public record FigureNode(
String figureId,
String sectionId,
String chapterId,
String bookId,
String label, // "Fig. 12-4"
String caption,
FigureType type,
int page,
String imagePath, // relative: "figures/{bookId}/{figureId}.png"
UUID captionEmbeddingId // ID in vector_store
) {}
```
### Figure type enum
```java
public enum FigureType {
ANATOMICAL_DIAGRAM,
SURGICAL_PHOTOGRAPH,
MRI_CT_SCAN,
TABLE,
CHART,
INTRAOPERATIVE_IMAGE
}
```
Classification heuristic (applied to caption + surrounding text):
| Keyword(s) | FigureType |
|-----------|-----------|
| `MRI`, `CT`, `magnetic`, `resonance`, `tomography` | `MRI_CT_SCAN` |
| `intraoperative`, `intra-op` | `INTRAOPERATIVE_IMAGE` |
| `table`, `Table` (at line start) | `TABLE` |
| `chart`, `graph`, `histogram` | `CHART` |
| `photograph`, `photo` | `SURGICAL_PHOTOGRAPH` |
| (default) | `ANATOMICAL_DIAGRAM` |
### Chunkfigure join record
```java
// Maps to `chunk_figure_ref` table
public record ChunkFigureRef(
UUID chunkId,
String figureId,
int mentionPage
) {}
```
---
## Vector Store Documents
All documents in `vector_store` carry a `metadata` JSON column with a `type` field for filtering.
### Text chunk document
| Field | Value |
|-------|-------|
| `content` | chunk text (400600 tokens) |
| `metadata.type` | `"TEXT"` |
| `metadata.book_id` | book UUID |
| `metadata.book_title` | book title string |
| `metadata.chapter_id` | chapter ID string |
| `metadata.section_id` | section ID string |
| `metadata.section_title` | section title string |
| `metadata.page_start` | int |
| `metadata.page_end` | int |
| `metadata.chunk_index` | int (0-based) |
| `metadata.total_chunks` | int |
### Figure caption document
| Field | Value |
|-------|-------|
| `content` | vision-generated description + caption text |
| `metadata.type` | `"FIGURE"` |
| `metadata.book_id` | book UUID |
| `metadata.book_title` | book title string |
| `metadata.chapter_id` | chapter ID string |
| `metadata.section_id` | section ID string |
| `metadata.figure_id` | figure ID string |
| `metadata.figure_type` | enum name string |
| `metadata.image_path` | relative file path |
| `metadata.label` | caption label e.g. `"Fig. 12-4"` |
| `metadata.page` | int |
---
## File Store Layout
```
uploads/
└── figures/
└── {bookId}/
├── {figureId}.png
└── ...
```
- Base path configurable via `app.figure-storage.base-path` (default: `./uploads`)
- Files are served via `GET /api/v1/figures/{bookId}/{filename}` (static resource mapping)
- Gitignored; not version-controlled
---
## State Transitions
Book processing extends the existing `BookStatus` state machine:
```
PENDING → PROCESSING → READY
↘ FAILED
```
During `PROCESSING`:
1. Parse PDF structure → extract chapters/sections → persist to Postgres
2. Split sections into text chunks → embed → write to vector_store
3. Extract images per page → filter by min size → save PNG → generate vision description → embed caption → write figure to Postgres + vector_store
4. Write chunk_figure_refs for all detected figure references in text
Failure at step 3 (individual page) → log + skip that page's images; continue.
Failure at any other step → set `BookStatus.FAILED`.
---
## Retrieval Result Structure
```java
public record RetrievalResult(
List<SectionNode> parentSections, // expanded full-text context
List<Document> figureVectorHits, // semantic figure matches
List<FigureNode> linkedFigures // figures explicitly referenced in text chunks
) {}
```
The `NeurosurgeryRetriever` service deduplicates figures across both lists before passing
the result to the LLM prompt builder.
+85
View File
@@ -0,0 +1,85 @@
# Implementation Plan: Enhanced Embedding with Image Parsing and Metadata
**Branch**: `002-image-aware-embedding` | **Date**: 2026-04-04 | **Spec**: [spec.md](spec.md)
**Input**: Feature specification from `/specs/002-image-aware-embedding/spec.md`
## Summary
Enhance the PDF embedding pipeline to extract figures and generate AI descriptions for them,
making image content semantically searchable alongside text. PDF parsing and figure extraction
are delegated to a local **Marker** server (`http://localhost:8000/marker/upload`), which
returns reading-order text and pre-cropped figure images (base64) in a single JSON response,
eliminating the need for PDFBox column heuristics and figure bbox rendering.
## Technical Context
**Language/Version**: Java 25 (backend), TypeScript / Node 20 (frontend)
**Primary Dependencies**: Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API (embeddings +
GPT-4o vision), PDFBox 3.0.3 (via `spring-ai-pdf-document-reader` — retained transitively,
no longer used directly), Marker local HTTP API (`http://localhost:8000/marker/upload`)
**Storage**: PostgreSQL (JPA + Flyway), pgvector (Spring AI `VectorStore`), S3-compatible
object store (figure images via `FigureStorageService`)
**Testing**: Maven / JUnit 5 (`spring-boot-starter-test`)
**Target Platform**: Linux server
**Project Type**: Web application (backend API + frontend client)
**Performance Goals**: SC-003 — book processing time ≤ 3× text-only for ≤ 500 pages
**Constraints**: REST API only (Constitution III); Marker server must be running locally;
S3-compatible storage configured via env vars
**Scale/Scope**: POC — handful of books, <10 users
## Constitution Check
*GATE: Must pass before Phase 0 research. Re-checked after Phase 1 design.*
| Principle | Status | Notes |
|-----------|--------|-------|
| **I. KISS** | ✅ Justified | Marker replaces a bespoke PDFBox column heuristic + Google Cloud SDK with one HTTP call. Net complexity reduction vs. the Document AI approach. |
| **II. Easy to Change** | ✅ | `MarkerPageParser` is the only class that knows about Marker; swap the implementation to replace Marker with any other parser. `PageResult` DTO remains unchanged. |
| **III. Web-First** | ✅ | Internal pipeline change; no public API contract change. |
| **IV. Documentation** | ✅ | README must be updated to show Marker as a local external service. |
## Project Structure
### Documentation (this feature)
```text
specs/002-image-aware-embedding/
├── plan.md # This file
├── research.md # Phase 0 output
├── data-model.md # Phase 1 output
├── quickstart.md # Phase 1 output
├── contracts/
│ ├── api.md # HTTP API contracts (unchanged from initial plan)
│ └── marker-page-result.md # Internal DTO contract (MarkerPageParser → downstream)
└── tasks.md # Phase 2 output (/speckit.tasks — not created here)
```
### Source Code
```text
backend/
├── src/main/java/com/aiteacher/
│ ├── config/
│ │ └── MarkerConfig.java # NEW: RestClient bean + base-url property
│ ├── document/
│ │ ├── MarkerPageParser.java # NEW: replaces DocumentAiPageParser + PdfStructureParser
│ │ ├── PageResult.java # UPDATED: FigureBbox → FigureData (bytes not bbox)
│ │ ├── FigureExtractionService.java # UPDATED: no PDFBox render; decode bytes directly
│ │ ├── TextChunkingService.java # UNCHANGED
│ │ ├── VisionDescriptionService.java # UNCHANGED
│ │ └── [removed] DocumentAiPageParser.java
│ ├── book/
│ │ └── BookEmbeddingService.java # MINOR UPDATE: inject MarkerPageParser, drop DocumentAiPageParser
│ └── [removed] config/DocumentAiConfig.java
├── src/main/resources/
│ └── application.yaml # UPDATED: remove document-ai.*, add marker.base-url
└── pom.xml # UPDATED: remove google-cloud-document-ai
```
**Structure Decision**: Option 2 (backend + frontend) per constitution Technology Constraints.
Frontend changes are display-only (render figure citations inline).
## Complexity Tracking
> No constitution violations — Marker reduces complexity compared to the previous
> Google Document AI approach (fewer dependencies, no GCP credentials, no 15-page batching).
@@ -0,0 +1,121 @@
# Quickstart: Enhanced Embedding with Image Parsing and Metadata
**Branch**: `002-image-aware-embedding` | **Date**: 2026-04-04 (updated: Marker replaces Google Document AI)
---
## Prerequisites
- Docker Compose running (PostgreSQL + pgvector)
- OpenAI API key set as env var `OPENAI_API_KEY`
- Java 25 + Maven on PATH
- **Marker server running** on `http://localhost:8000` (see setup below)
- S3-compatible bucket configured (existing setup)
---
## Marker Server Setup (one-time)
Marker is a local Python service — no cloud credentials required.
```bash
# Install (Python 3.10+ required)
pip install marker-pdf
# Start the server on port 8000
marker_server --port 8000
```
The server is ready when you see:
```
INFO: Uvicorn running on http://0.0.0.0:8000
```
Keep the server running in the background (or use a process manager like `systemd` or `screen`).
---
## Backend Configuration
Add or update `backend/src/main/resources/application.yaml`:
```yaml
app:
figure-storage:
endpoint: https://your-s3-endpoint
region: your-region
bucket: ${S3_BUCKET:aiteacher}
access-key-id: ${S3_ACCESS_KEY_ID}
secret-access-key: ${S3_SECRET_ACCESS_KEY}
min-image-size-px: 100 # skip decorative images smaller than 100×100 px
marker:
base-url: ${MARKER_BASE_URL:http://localhost:8000}
embedding:
batch-size: 20
batch-delay-ms: 2000
```
No GCP credentials or project IDs are needed.
---
## Database Migration
Two Flyway migrations run automatically on startup:
- `V4__document_hierarchy.sql` — adds `chapter` and `section` tables
- `V5__figures_and_refs.sql` — adds `figure` and `chunk_figure_ref` tables
No manual DB setup needed.
---
## Re-embedding Existing Books
Books embedded by feature 001 (text-only) remain functional for text queries. To add image
support, trigger a re-embed:
```bash
curl -X POST http://localhost:8080/api/v1/books/{bookId}/reembed \
-u admin:password
```
The book transitions to `PROCESSING`, old chunks and figures are deleted, and the new
image-aware pipeline runs. Status can be polled via `GET /api/v1/books`.
---
## Verifying Image Extraction
1. Ensure Marker is running: `curl http://localhost:8000` should respond.
2. Upload a PDF with diagrams: `POST /api/v1/books/upload`
3. Wait for `status: "READY"` via `GET /api/v1/books`
4. List figures: `GET /api/v1/books/{id}/figures` — should return at least one entry per image page
5. Ask a diagram-specific question in chat — response `sources` should include a `type: "FIGURE"` entry
---
## Frontend: Rendering Inline Figures
The assistant message `content` field will contain figure references in the format
`[Fig. 12-4, p.184]`. The frontend should:
1. Parse `[Fig. X, p.N]` patterns in assistant message text
2. Look up the matching entry in `sources` where `type === "FIGURE"`
3. Render the figure inline using the `imageUrl` field
---
## Running Tests
```bash
cd backend
mvn test
```
Key new test classes:
- `MarkerPageParserTest` — unit tests for JSON parsing and block-to-PageResult mapping
- `FigureExtractionServiceTest` — unit tests for base64 decode, size filtering, classification
- `NeurosurgeryRetrieverTest` — unit tests for dual-search merge and deduplication
- `BookEmbeddingServiceIntegrationTest` — integration test: upload PDF with known figures,
verify figures appear in `GET /api/v1/books/{id}/figures`
+411
View File
@@ -0,0 +1,411 @@
# Research: Enhanced Embedding with Image Parsing and Metadata
**Branch**: `002-image-aware-embedding` | **Date**: 2026-04-04 (updated: Marker replaces Google Document AI)
This document resolves all technical unknowns identified during planning. Decisions 110 cover
the core pipeline. The **Marker Study** section at the bottom explains why Marker was chosen
over Google Document AI to drive PDF parsing and figure extraction.
---
## Decision 1: Document Hierarchy Model
**Decision**: Adopt a four-level hierarchy — `BookNode``ChapterNode``SectionNode`
`TextChunkNode` + `FigureNode`. The `SectionNode` is the pivotal unit: it holds the full section
text in Postgres and is used for parent-child context expansion at retrieval time.
**Rationale**: A flat page-per-document model (current implementation) loses structural context.
When a user asks a multi-faceted clinical question, the LLM needs the surrounding section text,
not just the matching fragment. Parent-child retrieval — where chunks point to their parent
section — is the established pattern for RAG precision. The hierarchy also makes figure-to-section
association explicit and queryable.
**Alternatives considered**:
- Keep flat page model, add metadata only → rejected: insufficient for precise citation and
context expansion
- Chapter-level retrieval (coarser than section) → rejected: too much irrelevant context sent
to LLM; cost and latency increase
---
## Decision 2: Document Parsing Strategy
**Decision**: Use **Marker** (local HTTP server, `http://localhost:8000/marker/upload`) as the
single entry point for PDF parsing. A single `POST` with `output_format=json` returns:
- Reading-order text blocks (headings, paragraphs) — no column-split heuristic needed
- Pre-cropped figure images as base64-encoded PNG in the `images` map of each `Figure` block
- Table, equation, and code blocks as structured HTML
`MarkerPageParser` translates the Marker JSON response into `List<PageResult>`, which is the
same internal DTO used by the rest of the pipeline.
**Rationale**: Marker handles column reordering, scanned-page OCR, and figure cropping in one
call, eliminating the PDFBox column heuristic (`PdfStructureParser`) and the PDFBox
render+crop loop in `FigureExtractionService`. Net result: fewer classes, no cloud dependency,
no GCP credentials.
**Alternatives considered**:
- PDFBox column heuristic (previous approach) → rejected: 50/50 split fails on asymmetric
columns and scanned pages
- Google Document AI Layout Parser → rejected: adds GCP credentials, per-page billing, 15-page
batch limit, and still requires PDFBox to render+crop figure regions from bounding boxes.
See Marker Study below for detailed comparison.
- Screenshot each page + OCR → far slower; loses digital text quality
---
## Decision 3: Figure Content Representation
**Decision**: Generate a textual description of each extracted image using the OpenAI vision
model (GPT-4o). This description becomes the `content` field of the figure's vector store
document. The figure caption (parsed from the surrounding text) is also included to maximise
retrieval signal.
**Rationale**: Caption-only embedding would miss figures with no caption or with sparse labels.
Vision-generated descriptions produce richer semantic content (anatomy terms, structural
relationships) that matches clinical queries. The OpenAI client already in use supports image
inputs; no additional dependency is required.
**Alternatives considered**:
- Caption-only embedding → insufficient when captions are absent or terse (common in textbooks)
- Local vision model (LLaVA) → requires self-hosting; out of scope for POC
- OCR only → extracts text visible in image but misses non-text visual content (diagrams, MRI)
---
## Decision 4: Dual Vector Search
**Decision**: At query time, run two parallel similarity searches:
1. Text chunk search (filtered by `type = "TEXT"` and `book_id`)
2. Figure caption search (filtered by `type = "FIGURE"` and `book_id`)
Results are merged and deduplicated. The LLM prompt receives the expanded parent section text
plus a structured figure reference list.
**Rationale**: A single search would rank text and figures against each other; figures with
terse captions would systematically lose to text chunks. Separate searches with independent
`topK` allow tuning each modality independently.
**Alternatives considered**:
- Single search, filter by relevance score → figure captions score lower than text; figures
are systematically under-retrieved
- Post-process text results to look up linked figures only → misses figures that are relevant
to the query but not explicitly referenced in the retrieved text chunks
---
## Decision 5: Chunk-to-Figure Linking
**Decision**: During text parsing, whenever a pattern matching `Fig.\s+\d+[\-\.]\d+` or
`Figure\s+\d+[\-\.]\d+` is found in a chunk, insert a row into the `chunk_figure_refs` table
linking `chunkId``figureId`. At retrieval time, after text chunks are retrieved, their
associated figures are fetched from this table and added to the LLM prompt.
**Rationale**: Explicit linking ensures that when a text chunk is retrieved, its referenced
figures are always surfaced — even if the figure's caption did not score highly in the vector
search. This is the higher-recall path; dual search (Decision 4) is the higher-precision path.
**Alternatives considered**:
- Rely entirely on dual vector search → may miss figures referenced in retrieved text but
scoring below the topK threshold in the figure search
---
## Decision 6: Image Storage
**Decision**: Marker returns figure images as base64-encoded PNG bytes in the JSON response.
`FigureExtractionService` decodes these bytes and passes them to `FigureStorageService`, which
persists them to an S3-compatible bucket (`${app.figure-storage.bucket}`). The image path/URL
is stored in `figure.image_path` in Postgres.
The `FigureStorageService` interface is unchanged; only the caller changes (from PDFBox crop
to base64 decode).
**Rationale**: Marker's pre-cropped images remove the need for PDFBox rendering.
`FigureStorageService` interface boundary satisfies Constitution Principle II (Easy to Change).
**Alternatives considered**:
- Store base64 in Postgres JSONB → bloats DB; complicates backup; query performance degrades
---
## Decision 7: Figure Type Classification
**Decision**: Use the enum `FigureType { ANATOMICAL_DIAGRAM, SURGICAL_PHOTOGRAPH, MRI_CT_SCAN,
TABLE, CHART, INTRAOPERATIVE_IMAGE }`. Classification is derived from:
1. Caption keywords ("MRI", "CT", "Fig.", "Table") — heuristic, no model needed
2. Marker `block_type` hint (`"Table"` → TABLE, `"Figure"` / `"Picture"` → ANATOMICAL_DIAGRAM default)
3. Fall back to `ANATOMICAL_DIAGRAM` if unclassifiable
**Rationale**: Allows the frontend to render different icon/label per type (e.g., "MRI" badge).
Heuristic classification avoids a separate model call per image at extraction time.
**Alternatives considered**:
- Vision model classification → accurate but adds latency and cost per figure; deferrable
- Single `FIGURE` type → loses citation granularity required by spec FR-004
---
## Decision 8: Metadata Schema for Vector Store Documents
**Decision**: All vector store documents carry a flat `Map<String, Object>` metadata for Spring
AI filtering. Schema:
| Field | Text Chunk | Figure Chunk |
|-------|-----------|-------------|
| `type` | `"TEXT"` | `"FIGURE"` |
| `book_id` | ✓ | ✓ |
| `book_title` | ✓ | ✓ |
| `chapter_id` | ✓ | ✓ |
| `section_id` | ✓ | ✓ |
| `section_title` | ✓ | ✓ |
| `page_start` | ✓ | — |
| `page_end` | ✓ | — |
| `chunk_index` | ✓ | — |
| `total_chunks` | ✓ | — |
| `figure_id` | — | ✓ |
| `figure_type` | — | ✓ |
| `image_path` | — | ✓ |
| `label` | — | ✓ |
| `page` | — | ✓ |
**Rationale**: Flat map is required by Spring AI `FilterExpressionBuilder`. Separation by `type`
allows independent filtering in dual search.
---
## Decision 9: Re-embedding Existing Books
**Decision**: Books already processed under feature 001 (text-only) are NOT automatically
re-embedded. An explicit re-embed action is exposed via `POST /api/v1/books/{id}/reembed`
(admin-triggered). The existing chunks remain valid for text queries until re-embedding completes.
**Rationale**: Automatic re-embedding on deploy would block the system and risk data loss if
the process fails mid-way. An explicit, idempotent trigger is safer and more observable.
---
## Decision 10: Minimum Image Size Threshold
**Decision**: Images smaller than 100×100 pixels are discarded and no chunk is created. Marker
returns PNG bytes; `FigureExtractionService` decodes to `BufferedImage` solely to check
dimensions. This threshold filters out decorative elements without a classification model.
**Rationale**: Neurosurgery textbook diagrams and MRI scans are never smaller than 100×100 px.
The threshold is configurable via `app.figure-storage.min-image-size-px`.
**Alternatives considered**:
- No threshold → decorative icons pollute the figure index
- ML-based classification → accurate but adds model dependency; not needed at POC scale
---
# Marker Study — Why Marker Replaces Google Document AI
*Added 2026-04-04.*
## What Marker Offers
Marker is an open-source, locally-runnable PDF-to-structured-content converter that uses a
pipeline of deep-learning models (surya for OCR + layout detection, texify for equations).
Key capabilities relevant to this project:
| Capability | Marker | Google Document AI |
|-----------|--------|--------------------|
| Multi-column reading order | ✅ | ✅ |
| OCR on scanned pages | ✅ | ✅ |
| Figure detection | ✅ returns pre-cropped images | ⚠️ returns bbox only; PDFBox still needed |
| Table extraction | ✅ HTML tables | ✅ |
| JSON output with image bytes | ✅ base64 in `images` map | ❌ |
| No cloud credentials | ✅ | ❌ GCP service account required |
| No per-page billing | ✅ | ❌ ~$10/1,000 pages |
| Batch size limits | None (local) | 15 pages / 20 MB per sync call |
| Setup | `pip install marker-pdf && marker_server` | GCP project + processor + IAM |
---
## Does Marker Solve the Current Pain Points?
### Pain Point 1: Naive 50/50 Column Split
**Answer: Yes, Marker fixes this completely.**
`PdfStructureParser.extractPageText()` splits pages at the horizontal midpoint with a 20%
threshold. This fails on asymmetric columns and scanned pages. Marker's surya layout model
returns blocks in natural reading order — no heuristic needed.
### Pain Point 2: Figure Detection Misses Rasterized Figures
**Answer: Yes, Marker fixes this for most cases.**
`FigureExtractionService` previously iterated PDF XObjects (only finds embedded XObject images,
misses rasterized figures and vector-path drawings). Marker's layout model detects visual
elements by type and returns the cropped image bytes directly — no PDFBox page rendering needed.
### Pain Point 3: OCR on Scanned Pages
**Answer: Yes, Marker handles scanned pages transparently via surya OCR.**
### Pain Point 4: Caption Detection
**Answer: Improved — Marker groups caption blocks with their figure block.**
The `block_type = "Caption"` block appears as a sibling or child adjacent to the `"Figure"`
block in the Marker JSON, making caption association structural rather than regex-based.
---
## Marker API Integration
### Local Server Setup
```bash
pip install marker-pdf
marker_server --port 8000
```
The server exposes `POST /marker/upload` (the user's configured endpoint).
### Request
```
POST http://localhost:8000/marker/upload
Content-Type: multipart/form-data
file=@document.pdf
output_format=json
```
### Response (abbreviated)
```json
{
"output_format": "json",
"output": {
"block_type": "Document",
"children": [
{
"block_type": "Page",
"id": "/page/0/Page/0",
"children": [
{
"block_type": "SectionHeader",
"id": "/page/0/SectionHeader/0",
"html": "<h1>Cavernous Sinus Anatomy</h1>"
},
{
"block_type": "Text",
"id": "/page/0/Text/1",
"html": "<p>The cavernous sinus contains...</p>"
},
{
"block_type": "Figure",
"id": "/page/0/Figure/2",
"html": "<figure><img src='/page/0/Figure/2'/></figure>",
"images": {
"/page/0/Figure/2": "iVBORw0KGgo..."
}
},
{
"block_type": "Caption",
"id": "/page/0/Caption/3",
"html": "<p>Fig. 12-4. Coronal cross-section...</p>"
}
]
}
],
"metadata": { "page_stats": [...] }
}
}
```
### Java Integration Pattern
```java
// MarkerPageParser — core call
MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
body.add("file", new FileSystemResource(pdfPath));
body.add("output_format", "json");
JsonNode response = restClient.post()
.uri(baseUrl + "/marker/upload")
.contentType(MediaType.MULTIPART_FORM_DATA)
.body(body)
.retrieve()
.body(JsonNode.class);
JsonNode document = response.get("output");
```
### Mapping Marker Blocks to PageResult
```
Page block (id "/page/N/Page/M") → PageResult(pageNumber = N+1)
SectionHeader children → headingTitle (first match)
Text, TextInlineMath children → orderedText (HTML stripped, joined \n\n)
Figure children with images map → FigureData(imageBytes = base64decode(images[id]))
Caption sibling of Figure → FigureData.nearestCaption
```
---
## Architecture Change
```
Before (Document AI — removed):
DocumentAiPageParser
→ Google Document AI API (GCP, 15-page batches, credentials)
→ returns text blocks + figure bboxes
PdfStructureParser (PDFBox column heuristic)
FigureExtractionService
→ renders page via PDFBox at 150 DPI
→ crops bbox region
After (Marker):
MarkerPageParser
→ POST PDF to http://localhost:8000/marker/upload (output_format=json)
→ returns text blocks (correct reading order) + Figure blocks with base64 images
→ produces List<PageResult> (same DTO, FigureData carries bytes not bbox)
FigureExtractionService (simplified)
→ base64-decodes image bytes from PageResult.FigureData
→ checks min size (ImageIO.read → getWidth/getHeight)
→ saves to S3 via FigureStorageService (UNCHANGED)
VisionDescriptionService (UNCHANGED)
BookEmbeddingService orchestration (MINOR: inject MarkerPageParser)
```
**What is removed**:
- `DocumentAiPageParser` — replaced by `MarkerPageParser`
- `DocumentAiConfig` — replaced by `MarkerConfig`
- `PdfStructureParser` — Marker handles reading order
- `google-cloud-document-ai` Maven dependency
- `app.document-ai.*` configuration properties
**What stays the same**:
- `PageResult` DTO structure (fields renamed, not restructured)
- `FigureExtractionService` public interface
- `TextChunkingService`, `VisionDescriptionService`, `BookEmbeddingService` orchestration
- All JPA entities, repositories, vector store, S3 storage
---
## Constitution Compliance
| Principle | Assessment |
|-----------|------------|
| **I. KISS** | ✅ Simpler than Document AI — one HTTP call replaces GCP SDK + PDFBox render loop. No new dependency beyond an HTTP client (Spring RestClient, already available). |
| **II. Easy to Change** | ✅ `MarkerPageParser` is the only Marker-aware class. Swap it to use any other parser. `PageResult` DTO unchanged in contract. |
| **III. Web-First** | ✅ Internal pipeline change; no API contract change. |
| **IV. Documentation** | ✅ README must show Marker as a local external service dependency. |
---
## Risks & Mitigations
| Risk | Likelihood | Mitigation |
|------|-----------|------------|
| Marker server not running when book is uploaded | Medium | `BookEmbeddingService` catches exception from `MarkerPageParser`, marks book as `FAILED`, logs full error. |
| Marker misses some figures (complex PDFs) | Medium | `app.figure-storage.min-image-size-px` threshold can be tuned. Add fallback: if Marker returns 0 figures for a page with known images, log a warning. |
| SC-003 (≤ 3× processing time) violated | Low | Marker runs locally (no network latency to cloud). Benchmark with a real 500-page book early. |
| Large PDF upload to Marker (>100MB) | Low | Marker server handles the full file; no batching needed. Multipart upload limit configurable. |
| Marker image quality vs PDFBox crop | Low | Marker crops at native resolution; quality is equivalent or better than 150 DPI PDFBox render. |
+176
View File
@@ -0,0 +1,176 @@
# Feature Specification: Enhanced Embedding with Image Parsing and Metadata
**Feature Branch**: `002-image-aware-embedding`
**Created**: 2026-04-03
**Status**: Draft
**Input**: User description: "I want to enhance the embedding process. I want also parse image from each pages if any and add proper metadata so that it can match the retrieved chunk/vector that match what user are querying."
## User Scenarios & Testing *(mandatory)*
### User Story 1 - Image Content Surfaced in Query Results (Priority: P1)
A neurosurgeon asks a question in the chat (e.g., "Show me the anatomy of the Circle of Willis")
that is best answered by a diagram or figure in an uploaded book. The system retrieves the image
content — its description and surrounding context — and uses it to construct a grounded answer,
citing the page and book where the image appeared.
**Why this priority**: This is the direct, user-visible payoff of the feature. Without it, the
enhancement has no observable benefit. All other stories support this outcome.
**Independent Test**: Upload a book containing a labelled anatomical diagram. Ask a query whose
answer is conveyed by that diagram (not in the surrounding text). Confirm the system returns an
answer that references the diagram's content and cites the correct book and page.
**Acceptance Scenarios**:
1. **Given** a book with an anatomical diagram on page 42, **When** a user asks a question whose
answer is only depicted in that diagram, **Then** the system returns a response that draws on
the diagram's content and cites "Page 42, [Book Title]".
2. **Given** a page with both text and an image, **When** the system retrieves that page's content,
**Then** the image-derived content and the surrounding text are each independently retrievable
and independently citable.
3. **Given** a query that has no relevant image in any uploaded book, **When** the system searches,
**Then** it does not fabricate image-derived content and falls back to text-only results (or
states no relevant content was found).
---
### User Story 2 - All Pages Scanned for Images During Embedding (Priority: P1)
When a book is uploaded and processed, every page is inspected for images. Any image found is
extracted and represented as a searchable content chunk enriched with metadata (page number,
book title, position on page, caption if present). Pages without images are processed as
text-only chunks, unchanged from the existing behaviour.
**Why this priority**: This is the prerequisite for User Story 1. Without systematic per-page
image detection, image content cannot be retrieved.
**Independent Test**: Upload a book whose pages include a mix of text-only and image-containing
pages. After processing completes, verify that chunks exist for each image page and that each
image chunk carries the correct metadata (page number, source book, caption).
**Acceptance Scenarios**:
1. **Given** a book being processed, **When** the embedding pipeline runs, **Then** every page
is evaluated for images and each detected image generates at least one content chunk.
2. **Given** an image with a caption or label, **When** the chunk is created, **Then** the
caption or label text is included in the chunk's content and metadata.
3. **Given** a page with multiple images, **When** processing completes, **Then** each image is
represented as a separate chunk with its own metadata, not merged into a single chunk.
4. **Given** a page with no images, **When** processing completes, **Then** no image chunk is
created for that page and text processing is unaffected.
---
### User Story 3 - Rich Metadata Enables Precise Source Attribution (Priority: P2)
When the system returns a result based on image content, the user can see exactly where that
image appeared: which book, which page, and what type of content (diagram, table, photograph,
etc.). This gives the user confidence in the source and lets them locate the original image
in their physical or digital copy of the book.
**Why this priority**: Metadata quality directly impacts user trust. Neurosurgeons require
traceable, citable evidence. Richer metadata also improves retrieval accuracy by giving the
search engine more signals to match against a query.
**Independent Test**: Retrieve a result sourced from an image chunk. Inspect the displayed
citation and verify it includes: book title, page number, content type (e.g., "diagram"),
and caption (if present in the original).
**Acceptance Scenarios**:
1. **Given** a retrieved image chunk, **When** the system displays the source citation,
**Then** the citation includes at minimum: book title, page number, and a content-type
label (e.g., diagram, table, figure).
2. **Given** an image chunk with a detected caption, **When** the citation is displayed,
**Then** the caption text is shown alongside the other metadata fields.
3. **Given** a topic summary that draws on both text and image chunks, **When** the user
inspects citations, **Then** image-sourced and text-sourced claims are distinguishable
from each other.
---
### Edge Cases
- What happens when an image is too small to contain meaningful content (e.g., a decorative
bullet icon or a publisher logo)?
- How does the system handle a page that is entirely an image (scanned page with no digital text)?
- What if an image spans multiple pages (e.g., a fold-out diagram)?
- How does the system behave when an image has no caption and its surrounding text provides
no useful context?
- What happens if image processing fails for a specific page — does it abort the whole book
or continue with the remaining pages?
## Requirements *(mandatory)*
### Functional Requirements
- **FR-001**: System MUST inspect every page of an uploaded book for the presence of images
during the embedding process.
- **FR-002**: System MUST extract each detected image and create a dedicated, independently
searchable content chunk for it.
- **FR-003**: System MUST generate a descriptive textual representation of each extracted
image so its content is semantically searchable by the retrieval system.
- **FR-004**: System MUST associate the following metadata with every image chunk: book title,
page number, content type (e.g., diagram, table, figure, photograph), and caption text
(where present).
- **FR-005**: System MUST include the same base metadata (book title, page number) on text
chunks so that all retrieved content — image or text — carries consistent, comparable
source attribution.
- **FR-006**: System MUST treat image chunks as first-class retrievable units: they must be
ranked and returned alongside text chunks when they are relevant to a user query.
- **FR-007**: System MUST skip images that fall below a minimum meaningful-content threshold
(e.g., decorative icons, page separators) and MUST NOT create chunks for them.
- **FR-008**: If image processing fails for a specific page, the system MUST log the failure,
skip that page's image, and continue processing the remaining pages and text content of
the book.
- **FR-009**: System MUST display image-sourced content citations distinctly from text-sourced
citations so users can identify when a result originates from a visual element.
- **FR-010**: Processing a book that contains images MUST NOT degrade the accuracy or
completeness of the existing text-only embedding for that book.
### Key Entities
- **Image Chunk**: A searchable content unit derived from a page image. Attributes: generated
description, source book title, page number, content type, caption (optional), embedding vector.
- **Text Chunk**: Existing unit; extended to carry explicit metadata: source book title,
page number, section heading (if detectable), content type ("text").
- **Chunk Metadata**: Structured attributes attached to every chunk regardless of type,
enabling consistent filtering and citation. Mandatory fields: book title, page number,
content type. Optional fields: caption, section heading.
## Success Criteria *(mandatory)*
### Measurable Outcomes
- **SC-001**: At least 90% of pages containing images in a test book result in a retrievable
image chunk after processing completes.
- **SC-002**: A controlled set of 10 queries whose answers are conveyed by diagrams in an
uploaded book returns at least 7 correct image-sourced answers (70% recall on image queries).
- **SC-003**: Embedding processing time for a book with images increases by no more than 3×
compared to processing the same book as text-only, for books up to 500 pages.
- **SC-004**: Every retrieved result — text or image — includes a citation that identifies
at minimum the source book title and page number, with 100% coverage across a test result set.
- **SC-005**: In a user evaluation with 5 representative queries that previously returned
no useful results (because the answer was only in a diagram), at least 4 now return a
useful, grounded answer.
## Assumptions
- Books are still uploaded exclusively as PDFs; image parsing applies to PDF pages only.
- The platform already has a working text-only embedding pipeline (from feature 001); this
feature enhances it without replacing or rewriting the text processing logic.
- Images worth processing are those that occupy a meaningful portion of the page; small
decorative or structural images (logos, dividers, icons) are excluded based on a size
threshold determined during implementation.
- The descriptive representation of an image (FR-003) is generated at embedding time, not
at query time; query latency is not affected by image interpretation.
- The shared global book library model from feature 001 is retained; image chunks from a
processed book are available to all users immediately upon completion.
- Scanned pages (fully rasterised pages with no digital text layer) are treated as a single
full-page image; the system attempts to extract content from them but does not guarantee
the same fidelity as pages with digital text.
- Per-chunk metadata is stored alongside the vector so it can be used for both retrieval
filtering and source citation display without a separate lookup.
- Books already processed under feature 001 (text-only) are not automatically re-processed;
re-embedding must be triggered explicitly by the user or an administrator.
+169
View File
@@ -0,0 +1,169 @@
# Tasks: Enhanced Embedding with Image Parsing and Metadata
**Input**: Design documents from `/specs/002-image-aware-embedding/`
**Prerequisites**: plan.md ✓ | spec.md ✓ | research.md ✓ | data-model.md ✓ | contracts/ ✓
**Organization**: Tasks grouped by user story to enable independent implementation and testing.
## Format: `[ID] [P?] [Story] Description`
- **[P]**: Can run in parallel (different files, no shared dependencies)
- **[US1/US2/US3]**: Which user story this task belongs to
---
## Phase 1: Setup (Shared Infrastructure)
**Purpose**: Database migrations and configuration that establish the foundation for all new code
- [X] T001 Create Flyway migration `V4__document_hierarchy.sql` — add `chapter` and `section` tables per data-model.md §Postgres Schema in `backend/src/main/resources/db/migration/V4__document_hierarchy.sql`
- [X] T002 Create Flyway migration `V5__figures_and_refs.sql` — add `figure` and `chunk_figure_ref` tables per data-model.md §Postgres Schema in `backend/src/main/resources/db/migration/V5__figures_and_refs.sql`
- [X] T003 Add figure-storage configuration keys to `backend/src/main/resources/application.properties`: `app.figure-storage.base-path=./uploads` and `app.figure-storage.min-image-size-px=100`
- [X] T004 Add `uploads/` directory to `.gitignore` at repo root; create `uploads/figures/.gitkeep` to preserve directory structure
---
## Phase 2: Foundational (Blocking Prerequisites)
**Purpose**: Core types and infrastructure that ALL user stories depend on — nothing in Phase 3+ can start until this phase is complete
**⚠️ CRITICAL**: No user story work can begin until this phase is complete
- [X] T005 [P] Create `FigureType` enum in `backend/src/main/java/com/aiteacher/document/FigureType.java` — values: `ANATOMICAL_DIAGRAM`, `SURGICAL_PHOTOGRAPH`, `MRI_CT_SCAN`, `TABLE`, `CHART`, `INTRAOPERATIVE_IMAGE`
- [X] T006 [P] Create `FigureStorageService` interface in `backend/src/main/java/com/aiteacher/figure/FigureStorageService.java` — declare `Path save(UUID bookId, String figureId, BufferedImage image)`, `Path resolve(UUID bookId, String filename)`, and `void delete(UUID bookId)`
- [X] T007 Create `LocalFigureStorageService` implementation in `backend/src/main/java/com/aiteacher/figure/LocalFigureStorageService.java` — writes PNG files under `${app.figure-storage.base-path}/figures/{bookId}/`; implements `FigureStorageService`; depends on T006
- [X] T008 Create `FigureStorageConfig` bean in `backend/src/main/java/com/aiteacher/config/FigureStorageConfig.java` — reads `app.figure-storage.base-path` and `app.figure-storage.min-image-size-px` as `@ConfigurationProperties`; registers `LocalFigureStorageService` as `@Bean`; adds `ResourceHandler` mapping `GET /api/v1/figures/**` to the base-path directory
- [X] T009 [P] Create `ChapterEntity` JPA entity and `ChapterRepository` in `backend/src/main/java/com/aiteacher/document/``@Entity(name="chapter")`, fields: `id` (String PK), `bookId` (UUID FK → book), `number` (int), `title` (String), `pageStart` (int), `createdAt` (Instant); `ChapterRepository extends JpaRepository<ChapterEntity, String>`
- [X] T010 [P] Create `SectionEntity` JPA entity and `SectionRepository` in `backend/src/main/java/com/aiteacher/document/``@Entity(name="section")`, fields: `id` (String PK), `chapterId` (String FK → chapter), `bookId` (UUID FK → book), `number` (String), `title` (String), `pageStart`/`pageEnd` (int), `fullText` (TEXT column), `createdAt` (Instant); `SectionRepository extends JpaRepository<SectionEntity, String>` with `findAllByBookId(UUID)`
- [X] T011 [P] Create `FigureEntity` JPA entity and `FigureRepository` in `backend/src/main/java/com/aiteacher/document/``@Entity(name="figure")`, fields: `id` (String PK), `bookId` (UUID), `sectionId` (String, nullable), `chapterId` (String, nullable), `label` (String), `caption` (TEXT), `figureType` (`@Enumerated` FigureType), `page` (int), `imagePath` (String), `captionEmbeddingId` (UUID, nullable), `createdAt` (Instant); `FigureRepository` with `findAllByBookId(UUID)`, `deleteAllByBookId(UUID)`
- [X] T012 Create `ChunkFigureRefEntity` JPA entity and `ChunkFigureRefRepository` in `backend/src/main/java/com/aiteacher/document/` — composite PK `(chunkId UUID, figureId String)`, `mentionPage` (int); `ChunkFigureRefRepository` with `findByChunkIdIn(List<UUID>)`, `deleteByFigureIdIn(List<String>)`
**Checkpoint**: Migrations will run on next startup; all JPA entities are wired; figure storage reads config correctly
---
## Phase 3: User Story 2 — All Pages Scanned for Images During Embedding (Priority: P1)
**Goal**: When a book is uploaded, every page is inspected for images; each found image is extracted, persisted, described, and embedded as a searchable chunk alongside its metadata
**Independent Test**: Upload a PDF containing at least one page with a labelled anatomical diagram. After status shows `READY`, call `GET /api/v1/books/{id}/figures` — response must contain at least one entry with `figureType`, `caption`, `page`, and `imageUrl` populated. Verify the PNG file exists at the path in `imagePath`.
- [X] T013 [US2] ~~Create `PdfStructureParser`~~**SUPERSEDED**: PDF parsing is handled by `MarkerPageParser` (see T013b). `PdfStructureParser` exists but is not wired into the pipeline.
- [X] T013b [US2] Create `MarkerPageParser` in `backend/src/main/java/com/aiteacher/document/MarkerPageParser.java` — POSTs PDF to `http://localhost:8000/marker/upload?output_format=json` via Spring `RestClient`; parses JSON response into `List<PageResult>` (one per page block); extracts heading, ordered text, and pre-cropped figure PNG bytes per page
- [X] T014 [US2] Update `FigureExtractionService` in `backend/src/main/java/com/aiteacher/document/FigureExtractionService.java`**Marker migration**: removed PDFBox rendering + bbox-crop loop; decodes PNG bytes from `PageResult.FigureData` via `ImageIO.read()`; skips images below `min-image-size-px`; classifies `FigureType`; saves via `FigureStorageService`; persists `FigureEntity`
- [X] T015 [US2] Create `VisionDescriptionService` in `backend/src/main/java/com/aiteacher/document/VisionDescriptionService.java` — accepts a `Path` to a PNG and a caption String; calls the OpenAI vision model (via Spring AI `ChatClient` with image media type) to generate a 24 sentence clinical description; returns the generated description string; handles API failures by returning the caption as fallback
- [X] T016 [US2] Create `TextChunkingService` in `backend/src/main/java/com/aiteacher/document/TextChunkingService.java` — accepts a `SectionEntity`; splits `fullText` into overlapping 400600 token windows (20-token overlap); wraps each window in a Spring AI `Document` with the flat metadata map defined in data-model.md §Text chunk document; returns `List<Document>`
- [X] T017 [US2] Create `ChunkFigureRefService` in `backend/src/main/java/com/aiteacher/document/ChunkFigureRefService.java` — accepts a Spring AI `Document` (with its `id` as `chunkId`) and a `List<FigureEntity>` for the book; scans chunk text for patterns `Fig\.\s*\d+[\-\.]\d+` and `Figure\s+\d+[\-\.]\d+`; matches against figure labels; persists `ChunkFigureRefEntity` rows via `ChunkFigureRefRepository`
- [X] T018 [US2] Update `BookEmbeddingService.embedBook()`**Marker migration**: injected `MarkerPageParser` replacing `DocumentAiPageParser`; updated `figureExtractionService.extract()` call (removed `pdfPath` arg); updated log message. Pipeline: (1) `MarkerPageParser``List<PageResult>`; (2) `buildAndSaveSections()` → sections; (3) `TextChunkingService` → chunks → embed; (4) `FigureExtractionService.extract()` → figures; (5) `VisionDescriptionService` → embed figure chunks; (6) `ChunkFigureRefService` → refs
- [X] T019 [US2] Extend `BookEmbeddingService.deleteBookChunks()` to also delete: all `ChunkFigureRefEntity` rows (via `findByFigureIdIn`), all `FigureEntity` rows (via `deleteAllByBookId`), all figure PNG files (via `FigureStorageService.delete(bookId)`), all `SectionEntity` and `ChapterEntity` rows for the book
- [X] T020 [US2] Add `POST /api/v1/books/{id}/reembed` endpoint to `BookController` in `backend/src/main/java/com/aiteacher/book/BookController.java` — returns `202` with `{ bookId, status: "PROCESSING" }`; returns `404` if not found; returns `409` if already `PROCESSING`; calls `deleteBookChunks()` then `embedBook()` asynchronously
**Checkpoint**: Upload a PDF with figures → poll `GET /api/v1/books` for `READY``GET /api/v1/books/{id}/figures` returns figure list → PNG accessible at `GET /api/v1/figures/{bookId}/{filename}`
---
## Phase 4: User Story 1 — Image Content Surfaced in Query Results (Priority: P1)
**Goal**: User asks a question answered by a diagram — the system retrieves that diagram's content and surfaces it in the chat response with a citation
**Independent Test**: With a book embedded (Phase 3 checkpoint passed), ask a chat question whose answer is depicted only in a diagram. The response `sources` array must contain at least one entry with `type: "FIGURE"` and a non-empty `imageUrl`.
- [X] T021 [US1] Create `NeurosurgeryRetriever` service in `backend/src/main/java/com/aiteacher/retrieval/NeurosurgeryRetriever.java` — (1) text chunk search: `vectorStore.similaritySearch` with filter `type == TEXT AND book_id == bookId`, topK=5; (2) figure search: same store, filter `type == FIGURE AND book_id == bookId`, topK=3; (3) expand text chunk results to parent sections via `SectionRepository.findAllById(sectionIds)`; (4) fetch explicitly linked figures via `ChunkFigureRefRepository.findByChunkIdIn(chunkIds)` + `FigureRepository.findAllById`; (5) deduplicate figures across lists by `figureId`; return `RetrievalResult(parentSections, figureVectorHits, linkedFigures)` — add `RetrievalResult` record in same package
- [X] T022 [US1] Refactor `ChatService.sendMessage()` in `backend/src/main/java/com/aiteacher/chat/ChatService.java` — replace `QuestionAnswerAdvisor` with a manual call to `NeurosurgeryRetriever`; build the LLM user message from: section full texts as `[Section X.Y — Title, pp.A-B]\n{fullText}` blocks, followed by `AVAILABLE FIGURES FOR THIS SECTION:` list with `- {label} (p.{page}): {caption} [image: {filename}]` lines per figure; append the instruction `When referencing diagrams, cite them as [Fig. X, p.N].`; send via `chatClient.prompt().system(SYSTEM_PROMPT).user(prompt).call()`
- [X] T023 [US1] Add `GET /api/v1/books/{id}/figures` endpoint to `BookController` — returns `200` with `List<FigureResponse>`; `FigureResponse` is a new record in `backend/src/main/java/com/aiteacher/book/FigureResponse.java` with fields `figureId`, `label`, `caption`, `figureType`, `page`, `imageUrl` (assembled as `/api/v1/figures/{bookId}/{filename}`), `sectionId`, `sectionTitle`; returns `404` if book not found
- [X] T024 [US1] Update `extractSources()` in `ChatService` to build both TEXT and FIGURE source entries: TEXT entries keep existing fields plus `"type": "TEXT"`; FIGURE entries add `"type": "FIGURE"`, `"figureId"`, `"label"`, `"caption"`, `"figureType"`, `"imageUrl"` — source data comes from `RetrievalResult` (text chunk Documents and merged FigureEntity list)
**Checkpoint**: Chat question answered by a diagram → response body contains `sources[n].type == "FIGURE"` with populated `imageUrl`; image loads from the returned URL
---
## Phase 5: User Story 3 — Rich Metadata Enables Precise Source Attribution (Priority: P2)
**Goal**: Users see distinct, informative citations for text vs. image sources; image sources render inline in the chat UI
**Independent Test**: After triggering a response with figure sources, inspect the chat message in the UI — text sources and figure sources are visually distinguishable; figure sources render the actual image inline using the `imageUrl`
- [X] T025 [P] [US3] Update API response types in `frontend/src/services/api.ts` — extend the `Source` type to include `type: 'TEXT' | 'FIGURE'`, `figureId?: string`, `label?: string`, `caption?: string`, `figureType?: string`, `imageUrl?: string`
- [X] T026 [P] [US3] Update the chat source/citation display in the frontend (wherever sources are currently rendered, e.g. `frontend/src/components/` or `frontend/src/views/`) — render TEXT sources with a document icon and page number; render FIGURE sources with the image (`<img :src="source.imageUrl">`) below the label and caption text
- [X] T027 [US3] Add figure-type badge rendering in the frontend figure display: show a label derived from `figureType` (e.g. "MRI / CT", "Anatomical Diagram", "Table") alongside the figure caption so users can identify content type without opening the image
---
## Phase 6: Polish & Cross-Cutting Concerns
- [X] T028 Update `README.md` Mermaid architecture diagram to show three storage tiers: pgvector (semantic search), Postgres (source of truth — sections, figures, refs), and file store (extracted PNGs) — **required by Constitution Principle IV in the same PR as the other changes**
- [X] T029 [P] Write `FigureExtractionServiceTest` unit test in `backend/src/test/java/com/aiteacher/document/FigureExtractionServiceTest.java` — test: images below min size are skipped; `FigureType` classification matches keyword table in data-model.md; caption parsed from adjacent text line
- [X] T030 [P] Write `NeurosurgeryRetrieverTest` unit test in `backend/src/test/java/com/aiteacher/retrieval/NeurosurgeryRetrieverTest.java` — test: figure IDs from both vector hits and chunk refs are merged without duplicates; `RetrievalResult` contains the deduplicated set
- [X] T031 Run quickstart.md validation end-to-end: upload a real PDF with a labelled diagram → wait for `READY` → call `GET /api/v1/books/{id}/figures` → send a chat message about the diagram → verify `sources` contains a `FIGURE` entry → verify `imageUrl` resolves to a PNG
---
## Dependencies & Execution Order
### Phase Dependencies
- **Phase 1 (Setup)**: No dependencies — start immediately
- **Phase 2 (Foundational)**: Requires Phase 1 complete (migrations must run before JPA entities can be wired)
- **Phase 3 (US2)**: Requires Phase 2 complete — all JPA entities + FigureStorageService must exist
- **Phase 4 (US1)**: Requires Phase 3 complete — figures must exist in Postgres + vector store before retrieval can surface them
- **Phase 5 (US3)**: Requires Phase 4 complete — frontend depends on the extended `sources` format from T024
- **Phase 6 (Polish)**: Requires all story phases complete
### Within Phase 3 (Embedding Pipeline)
```
T013 (PdfStructureParser) ──────────────────────────┐
T014 (FigureExtractionService) ─────────────────────┤
T015 (VisionDescriptionService) ────────────────────┤─→ T018 (BookEmbeddingService orchestrator)
T016 (TextChunkingService) ─────────────────────────┤ └─→ T019 (cleanup)
T017 (ChunkFigureRefService) ───────────────────────┘ └─→ T020 (reembed endpoint)
```
T013T017 can be implemented in parallel (different files, no shared dependencies). T018 depends on all of them.
### Within Phase 4 (Retrieval)
```
T021 (NeurosurgeryRetriever) ──────────────────────┐
└─→ T022 (ChatService update)
└─→ T024 (extractSources update)
T023 (figures endpoint) ── independent [P]
```
### Parallel Opportunities per Phase
**Phase 2**: T005, T006, T009, T010, T011 can all run in parallel. T007 depends on T006. T012 can follow T010/T011.
**Phase 3**: T013, T014, T015, T016, T017 all in parallel. T018 depends on all.
**Phase 5**: T025 and T026 in parallel; T027 can follow T026.
**Phase 6**: T029 and T030 in parallel.
---
## Implementation Strategy
### MVP: User Story 2 Only (Embedding Pipeline)
1. Phase 1 (Setup) → Phase 2 (Foundational) → Phase 3 (US2, T013T020)
2. **Validate**: `GET /api/v1/books/{id}/figures` returns figures for a test book
3. **Stop and demo** — the pipeline produces image chunks without any retrieval changes
### Full Feature Delivery
1. Phase 1 + 2 → Foundation ready
2. Phase 3 (US2) → Embedding pipeline produces image chunks ← **demo point**
3. Phase 4 (US1) → Chat surfaces image content in responses ← **core payoff**
4. Phase 5 (US3) → Frontend renders inline figures with type badges
5. Phase 6 (Polish) → README, tests, end-to-end validation
---
## Notes
- [P] tasks = different files, no dependencies on each other within the same phase
- [US1/US2/US3] label maps each task to a user story for traceability
- Phase 3 (US2) must be fully complete before beginning Phase 4 (US1) — retrieval cannot surface figures that do not yet exist
- The `uploads/figures/` directory must exist and be writable at runtime; `FigureStorageService` creates subdirectories automatically
- Re-embedding (T020) deletes all existing chunks and figures for the book before re-running — safe to call on books processed by feature 001