21 Commits

Author SHA1 Message Date
Adrien 0c226483c0 fix deserialization error in native image 2026-04-18 20:46:16 +02:00
Adrien ff97c24a55 Add thai support in summary 2026-04-18 19:55:19 +02:00
Adrien c7a77af2f4 add new concept report 2026-04-18 17:54:54 +02:00
Adrien 5f03e1f41b improve topics and chat source display 2026-04-12 18:56:18 +02:00
Adrien c98fe9ceaa update readme 2026-04-12 18:25:12 +02:00
Adrien 767d1e2dbc enhance illustration being taken into account in the response 2026-04-12 16:26:25 +02:00
Adrien 820734c251 fix api url setup 2026-04-10 13:55:05 +02:00
Adrien 0711e40c66 Improved responsiveness on mobile phone 2026-04-10 13:41:26 +02:00
Adrien 0db31e91ab try change image building to buildah 2026-04-09 22:44:14 +02:00
Adrien d480d04145 change base image 2026-04-09 21:45:22 +02:00
Adrien c2d034d1fe Add missing env variables 2026-04-09 20:37:11 +02:00
Adrien 0908355704 Adpat frontend to build docker image with buildah 2026-04-09 19:47:28 +02:00
Adrien 8e227a9429 fine-tune native image config 2026-04-09 18:20:39 +02:00
Adrien d8bcdce879 Squashed commit of the following:
commit 0d624137c2557c6eeb87020749e4977b821c2b5c
Author: Adrien <adrien.cesaro@proton.me>
Date:   Thu Apr 9 11:55:22 2026 +0200

    backend native image setup
2026-04-09 12:05:02 +02:00
Adrien aee6a9dfba enhance rag retrieval + summary 2026-04-07 22:39:28 +02:00
Adrien 0cf318f0a7 Add simple auth 2026-04-06 14:29:53 +02:00
Adrien e5d53b4e80 add possibility to disable delete and upload of books 2026-04-06 14:09:17 +02:00
Adrien 5c641f4bcc enhance page parsing using json output and html 2026-04-05 21:55:30 +02:00
Adrien ea1276dc2e adding Marker to parse effectively pdf 2026-04-04 21:30:18 +02:00
Adrien b154e29f2d s3 bucket integration for image storage 2026-04-04 13:26:55 +02:00
Adrien 5acfdd33c1 first implementation - image/drawing integration 2026-04-04 12:56:56 +02:00
142 changed files with 12645 additions and 453 deletions
+27
View File
@@ -0,0 +1,27 @@
.git/
.gitignore
*.md
.DS_Store
Thumbs.db
# Java build artifacts
target/
*.class
*.jar
# Node
node_modules/
dist/
*.log
# Env files (never bake secrets into images)
.env
.env.*
!.env.example
# Spec / docs
specs/
# Editor
.vscode/
.idea/
+13
View File
@@ -0,0 +1,13 @@
# Copy this file to .env and fill in your values before running docker-compose.native.yml
# .env is gitignored — never commit real credentials
# OpenAI
OPENAI_API_KEY=sk-...
# AWS S3 (figure storage — leave blank if using local filesystem)
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_REGION=eu-west-1
# S3 bucket name (if S3 storage enabled)
APP_STORAGE_S3_BUCKET=ai-teacher-figures
+12
View File
@@ -1,3 +1,15 @@
# Runtime uploads (extracted figures)
uploads/
# Java build
target/
*.class
*.jar
# Node
node_modules/
dist/
# OS # OS
.DS_Store .DS_Store
Thumbs.db Thumbs.db
+19 -2
View File
@@ -1,8 +1,23 @@
# ai-teacher Development Guidelines # ai-teacher Development Guidelines
Auto-generated from all feature plans. Last updated: 2026-03-31 Auto-generated from all feature plans. Last updated: 2026-04-10
## Active Technologies ## Active Technologies
- Java 25 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API (embeddings + chat), PDFBox (via Spring AI PDF reader dependency) (002-image-aware-embedding)
- PostgreSQL (JPA + Flyway), pgvector (Spring AI `VectorStore`), local file system (extracted images — `/uploads/figures/`) (002-image-aware-embedding)
- Java 25 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API, PDFBox (rendering only), `com.google.cloud:google-cloud-documentai` (~2.40.x) (002-image-aware-embedding)
- PostgreSQL (JPA + Flyway), pgvector (Spring AI VectorStore), S3 / local filesystem (figure images) (002-image-aware-embedding)
- PostgreSQL (JPA + Flyway), pgvector (Spring AI `VectorStore`), S3-compatible (002-image-aware-embedding)
- Java 21 (backend) / TypeScript + Node 20 (frontend) + Spring Boot 4.0.5, Spring Security (already included), Vue 3.4, Vue Router 4.3, Pinia 2.1, Axios 1.7 (003-basic-login)
- No new storage — credentials held in browser `sessionStorage` (frontend only) (003-basic-login)
- Java 21 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API (chat + embeddings), pgvector, Vue 3.4, Pinia 2.1 (004-rag-retrieval-quality)
- PostgreSQL (sections, figures, messages — unchanged). No new tables needed. (004-rag-retrieval-quality)
- Java 21 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API (chat + embeddings), Vue 3.4, Pinia 2.1, Axios 1.7 (004-rag-retrieval-quality)
- PostgreSQL (JPA + Flyway), pgvector (`VectorStore`) (004-rag-retrieval-quality)
- Java 25 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, `native-maven-plugin` 0.10.6, (005-native-image-deployment)
- PostgreSQL 16 + pgvector (unchanged) (005-native-image-deployment)
- TypeScript / Node 20 (frontend only) + Vue 3.4, Vue Router 4.3, Pinia 2.1 — no changes (006-mobile-responsive-ui)
- N/A (frontend-only change) (006-mobile-responsive-ui)
- Java 21 (backend), TypeScript / Node 20 (frontend) (001-neuro-rag-learning) - Java 21 (backend), TypeScript / Node 20 (frontend) (001-neuro-rag-learning)
@@ -22,8 +37,10 @@ npm test && npm run lint
Java 21 (backend), TypeScript / Node 20 (frontend): Follow standard conventions Java 21 (backend), TypeScript / Node 20 (frontend): Follow standard conventions
## Recent Changes ## Recent Changes
- 006-mobile-responsive-ui: Added TypeScript / Node 20 (frontend only) + Vue 3.4, Vue Router 4.3, Pinia 2.1 — no changes
- 005-native-image-deployment: Added Java 25 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, `native-maven-plugin` 0.10.6,
- 004-rag-retrieval-quality: Added Java 21 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API (chat + embeddings), Vue 3.4, Pinia 2.1, Axios 1.7
- 001-neuro-rag-learning: Added Java 21 (backend), TypeScript / Node 20 (frontend)
<!-- MANUAL ADDITIONS START --> <!-- MANUAL ADDITIONS START -->
<!-- MANUAL ADDITIONS END --> <!-- MANUAL ADDITIONS END -->
+566
View File
@@ -0,0 +1,566 @@
# Marker
Marker converts documents to markdown, JSON, chunks, and HTML quickly and accurately.
- Converts PDF, image, PPTX, DOCX, XLSX, HTML, EPUB files in all languages
- Formats tables, forms, equations, inline math, links, references, and code blocks
- Extracts and saves images
- Removes headers/footers/other artifacts
- Extensible with your own formatting and logic
- Does structured extraction, given a JSON schema (beta)
- Optionally boost accuracy with LLMs (and your own prompt)
- Works on GPU, CPU, or MPS
For our managed API or on-prem document intelligence solution, check out [our platform here](https://datalab.to?utm_source=gh-marker).
## Performance
<img src="data/images/overall.png" width="800px"/>
Marker benchmarks favorably compared to cloud services like Llamaparse and Mathpix, as well as other open source tools.
The above results are running single PDF pages serially. Marker is significantly faster when running in batch mode, with a projected throughput of 25 pages/second on an H100.
See [below](#benchmarks) for detailed speed and accuracy benchmarks, and instructions on how to run your own benchmarks.
## Hybrid Mode
For the highest accuracy, pass the `--use_llm` flag to use an LLM alongside marker. This will do things like merge tables across pages, handle inline math, format tables properly, and extract values from forms. It can use any gemini or ollama model. By default, it uses `gemini-2.0-flash`. See [below](#llm-services) for details.
Here is a table benchmark comparing marker, gemini flash alone, and marker with use_llm:
<img src="data/images/table.png" width="400px"/>
As you can see, the use_llm mode offers higher accuracy than marker or gemini alone.
## Examples
| PDF | File type | Markdown | JSON |
|-----|-----------|------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------|
| [Think Python](https://greenteapress.com/thinkpython/thinkpython.pdf) | Textbook | [View](https://github.com/VikParuchuri/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md) | [View](https://github.com/VikParuchuri/marker/blob/master/data/examples/json/thinkpython.json) |
| [Switch Transformers](https://arxiv.org/pdf/2101.03961.pdf) | arXiv paper | [View](https://github.com/VikParuchuri/marker/blob/master/data/examples/markdown/switch_transformers/switch_trans.md) | [View](https://github.com/VikParuchuri/marker/blob/master/data/examples/json/switch_trans.json) |
| [Multi-column CNN](https://arxiv.org/pdf/1804.07821.pdf) | arXiv paper | [View](https://github.com/VikParuchuri/marker/blob/master/data/examples/markdown/multicolcnn/multicolcnn.md) | [View](https://github.com/VikParuchuri/marker/blob/master/data/examples/json/multicolcnn.json) |
# Commercial usage
Our model weights use a modified AI Pubs Open Rail-M license (free for research, personal use, and startups under $2M funding/revenue) and our code is GPL. For broader commercial licensing or to remove GPL requirements, visit our pricing page [here](https://www.datalab.to/pricing?utm_source=gh-marker).
# Hosted API & On-prem
There's a [hosted API](https://www.datalab.to?utm_source=gh-marker) and [painless on-prem solution](https://www.datalab.to/blog/self-serve-on-prem-licensing) for marker - it's free to sign up, and we'll throw in credits for you to test it out.
The API:
- Supports PDF, image, PPT, PPTX, DOC, DOCX, XLS, XLSX, HTML, EPUB files
- Is 1/4th the price of leading cloud-based competitors
- Fast - ~15s for a 250 page PDF
- Supports LLM mode
- High uptime (99.99%)
# Community
[Discord](https://discord.gg//KuZwXNGnfH) is where we discuss future development.
# Installation
You'll need python 3.10+ and [PyTorch](https://pytorch.org/get-started/locally/).
Install with:
```shell
pip install marker-pdf
```
If you want to use marker on documents other than PDFs, you will need to install additional dependencies with:
```shell
pip install marker-pdf[full]
```
# Usage
First, some configuration:
- Your torch device will be automatically detected, but you can override this. For example, `TORCH_DEVICE=cuda`.
- Some PDFs, even digital ones, have bad text in them. Set `--force_ocr` to force OCR on all lines, or the `strip_existing_ocr` to keep all digital text, and strip out any existing OCR text.
- If you care about inline math, set `force_ocr` to convert inline math to LaTeX.
## Interactive App
I've included a streamlit app that lets you interactively try marker with some basic options. Run it with:
```shell
pip install streamlit streamlit-ace
marker_gui
```
## Convert a single file
```shell
marker_single /path/to/file.pdf
```
You can pass in PDFs or images.
Options:
- `--page_range TEXT`: Specify which pages to process. Accepts comma-separated page numbers and ranges. Example: `--page_range "0,5-10,20"` will process pages 0, 5 through 10, and page 20.
- `--output_format [markdown|json|html|chunks]`: Specify the format for the output results.
- `--output_dir PATH`: Directory where output files will be saved. Defaults to the value specified in settings.OUTPUT_DIR.
- `--paginate_output`: Paginates the output, using `\n\n{PAGE_NUMBER}` followed by `-` * 48, then `\n\n`
- `--use_llm`: Uses an LLM to improve accuracy. You will need to configure the LLM backend - see [below](#llm-services).
- `--force_ocr`: Force OCR processing on the entire document, even for pages that might contain extractable text. This will also format inline math properly.
- `--block_correction_prompt`: if LLM mode is active, an optional prompt that will be used to correct the output of marker. This is useful for custom formatting or logic that you want to apply to the output.
- `--strip_existing_ocr`: Remove all existing OCR text in the document and re-OCR with surya.
- `--redo_inline_math`: If you want the absolute highest quality inline math conversion, use this along with `--use_llm`.
- `--disable_image_extraction`: Don't extract images from the PDF. If you also specify `--use_llm`, then images will be replaced with a description.
- `--debug`: Enable debug mode for additional logging and diagnostic information.
- `--processors TEXT`: Override the default processors by providing their full module paths, separated by commas. Example: `--processors "module1.processor1,module2.processor2"`
- `--config_json PATH`: Path to a JSON configuration file containing additional settings.
- `config --help`: List all available builders, processors, and converters, and their associated configuration. These values can be used to build a JSON configuration file for additional tweaking of marker defaults.
- `--converter_cls`: One of `marker.converters.pdf.PdfConverter` (default) or `marker.converters.table.TableConverter`. The `PdfConverter` will convert the whole PDF, the `TableConverter` will only extract and convert tables.
- `--llm_service`: Which llm service to use if `--use_llm` is passed. This defaults to `marker.services.gemini.GoogleGeminiService`.
- `--help`: see all of the flags that can be passed into marker. (it supports many more options then are listed above)
The list of supported languages for surya OCR is [here](https://github.com/VikParuchuri/surya/blob/master/surya/recognition/languages.py). If you don't need OCR, marker can work with any language.
## Convert multiple files
```shell
marker /path/to/input/folder
```
- `marker` supports all the same options from `marker_single` above.
- `--workers` is the number of conversion workers to run simultaneously. This is automatically set by default, but you can increase it to increase throughput, at the cost of more CPU/GPU usage. Marker will use 5GB of VRAM per worker at the peak, and 3.5GB average.
## Convert multiple files on multiple GPUs
```shell
NUM_DEVICES=4 NUM_WORKERS=15 marker_chunk_convert ../pdf_in ../md_out
```
- `NUM_DEVICES` is the number of GPUs to use. Should be `2` or greater.
- `NUM_WORKERS` is the number of parallel processes to run on each GPU.
## Use from python
See the `PdfConverter` class at `marker/converters/pdf.py` function for additional arguments that can be passed.
```python
from marker.converters.pdf import PdfConverter
from marker.models import create_model_dict
from marker.output import text_from_rendered
converter = PdfConverter(
artifact_dict=create_model_dict(),
)
rendered = converter("FILEPATH")
text, _, images = text_from_rendered(rendered)
```
`rendered` will be a pydantic basemodel with different properties depending on the output type requested. With markdown output (default), you'll have the properties `markdown`, `metadata`, and `images`. For json output, you'll have `children`, `block_type`, and `metadata`.
### Custom configuration
You can pass configuration using the `ConfigParser`. To see all available options, do `marker_single --help`.
```python
from marker.converters.pdf import PdfConverter
from marker.models import create_model_dict
from marker.config.parser import ConfigParser
config = {
"output_format": "json",
"ADDITIONAL_KEY": "VALUE"
}
config_parser = ConfigParser(config)
converter = PdfConverter(
config=config_parser.generate_config_dict(),
artifact_dict=create_model_dict(),
processor_list=config_parser.get_processors(),
renderer=config_parser.get_renderer(),
llm_service=config_parser.get_llm_service()
)
rendered = converter("FILEPATH")
```
### Extract blocks
Each document consists of one or more pages. Pages contain blocks, which can themselves contain other blocks. It's possible to programmatically manipulate these blocks.
Here's an example of extracting all forms from a document:
```python
from marker.converters.pdf import PdfConverter
from marker.models import create_model_dict
from marker.schema import BlockTypes
converter = PdfConverter(
artifact_dict=create_model_dict(),
)
document = converter.build_document("FILEPATH")
forms = document.contained_blocks((BlockTypes.Form,))
```
Look at the processors for more examples of extracting and manipulating blocks.
## Other converters
You can also use other converters that define different conversion pipelines:
### Extract tables
The `TableConverter` will only convert and extract tables:
```python
from marker.converters.table import TableConverter
from marker.models import create_model_dict
from marker.output import text_from_rendered
converter = TableConverter(
artifact_dict=create_model_dict(),
)
rendered = converter("FILEPATH")
text, _, images = text_from_rendered(rendered)
```
This takes all the same configuration as the PdfConverter. You can specify the configuration `force_layout_block=Table` to avoid layout detection and instead assume every page is a table. Set `output_format=json` to also get cell bounding boxes.
You can also run this via the CLI with
```shell
marker_single FILENAME --use_llm --force_layout_block Table --converter_cls marker.converters.table.TableConverter --output_format json
```
### OCR Only
If you only want to run OCR, you can also do that through the `OCRConverter`. Set `--keep_chars` to keep individual characters and bounding boxes.
```python
from marker.converters.ocr import OCRConverter
from marker.models import create_model_dict
converter = OCRConverter(
artifact_dict=create_model_dict(),
)
rendered = converter("FILEPATH")
```
This takes all the same configuration as the PdfConverter.
You can also run this via the CLI with
```shell
marker_single FILENAME --converter_cls marker.converters.ocr.OCRConverter
```
### Structured Extraction (beta)
You can run structured extraction via the `ExtractionConverter`. This requires an llm service to be setup first (see [here](#llm-services) for details). You'll get a JSON output with the extracted values.
```python
from marker.converters.extraction import ExtractionConverter
from marker.models import create_model_dict
from marker.config.parser import ConfigParser
from pydantic import BaseModel
class Links(BaseModel):
links: list[str]
schema = Links.model_json_schema()
config_parser = ConfigParser({
"page_schema": schema
})
converter = ExtractionConverter(
artifact_dict=create_model_dict(),
config=config_parser.generate_config_dict(),
llm_service=config_parser.get_llm_service(),
)
rendered = converter("FILEPATH")
```
Rendered will have an `original_markdown` field. If you pass this back in next time you run the converter, as the `existing_markdown` config key, you can skip re-parsing the document.
# Output Formats
## Markdown
Markdown output will include:
- image links (images will be saved in the same folder)
- formatted tables
- embedded LaTeX equations (fenced with `$$`)
- Code is fenced with triple backticks
- Superscripts for footnotes
## HTML
HTML output is similar to markdown output:
- Images are included via `img` tags
- equations are fenced with `<math>` tags
- code is in `pre` tags
## JSON
JSON output will be organized in a tree-like structure, with the leaf nodes being blocks. Examples of leaf nodes are a single list item, a paragraph of text, or an image.
The output will be a list, with each list item representing a page. Each page is considered a block in the internal marker schema. There are different types of blocks to represent different elements.
Pages have the keys:
- `id` - unique id for the block.
- `block_type` - the type of block. The possible block types can be seen in `marker/schema/__init__.py`. As of this writing, they are ["Line", "Span", "FigureGroup", "TableGroup", "ListGroup", "PictureGroup", "Page", "Caption", "Code", "Figure", "Footnote", "Form", "Equation", "Handwriting", "TextInlineMath", "ListItem", "PageFooter", "PageHeader", "Picture", "SectionHeader", "Table", "Text", "TableOfContents", "Document"]
- `html` - the HTML for the page. Note that this will have recursive references to children. The `content-ref` tags must be replaced with the child content if you want the full html. You can see an example of this at `marker/output.py:json_to_html`. That function will take in a single block from the json output, and turn it into HTML.
- `polygon` - the 4-corner polygon of the page, in (x1,y1), (x2,y2), (x3, y3), (x4, y4) format. (x1,y1) is the top left, and coordinates go clockwise.
- `children` - the child blocks.
The child blocks have two additional keys:
- `section_hierarchy` - indicates the sections that the block is part of. `1` indicates an h1 tag, `2` an h2, and so on.
- `images` - base64 encoded images. The key will be the block id, and the data will be the encoded image.
Note that child blocks of pages can have their own children as well (a tree structure).
```json
{
"id": "/page/10/Page/366",
"block_type": "Page",
"html": "<content-ref src='/page/10/SectionHeader/0'></content-ref><content-ref src='/page/10/SectionHeader/1'></content-ref><content-ref src='/page/10/Text/2'></content-ref><content-ref src='/page/10/Text/3'></content-ref><content-ref src='/page/10/Figure/4'></content-ref><content-ref src='/page/10/SectionHeader/5'></content-ref><content-ref src='/page/10/SectionHeader/6'></content-ref><content-ref src='/page/10/TextInlineMath/7'></content-ref><content-ref src='/page/10/TextInlineMath/8'></content-ref><content-ref src='/page/10/Table/9'></content-ref><content-ref src='/page/10/SectionHeader/10'></content-ref><content-ref src='/page/10/Text/11'></content-ref>",
"polygon": [[0.0, 0.0], [612.0, 0.0], [612.0, 792.0], [0.0, 792.0]],
"children": [
{
"id": "/page/10/SectionHeader/0",
"block_type": "SectionHeader",
"html": "<h1>Supplementary Material for <i>Subspace Adversarial Training</i> </h1>",
"polygon": [
[217.845703125, 80.630859375], [374.73046875, 80.630859375],
[374.73046875, 107.0],
[217.845703125, 107.0]
],
"children": null,
"section_hierarchy": {
"1": "/page/10/SectionHeader/1"
},
"images": {}
},
...
]
}
```
## Chunks
Chunks format is similar to JSON, but flattens everything into a single list instead of a tree. Only the top level blocks from each page show up. It also has the full HTML of each block inside, so you don't need to crawl the tree to reconstruct it. This enable flexible and easy chunking for RAG.
## Metadata
All output formats will return a metadata dictionary, with the following fields:
```json
{
"table_of_contents": [
{
"title": "Introduction",
"heading_level": 1,
"page_id": 0,
"polygon": [...]
}
], // computed PDF table of contents
"page_stats": [
{
"page_id": 0,
"text_extraction_method": "pdftext",
"block_counts": [("Span", 200), ...]
},
...
]
}
```
# LLM Services
When running with the `--use_llm` flag, you have a choice of services you can use:
- `Gemini` - this will use the Gemini developer API by default. You'll need to pass `--gemini_api_key` to configuration.
- `Google Vertex` - this will use vertex, which can be more reliable. You'll need to pass `--vertex_project_id`. To use it, set `--llm_service=marker.services.vertex.GoogleVertexService`.
- `Ollama` - this will use local models. You can configure `--ollama_base_url` and `--ollama_model`. To use it, set `--llm_service=marker.services.ollama.OllamaService`.
- `Claude` - this will use the anthropic API. You can configure `--claude_api_key`, and `--claude_model_name`. To use it, set `--llm_service=marker.services.claude.ClaudeService`.
- `OpenAI` - this supports any openai-like endpoint. You can configure `--openai_api_key`, `--openai_model`, and `--openai_base_url`. To use it, set `--llm_service=marker.services.openai.OpenAIService`.
- `Azure OpenAI` - this uses the Azure OpenAI service. You can configure `--azure_endpoint`, `--azure_api_key`, and `--deployment_name`. To use it, set `--llm_service=marker.services.azure_openai.AzureOpenAIService`.
These services may have additional optional configuration as well - you can see it by viewing the classes.
# Internals
Marker is easy to extend. The core units of marker are:
- `Providers`, at `marker/providers`. These provide information from a source file, like a PDF.
- `Builders`, at `marker/builders`. These generate the initial document blocks and fill in text, using info from the providers.
- `Processors`, at `marker/processors`. These process specific blocks, for example the table formatter is a processor.
- `Renderers`, at `marker/renderers`. These use the blocks to render output.
- `Schema`, at `marker/schema`. The classes for all the block types.
- `Converters`, at `marker/converters`. They run the whole end to end pipeline.
To customize processing behavior, override the `processors`. To add new output formats, write a new `renderer`. For additional input formats, write a new `provider.`
Processors and renderers can be directly passed into the base `PDFConverter`, so you can specify your own custom processing easily.
## API server
There is a very simple API server you can run like this:
```shell
pip install -U uvicorn fastapi python-multipart
marker_server --port 8001
```
This will start a fastapi server that you can access at `localhost:8001`. You can go to `localhost:8001/docs` to see the endpoint options.
You can send requests like this:
```
import requests
import json
post_data = {
'filepath': 'FILEPATH',
# Add other params here
}
requests.post("http://localhost:8001/marker", data=json.dumps(post_data)).json()
```
Note that this is not a very robust API, and is only intended for small-scale use. If you want to use this server, but want a more robust conversion option, you can use the hosted [Datalab API](https://www.datalab.to/plans).
# Troubleshooting
There are some settings that you may find useful if things aren't working the way you expect:
- If you have issues with accuracy, try setting `--use_llm` to use an LLM to improve quality. You must set `GOOGLE_API_KEY` to a Gemini API key for this to work.
- Make sure to set `force_ocr` if you see garbled text - this will re-OCR the document.
- `TORCH_DEVICE` - set this to force marker to use a given torch device for inference.
- If you're getting out of memory errors, decrease worker count. You can also try splitting up long PDFs into multiple files.
## Debugging
Pass the `debug` option to activate debug mode. This will save images of each page with detected layout and text, as well as output a json file with additional bounding box information.
# Benchmarks
## Overall PDF Conversion
We created a [benchmark set](https://huggingface.co/datasets/datalab-to/marker_benchmark) by extracting single PDF pages from common crawl. We scored based on a heuristic that aligns text with ground truth text segments, and an LLM as a judge scoring method.
| Method | Avg Time | Heuristic Score | LLM Score |
|------------|----------|-----------------|-----------|
| marker | 2.83837 | 95.6709 | 4.23916 |
| llamaparse | 23.348 | 84.2442 | 3.97619 |
| mathpix | 6.36223 | 86.4281 | 4.15626 |
| docling | 3.69949 | 86.7073 | 3.70429 |
Benchmarks were run on an H100 for markjer and docling - llamaparse and mathpix used their cloud services. We can also look at it by document type:
<img src="data/images/per_doc.png" width="1000px"/>
| Document Type | Marker heuristic | Marker LLM | Llamaparse Heuristic | Llamaparse LLM | Mathpix Heuristic | Mathpix LLM | Docling Heuristic | Docling LLM |
|----------------------|------------------|------------|----------------------|----------------|-------------------|-------------|-------------------|-------------|
| Scientific paper | 96.6737 | 4.34899 | 87.1651 | 3.96421 | 91.2267 | 4.46861 | 92.135 | 3.72422 |
| Book page | 97.1846 | 4.16168 | 90.9532 | 4.07186 | 93.8886 | 4.35329 | 90.0556 | 3.64671 |
| Other | 95.1632 | 4.25076 | 81.1385 | 4.01835 | 79.6231 | 4.00306 | 83.8223 | 3.76147 |
| Form | 88.0147 | 3.84663 | 66.3081 | 3.68712 | 64.7512 | 3.33129 | 68.3857 | 3.40491 |
| Presentation | 95.1562 | 4.13669 | 81.2261 | 4 | 83.6737 | 3.95683 | 84.8405 | 3.86331 |
| Financial document | 95.3697 | 4.39106 | 82.5812 | 4.16111 | 81.3115 | 4.05556 | 86.3882 | 3.8 |
| Letter | 98.4021 | 4.5 | 93.4477 | 4.28125 | 96.0383 | 4.45312 | 92.0952 | 4.09375 |
| Engineering document | 93.9244 | 4.04412 | 77.4854 | 3.72059 | 80.3319 | 3.88235 | 79.6807 | 3.42647 |
| Legal document | 96.689 | 4.27759 | 86.9769 | 3.87584 | 91.601 | 4.20805 | 87.8383 | 3.65552 |
| Newspaper page | 98.8733 | 4.25806 | 84.7492 | 3.90323 | 96.9963 | 4.45161 | 92.6496 | 3.51613 |
| Magazine page | 98.2145 | 4.38776 | 87.2902 | 3.97959 | 93.5934 | 4.16327 | 93.0892 | 4.02041 |
## Throughput
We benchmarked throughput using a [single long PDF](https://www.greenteapress.com/thinkpython/thinkpython.pdf).
| Method | Time per page | Time per document | VRAM used |
|---------|---------------|-------------------|---------- |
| marker | 0.18 | 43.42 | 3.17GB |
The projected throughput is 122 pages per second on an H100 - we can run 22 individual processes given the VRAM used.
## Table Conversion
Marker can extract tables from PDFs using `marker.converters.table.TableConverter`. The table extraction performance is measured by comparing the extracted HTML representation of tables against the original HTML representations using the test split of [FinTabNet](https://developer.ibm.com/exchanges/data/all/fintabnet/). The HTML representations are compared using a tree edit distance based metric to judge both structure and content. Marker detects and identifies the structure of all tables in a PDF page and achieves these scores:
| Method | Avg score | Total tables |
|------------------|-----------|--------------|
| marker | 0.816 | 99 |
| marker w/use_llm | 0.907 | 99 |
| gemini | 0.829 | 99 |
The `--use_llm` flag can significantly improve table recognition performance, as you can see.
We filter out tables that we cannot align with the ground truth, since fintabnet and our layout model have slightly different detection methods (this results in some tables being split/merged).
## Running your own benchmarks
You can benchmark the performance of marker on your machine. Install marker manually with:
```shell
git clone https://github.com/VikParuchuri/marker.git
poetry install
```
### Overall PDF Conversion
Download the benchmark data [here](https://drive.google.com/file/d/1ZSeWDo2g1y0BRLT7KnbmytV2bjWARWba/view?usp=sharing) and unzip. Then run the overall benchmark like this:
```shell
python benchmarks/overall.py --methods marker --scores heuristic,llm
```
Options:
- `--use_llm` use an llm to improve the marker results.
- `--max_rows` how many rows to process for the benchmark.
- `--methods` can be `llamaparse`, `mathpix`, `docling`, `marker`. Comma separated.
- `--scores` which scoring functions to use, can be `llm`, `heuristic`. Comma separated.
### Table Conversion
The processed FinTabNet dataset is hosted [here](https://huggingface.co/datasets/datalab-to/fintabnet-test) and is automatically downloaded. Run the benchmark with:
```shell
python benchmarks/table/table.py --max_rows 100
```
Options:
- `--use_llm` uses an llm with marker to improve accuracy.
- `--use_gemini` also benchmarks gemini 2.0 flash.
# How it works
Marker is a pipeline of deep learning models:
- Extract text, OCR if necessary (heuristics, [surya](https://github.com/VikParuchuri/surya))
- Detect page layout and find reading order ([surya](https://github.com/VikParuchuri/surya))
- Clean and format each block (heuristics, [texify](https://github.com/VikParuchuri/texify), [surya](https://github.com/VikParuchuri/surya))
- Optionally use an LLM to improve quality
- Combine blocks and postprocess complete text
It only uses models where necessary, which improves speed and accuracy.
# Limitations
PDF is a tricky format, so marker will not always work perfectly. Here are some known limitations that are on the roadmap to address:
- Very complex layouts, with nested tables and forms, may not work
- Forms may not be rendered well
Note: Passing the `--use_llm` and `--force_ocr` flags will mostly solve these issues.
# Usage and Deployment Examples
You can always run `marker` locally, but if you wanted to expose it as an API, we have a few options:
- Our platform API which is powered by `marker` and `surya` and is easy to test out - it's free to sign up, and we'll include credits, [try it out here](https://datalab.to)
- Our painless on-prem solution for commercial use, which you can [read about here](https://www.datalab.to/blog/self-serve-on-prem-licensing) and gives you privacy guarantees with high throughput inference optimizations.
- [Deployment example with Modal](./examples/README_MODAL.md) that shows you how to deploy and access `marker` through a web endpoint using [`Modal`](https://modal.com). Modal is an AI compute platform that enables developers to deploy and scale models on GPUs in minutes.
+272 -8
View File
@@ -9,20 +9,181 @@ AI-generated cross-book summaries, and engage in grounded RAG chat.
```mermaid ```mermaid
graph TD graph TD
User["Neurosurgeon (Browser)"] User["Neurosurgeon (Browser)"]
Login["Login Page\n(username + password form)"]
FE["Frontend\nVue.js 3 / Vite\n:5173"] FE["Frontend\nVue.js 3 / Vite\n:5173"]
BE["Backend\nSpring Boot 4 / Spring AI\n:8080"] BE["Backend\nSpring Boot 4 / Spring AI\n:8080"]
DB["PostgreSQL + pgvector\n(provided)"] Auth["Spring Security\nHTTP Basic Auth"]
LLM["LLM Provider\n(OpenAI / configurable)"] DB["PostgreSQL + pgvector\n(source of truth)"]
FS["File Store\nuploads/ (local disk)\nExtracted figure PNGs"]
LLM["LLM Provider\n(OpenAI)\nEmbeddings + Chat + Vision"]
User -->|HTTP| FE User -->|"First visit / unauthenticated"| Login
FE -->|REST /api/v1/...| BE Login -->|"POST credentials\n(GET /api/v1/auth/check)"| Auth
BE -->|JDBC / pgvector| DB Auth -->|"401 → back to login\n200 → app access"| Login
BE -->|Embedding + Chat API| LLM Login -->|"Authenticated"| FE
FE -->|"REST /api/v1/...\n(HTTP Basic on every request)"| Auth
Auth --> BE
BE -->|"JDBC — books, chapters,\nsections, figures, refs"| DB
BE -->|"pgvector — text chunks\n+ figure caption vectors"| DB
BE -->|"PNG read/write\n(figure extraction)"| FS
FE -->|"GET /api/v1/figures/**\n(static file serving)"| BE
BE -->|"Embedding + Chat\n+ Vision (image description)"| LLM
subgraph "Embedding Pipeline (per PDF upload)"
EP1["Parse pages → SectionEntity"]
EP2["Extract images → FigureEntity"]
EP3["Vision describe → embed caption"]
EP4["Chunk text → embed chunks"]
EP5["Link chunks ↔ figures"]
EP6["LLM enrich chunk\n(entities, facet, summary)\n→ chunk_metadata"]
EP1 --> EP2
EP1 --> EP4
EP2 --> EP3
EP4 --> EP5
EP3 --> EP5
EP4 --> EP6
end
subgraph "Retrieval Pipeline (per chat query)"
RP0["Query expansion\n(QueryExpansionService)\nlay → clinical terms"]
RP1["Text chunk search (topK=5)"]
RP2["Figure caption search (topK=3)"]
RP3["Expand chunks → ±1-page section text"]
RP4["Fetch linked figures (chunk_figure_ref)"]
RP5["Merge + deduplicate figures"]
RP6["Build labelled prompt\n[S1],[F1]… tags"]
RP7["LLM chat call"]
RP8["Citation validation\n(CitationValidatorService)\nstrip hallucinated refs"]
RP0 --> RP1
RP0 --> RP2
RP1 --> RP3
RP1 --> RP4
RP2 --> RP5
RP4 --> RP5
RP3 --> RP6
RP5 --> RP6
RP6 --> RP7
RP7 --> RP8
end
```
### Concept Retrieval Pipeline (per concept report)
Concept retrieval is an alternative to the semantic-similarity flow above. It uses the
LLM-tagged `chunk_metadata` rows written at indexing time to exhaustively gather every
chunk that *concerns* a concept (e.g. "aneurysm"), bucketed by facet. One synthesis call
per facet yields a structured, multi-section report.
```mermaid
sequenceDiagram
participant User
participant FE as Frontend
participant BE as Backend (ConceptReportService)
participant Retr as ConceptRetriever
participant DB as chunk_metadata (GIN)
participant Vec as vector_store
participant LLM
User->>FE: Click "Generate Concept Report" on topic
FE->>BE: POST /api/v1/topics/{id}/concept-reports
loop per READY book
BE->>Retr: retrieveByConcept(topicName, bookId)
Retr->>DB: WHERE entities @> [canonical]
alt SQL hits found
DB-->>Retr: chunks grouped by facet
else no match (typo / synonym)
Retr->>Vec: similaritySearch topK=30
Vec-->>Retr: chunk ids
Retr->>DB: findByChunkIdIn → group by facet
end
end
BE->>BE: merge facets across books, assign global [S#]/[F#]
loop per non-empty facet
BE->>LLM: synthesize facet section (focused prompt)
LLM-->>BE: facet markdown
end
BE->>BE: persist concept_report
BE-->>FE: { facets[], sources[] }
FE->>User: render facet-labelled report + inline figures
```
Backfill path for already-embedded books:
`POST /api/v1/admin/books/{id}/enrich` scans `vector_store` for TEXT chunks missing
`chunk_metadata` rows and enriches them in place. Idempotent — re-running is a no-op.
## Marker API Response Structure
The PDF parsing pipeline calls a local [Marker](https://github.com/VikParuchuri/marker) server (`POST /marker/upload`).
### Top-level envelope
```json
{
"format": "json",
"output": "<JSON-encoded string>"
}
```
`output` is a **JSON-encoded string** (not a nested object) and must be parsed a second time to get the document tree.
### Parsed `output` shape
```
{
"children": [ <Page block>, ... ]
}
```
### Block types
Every block shares these fields:
| Field | Type | Notes |
|------------------|-------------------|--------------------------------------------|
| `id` | string | e.g. `/page/0/Picture/2` |
| `block_type` | string | see table below |
| `html` | string | rendered HTML; may contain `<content-ref>` |
| `bbox` | `[x0,y0,x1,y1]` | bounding box in page coordinates |
| `children` | array or null | nested blocks |
| `images` | object or null | base64 PNG map (leaf image blocks only) |
| `section_hierarchy` | object | heading ancestry |
#### Known `block_type` values
| block_type | Category | Notes |
|------------------|----------|-------------------------------------------------------|
| `Page` | structure | Top-level; direct children are the page content |
| `SectionHeader` | text | Section / chapter heading |
| `Text` | text | |
| `TextInlineMath` | text | |
| `ListItem` | text | |
| `Table` | text | |
| `Code` | text | |
| `Equation` | text | |
| `Footnote` | text | |
| `Caption` | text | Usually a child of a `*Group` block |
| `PageHeader` | text | |
| `PageFooter` | text | |
| `Handwriting` | text | |
| `Picture` | image | Leaf block; `images` map holds base64 PNG keyed by ID |
| `Figure` | image | Leaf block; same as `Picture` |
| `PictureGroup` | container | Wraps one `Picture` + one `Caption` child |
| `FigureGroup` | container | Wraps one `Figure` + one `Caption` child |
### Image extraction
Images are only present on **leaf** image blocks (`Picture`, `Figure`).
Group blocks (`PictureGroup`, `FigureGroup`) have `images: null` — the base64 PNG lives on the child leaf block.
```
PictureGroup
├── Picture ← images: { "/page/0/Picture/2": "<base64 PNG>" }
└── Caption ← html: "<p>Figure 1 — ...</p>"
``` ```
## Stack ## Stack
- **Backend**: Spring Boot 4.0.5 + Spring AI 2.0.0-M4, Java 21, Maven - **Backend**: Spring Boot 4.0.5 + Spring AI 2.0.0-M4, Java 25, Maven
- **Frontend**: Vue.js 3 + Vite + TypeScript + Pinia + Axios - **Frontend**: Vue.js 3 + Vite + TypeScript + Pinia + Axios
- **Database**: PostgreSQL 16 + pgvector extension - **Database**: PostgreSQL 16 + pgvector extension
- **Auth**: HTTP Basic (single shared in-memory user) - **Auth**: HTTP Basic (single shared in-memory user)
@@ -31,7 +192,7 @@ graph TD
See [specs/001-neuro-rag-learning/quickstart.md](specs/001-neuro-rag-learning/quickstart.md) for full instructions. See [specs/001-neuro-rag-learning/quickstart.md](specs/001-neuro-rag-learning/quickstart.md) for full instructions.
### Local Dev ### Local Dev (JVM)
```bash ```bash
# Start the database # Start the database
@@ -47,8 +208,99 @@ npm install
npm run dev npm run dev
``` ```
### Native Image Build
Produces a GraalVM native binary packaged into a minimal Docker image via Jib.
**Prerequisite**: GraalVM 25 must be installed and set as `JAVA_HOME`.
```bash
# Install GraalVM 25 CE via sdkman (one-time)
sdk install java 25-graalce
sdk use java 25-graalce
# Build native executable + Docker image (requires Docker daemon)
cd backend
mvn -Pnative package jib:build -DskipTests
mvn -Pnative jib:build -Djib.to.auth.username=admin -Djib.to.auth.password=""
```
### Backend build (buildah)
**JVM image** (`Dockerfile` — Eclipse Temurin 21):
```bash
buildah build \
--platform linux/arm64 \
--tag zot.immich-ad.ovh/ai-teacher-backend:latest \
backend/
buildah login zot.immich-ad.ovh
buildah push --tls-verify=false zot.immich-ad.ovh/ai-teacher-backend:latest
```
**Native image** (`Dockerfile.native` — GraalVM 25, produces a minimal Debian-slim image):
```bash
buildah build \
--platform linux/arm64 \
--file backend/Dockerfile.native \
--tag zot.immich-ad.ovh/ai-teacher-backend-native:latest \
backend/
buildah push --tls-verify=false zot.immich-ad.ovh/ai-teacher-backend-native:latest
```
### Frontend build
```
buildah build \
--platform linux/arm64 \
--tag zot.immich-ad.ovh/ai-teacher-frontend:latest \
frontend/
buildah login zot.immich-ad.ovh
```
Push to the private repository:
```
buildah push --tls-verify=false zot.immich-ad.ovh/ai-teacher-frontend:latest
```
### Run Native Stack (Docker Compose)
```bash
# Copy and fill in secrets
cp .env.example .env
# edit .env — add OPENAI_API_KEY at minimum
# Start PostgreSQL + native backend
docker compose -f docker-compose.native.yml up
```
App available at `http://localhost:8080`.
### Build Pipeline (Native)
```mermaid
graph LR
SRC["Source Code\n(Java 25)"]
AOT["Spring Boot AOT\n(process-aot)"]
NI["GraalVM native-image\n(native-maven-plugin)"]
EXE["Native Executable\ntarget/ai-teacher-backend"]
JIB["Jib\n(jib-native-image-extension)"]
IMG["Docker Image\nai-teacher-backend:latest\n(distroless base)"]
SRC --> AOT
AOT --> NI
NI --> EXE
EXE --> JIB
JIB --> IMG
```
### Environment Variables ### Environment Variables
#### Backend
| Variable | Required | Description | | Variable | Required | Description |
|----------|----------|-------------| |----------|----------|-------------|
| `OPENAI_API_KEY` | Yes | OpenAI API key for embeddings and chat | | `OPENAI_API_KEY` | Yes | OpenAI API key for embeddings and chat |
@@ -56,3 +308,15 @@ npm run dev
| `DB_URL` | Yes | JDBC URL, e.g. `jdbc:postgresql://localhost:5432/aiteacher` | | `DB_URL` | Yes | JDBC URL, e.g. `jdbc:postgresql://localhost:5432/aiteacher` |
| `DB_USERNAME` | Yes | Database username | | `DB_USERNAME` | Yes | Database username |
| `DB_PASSWORD` | Yes | Database password | | `DB_PASSWORD` | Yes | Database password |
| `FIGURE_STORAGE_PATH` | No | Base path for uploaded PDFs and extracted figures (default: `./uploads`) |
| `UPLOAD_ENABLED` | No | Set to `false` to disable the book upload endpoint (default: `true`) |
| `DELETE_ENABLED` | No | Set to `false` to disable the book delete endpoint (default: `true`) |
#### Frontend
| Variable | Required | Description |
|----------|----------|-------------|
| `VITE_API_URL` | No | Backend API base URL (default: `/api/v1`) |
| `VITE_APP_PASSWORD` | Yes | Shared password for HTTP Basic auth (must match `APP_PASSWORD`) |
| `VITE_UPLOAD_ENABLED` | No | Set to `false` to hide the upload UI (default: `true`) |
| `VITE_DELETE_ENABLED` | No | Set to `false` to hide the delete button (default: `true`) |
+24
View File
@@ -0,0 +1,24 @@
# Java build artifacts
target/
*.class
*.jar
# Git
.git/
.gitignore
# Editor
.vscode/
.idea/
*.iml
# OS
.DS_Store
Thumbs.db
# Logs
*.log
# Environment
.env
.env.*
+25
View File
@@ -0,0 +1,25 @@
# ---- Pull Maven from its official image (avoids microdnf under QEMU) ----
FROM docker.io/library/maven:3.9.9-eclipse-temurin-21 AS maven-dist
# ---- Build stage: GraalVM 25 + Maven ----
FROM ghcr.io/graalvm/native-image-community:25 AS build
# Copy Maven from the official Maven image — no package installation needed
COPY --from=maven-dist /usr/share/maven /opt/maven
ENV PATH="/opt/maven/bin:$PATH"
WORKDIR /app
# Cache dependency resolution separately from source compilation
COPY pom.xml .
RUN mvn -Pnative dependency:resolve dependency:resolve-plugins -q
# Build native executable
COPY src ./src
RUN mvn -Pnative package -DskipTests
# ---- Runtime stage: slim Debian with glibc + libz (required by GraalVM native binary) ----
FROM docker.io/library/debian:12-slim
COPY --from=build /app/target/ai-teacher-backend /app/ai-teacher-backend
EXPOSE 8080
ENTRYPOINT ["/app/ai-teacher-backend"]
+126 -2
View File
@@ -32,6 +32,13 @@
<type>pom</type> <type>pom</type>
<scope>import</scope> <scope>import</scope>
</dependency> </dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>bom</artifactId>
<version>2.30.14</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies> </dependencies>
</dependencyManagement> </dependencyManagement>
@@ -95,12 +102,25 @@
<artifactId>spring-ai-advisors-vector-store</artifactId> <artifactId>spring-ai-advisors-vector-store</artifactId>
</dependency> </dependency>
<!-- Spring AI — PDF document reader --> <!-- Spring AI — PDF document reader (includes PDFBox transitively) -->
<dependency> <dependency>
<groupId>org.springframework.ai</groupId> <groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-pdf-document-reader</artifactId> <artifactId>spring-ai-pdf-document-reader</artifactId>
</dependency> </dependency>
<!-- PDFBox — page rendering and cropping for figure extraction -->
<dependency>
<groupId>org.apache.pdfbox</groupId>
<artifactId>pdfbox</artifactId>
<version>3.0.3</version>
</dependency>
<!-- AWS SDK v2 — S3 figure storage -->
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3</artifactId>
</dependency>
<!-- Jackson (JSON) --> <!-- Jackson (JSON) -->
<dependency> <dependency>
<groupId>com.fasterxml.jackson.core</groupId> <groupId>com.fasterxml.jackson.core</groupId>
@@ -120,15 +140,119 @@
</dependency> </dependency>
</dependencies> </dependencies>
<build> <build>
<plugins> <plugins>
<plugin>
<groupId>org.graalvm.buildtools</groupId>
<artifactId>native-maven-plugin</artifactId>
</plugin>
<plugin> <plugin>
<groupId>org.springframework.boot</groupId> <groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId> <artifactId>spring-boot-maven-plugin</artifactId>
</plugin> </plugin>
<!-- Jib — package native executable (or fat-jar) into Docker image -->
<plugin>
<groupId>com.google.cloud.tools</groupId>
<artifactId>jib-maven-plugin</artifactId>
<version>3.5.1</version>
<configuration>
<from>
<!-- distroless glibc base — includes libz + libssl needed by GraalVM native binary -->
<image>gcr.io/distroless/base-debian12</image>
</from>
<to>
<image>zot.immich-ad.ovh/ai-teacher-backend</image>
<tags>
<tag>latest</tag>
</tags>
</to>
<container>
<format>OCI</format>
<ports>
<port>8080</port>
</ports>
<!-- invoke the native binary directly — no JVM -->
<entrypoint>
<arg>/app/ai-teacher-backend</arg>
</entrypoint>
</container>
<!-- copy the GraalVM-compiled binary from target/ into /app/ -->
<extraDirectories>
<paths>
<path>
<from>${project.build.directory}</from>
<into>/app</into>
<includes>ai-teacher-backend</includes>
</path>
</paths>
<permissions>
<permission>
<file>/app/ai-teacher-backend</file>
<mode>755</mode>
</permission>
</permissions>
</extraDirectories>
</configuration>
</plugin>
</plugins> </plugins>
</build> </build>
<profiles>
<profile>
<id>native</id>
<build>
<plugins>
<!-- skip jib in native builds — use Dockerfile.native + buildah instead -->
<plugin>
<groupId>com.google.cloud.tools</groupId>
<artifactId>jib-maven-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
<!-- GraalVM native-image compilation -->
<plugin>
<groupId>org.graalvm.buildtools</groupId>
<artifactId>native-maven-plugin</artifactId>
<version>1.0.0</version>
<executions>
<execution>
<id>add-reachability-metadata</id>
<goals>
<goal>add-reachability-metadata</goal>
</goals>
</execution>
<execution>
<id>compile</id>
<goals>
<goal>compile-no-fork</goal>
</goals>
<phase>package</phase>
</execution>
</executions>
<configuration>
<imageName>ai-teacher-backend</imageName>
<buildArgs>
<buildArg>--initialize-at-build-time=org.slf4j,ch.qos.logback</buildArg>
<buildArg>-H:+ReportExceptionStackTraces</buildArg>
<buildArg>--gc=serial</buildArg>
<buildArg>-Os</buildArg>
<buildArg>-H:+RemoveUnusedSymbols</buildArg>
<buildArg>-H:-EnableLoggingFeature</buildArg>
<buildArg>-R:MaxHeapSize=128m</buildArg>
<buildArg>-R:MinHeapSize=32m</buildArg>
<!-- Limit native-image compiler RAM (build time, not runtime) -->
<buildArg>-J-Xmx8g</buildArg>
</buildArgs>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project> </project>
@@ -1,11 +1,15 @@
package com.aiteacher; package com.aiteacher;
import org.springframework.context.annotation.ImportRuntimeHints;
import org.springframework.boot.SpringApplication; import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.scheduling.annotation.EnableAsync; import org.springframework.scheduling.annotation.EnableAsync;
import com.aiteacher.config.NativeHintsConfig;
@SpringBootApplication @SpringBootApplication
@EnableAsync @EnableAsync
@ImportRuntimeHints(NativeHintsConfig.class)
public class AiTeacherApplication { public class AiTeacherApplication {
public static void main(String[] args) { public static void main(String[] args) {
@@ -0,0 +1,19 @@
package com.aiteacher.auth;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import java.security.Principal;
import java.util.Map;
@RestController
@RequestMapping("/api/v1/auth")
public class AuthController {
@GetMapping("/check")
public ResponseEntity<Map<String, String>> check(Principal principal) {
return ResponseEntity.ok(Map.of("username", principal.getName()));
}
}
@@ -1,6 +1,11 @@
package com.aiteacher.book; package com.aiteacher.book;
import com.aiteacher.document.FigureEntity;
import com.aiteacher.document.FigureRepository;
import com.aiteacher.document.MarkdownStorageService;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.http.HttpStatus; import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity; import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*; import org.springframework.web.bind.annotation.*;
import org.springframework.web.multipart.MultipartFile; import org.springframework.web.multipart.MultipartFile;
@@ -15,13 +20,25 @@ import java.util.UUID;
public class BookController { public class BookController {
private final BookService bookService; private final BookService bookService;
private final FigureRepository figureRepository;
private final MarkdownStorageService markdownStorageService;
public BookController(BookService bookService) { @Value("${app.features.upload-enabled:true}")
private boolean uploadEnabled;
@Value("${app.features.delete-enabled:true}")
private boolean deleteEnabled;
public BookController(BookService bookService, FigureRepository figureRepository,
MarkdownStorageService markdownStorageService) {
this.bookService = bookService; this.bookService = bookService;
this.figureRepository = figureRepository;
this.markdownStorageService = markdownStorageService;
} }
@PostMapping(consumes = "multipart/form-data") @PostMapping(consumes = "multipart/form-data")
public ResponseEntity<?> upload(@RequestParam("file") MultipartFile file) throws IOException { public ResponseEntity<?> upload(@RequestParam("file") MultipartFile file) throws IOException {
if (!uploadEnabled) return ResponseEntity.status(HttpStatus.METHOD_NOT_ALLOWED).build();
Book book = bookService.upload(file); Book book = bookService.upload(file);
return ResponseEntity.status(HttpStatus.ACCEPTED).body(toSummaryResponse(book)); return ResponseEntity.status(HttpStatus.ACCEPTED).body(toSummaryResponse(book));
} }
@@ -42,10 +59,52 @@ public class BookController {
@DeleteMapping("/{id}") @DeleteMapping("/{id}")
public ResponseEntity<Void> delete(@PathVariable UUID id) { public ResponseEntity<Void> delete(@PathVariable UUID id) {
if (!deleteEnabled) return ResponseEntity.status(HttpStatus.METHOD_NOT_ALLOWED).build();
bookService.delete(id); bookService.delete(id);
return ResponseEntity.noContent().build(); return ResponseEntity.noContent().build();
} }
@PostMapping("/{id}/reembed")
public ResponseEntity<Map<String, Object>> reembed(@PathVariable UUID id) {
Book book = bookService.reembed(id);
return ResponseEntity.accepted().body(Map.of(
"bookId", book.getId(),
"status", BookStatus.PROCESSING.name()
));
}
@GetMapping(value = "/{id}/pages/{pageNumber}/html", produces = MediaType.TEXT_HTML_VALUE)
public ResponseEntity<String> getPageHtml(@PathVariable UUID id,
@PathVariable int pageNumber) {
bookService.getById(id); // 404 if not found
try {
return ResponseEntity.ok(markdownStorageService.getText(id, pageNumber));
} catch (Exception e) {
return ResponseEntity.notFound().build();
}
}
@GetMapping("/{id}/figures")
public ResponseEntity<List<FigureResponse>> figures(@PathVariable UUID id) {
bookService.getById(id); // 404 if not found
List<FigureResponse> responses = figureRepository.findAllByBookId(id)
.stream()
.map(f -> toFigureResponse(id, f))
.toList();
return ResponseEntity.ok(responses);
}
private FigureResponse toFigureResponse(UUID bookId, FigureEntity f) {
String filename = f.getImagePath().substring(f.getImagePath().lastIndexOf('/') + 1);
String imageUrl = "/api/v1/figures/" + bookId + "/" + filename;
return new FigureResponse(
f.getId(), f.getLabel(), f.getCaption(),
f.getFigureType().name(), f.getPage(), imageUrl,
f.getSectionId(),
null // section title not eagerly loaded here
);
}
private Map<String, Object> toSummaryResponse(Book book) { private Map<String, Object> toSummaryResponse(Book book) {
return Map.of( return Map.of(
"id", book.getId(), "id", book.getId(),
@@ -1,41 +1,90 @@
package com.aiteacher.book; package com.aiteacher.book;
import com.aiteacher.document.*;
import com.aiteacher.enrichment.ChunkEnrichmentPipeline;
import com.aiteacher.enrichment.ChunkMetadataRepository;
import com.aiteacher.figure.FigureStorageService;
import org.slf4j.Logger; import org.slf4j.Logger;
import org.slf4j.LoggerFactory; import org.slf4j.LoggerFactory;
import org.springframework.ai.document.Document; import org.springframework.ai.document.Document;
import org.springframework.ai.reader.pdf.PagePdfDocumentReader;
import org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig;
import org.springframework.ai.vectorstore.VectorStore; import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.ai.vectorstore.filter.FilterExpressionBuilder; import org.springframework.ai.vectorstore.filter.FilterExpressionBuilder;
import org.springframework.core.io.FileSystemResource; import org.springframework.beans.factory.annotation.Value;
import org.springframework.scheduling.annotation.Async; import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.nio.file.Path; import java.nio.file.Path;
import java.util.List; import java.time.Instant;
import java.util.UUID; import java.util.*;
import java.util.regex.Pattern;
@Service @Service
public class BookEmbeddingService { public class BookEmbeddingService {
private static final Logger log = LoggerFactory.getLogger(BookEmbeddingService.class); private static final Logger log = LoggerFactory.getLogger(BookEmbeddingService.class);
// Pattern to detect diagram/figure captions
private static final Pattern CAPTION_PATTERN =
Pattern.compile("^(Figure|Fig\\.|Table|Diagram)\\s+[\\d.]+", Pattern.CASE_INSENSITIVE);
private final VectorStore vectorStore; private final VectorStore vectorStore;
private final BookRepository bookRepository; private final BookRepository bookRepository;
private final MarkerPageParser markerPageParser;
private final FigureExtractionService figureExtractionService;
private final VisionDescriptionService visionDescriptionService;
private final TextChunkingService textChunkingService;
private final ChunkFigureRefService chunkFigureRefService;
private final SectionRepository sectionRepository;
private final ChapterRepository chapterRepository;
private final FigureRepository figureRepository;
private final ChunkFigureRefRepository chunkFigureRefRepository;
private final FigureStorageService figureStorageService;
private final MarkdownStorageService markdownStorageService;
private final ChunkEnrichmentPipeline chunkEnrichmentPipeline;
private final ChunkMetadataRepository chunkMetadataRepository;
public BookEmbeddingService(VectorStore vectorStore, BookRepository bookRepository) { @Value("${app.embedding.batch-size:50}")
private int embeddingBatchSize;
@Value("${app.embedding.batch-delay-ms:1000}")
private long embeddingBatchDelayMs;
@Value("${app.embedding.skip-embedding:false}")
private boolean skipEmbedding;
public BookEmbeddingService(
VectorStore vectorStore,
BookRepository bookRepository,
MarkerPageParser markerPageParser,
FigureExtractionService figureExtractionService,
VisionDescriptionService visionDescriptionService,
TextChunkingService textChunkingService,
ChunkFigureRefService chunkFigureRefService,
SectionRepository sectionRepository,
ChapterRepository chapterRepository,
FigureRepository figureRepository,
ChunkFigureRefRepository chunkFigureRefRepository,
FigureStorageService figureStorageService,
MarkdownStorageService markdownStorageService,
ChunkEnrichmentPipeline chunkEnrichmentPipeline,
ChunkMetadataRepository chunkMetadataRepository) {
this.vectorStore = vectorStore; this.vectorStore = vectorStore;
this.bookRepository = bookRepository; this.bookRepository = bookRepository;
this.markerPageParser = markerPageParser;
this.figureExtractionService = figureExtractionService;
this.visionDescriptionService = visionDescriptionService;
this.textChunkingService = textChunkingService;
this.chunkFigureRefService = chunkFigureRefService;
this.sectionRepository = sectionRepository;
this.chapterRepository = chapterRepository;
this.figureRepository = figureRepository;
this.chunkFigureRefRepository = chunkFigureRefRepository;
this.figureStorageService = figureStorageService;
this.markdownStorageService = markdownStorageService;
this.chunkEnrichmentPipeline = chunkEnrichmentPipeline;
this.chunkMetadataRepository = chunkMetadataRepository;
} }
@Async @Async
public void embedBook(UUID bookId, String bookTitle, Path pdfPath) { public void embedBook(UUID bookId, String bookTitle, Path pdfPath) {
log.info("Starting embedding for book {} ({})", bookId, bookTitle); log.info("Starting Marker-powered embedding for book {} ({})", bookId, bookTitle);
Book book = bookRepository.findById(bookId).orElse(null); Book book = bookRepository.findById(bookId).orElse(null);
if (book == null) { if (book == null) {
@@ -47,29 +96,102 @@ public class BookEmbeddingService {
book.setStatus(BookStatus.PROCESSING); book.setStatus(BookStatus.PROCESSING);
bookRepository.save(book); bookRepository.save(book);
PagePdfDocumentReader reader = new PagePdfDocumentReader( String chapterId = bookId + "-ch1";
new FileSystemResource(pdfPath.toFile()), ChapterEntity chapter = new ChapterEntity(chapterId, bookId, 1, bookTitle, 1);
PdfDocumentReaderConfig.builder() chapterRepository.save(chapter);
.withPagesPerDocument(1)
.build()
);
List<Document> pages = reader.get(); // Step 1: Parse with Marker — split into 100-page chunks, then merge results
int pageCount = pages.size(); ParsedBook parsed = markerPageParser.parse(pdfPath);
// Enrich metadata and tag diagram captions List<PageResult> pageResults = parsed.pages();
List<Document> enriched = pages.stream()
.map(doc -> enrichDocument(doc, bookId.toString(), bookTitle))
.toList();
vectorStore.add(enriched); // Step 2: Build SectionEntity per page and persist
List<SectionEntity> sections = buildAndSaveSections(bookId, bookTitle, chapterId, pageResults);
// Step 3: Chunk and embed text
List<Document> allChunks = new ArrayList<>();
for (SectionEntity section : sections) {
allChunks.addAll(textChunkingService.chunk(section, bookTitle));
}
if (skipEmbedding) {
log.info("skip-embedding=true — skipping text embedding for book {}", bookId);
} else {
embedInBatches(allChunks, bookId);
log.info("Embedded {} text chunks for book {}", allChunks.size(), bookId);
Map<String, SectionEntity> sectionsById = new HashMap<>();
for (SectionEntity s : sections) sectionsById.put(s.getId(), s);
try {
chunkEnrichmentPipeline.enrichAndPersist(allChunks, sectionsById, bookTitle);
} catch (Exception ex) {
log.warn("Chunk enrichment failed for book {} — backfill endpoint can recover: {}",
bookId, ex.getMessage());
}
}
// Step 4: Decode pre-cropped figures from Marker output
FigureExtractionService.ExtractionResult extraction =
figureExtractionService.extract(bookId, chapterId, pageResults);
List<FigureEntity> figures = extraction.figures();
// Step 4b: Save per-page HTML to S3, replacing Marker image src with API URLs
parsed.htmlByPage().forEach((pageNumber, html) -> {
String resolved = resolveImageSrcs(html, bookId, extraction.blockIdToFigureId());
markdownStorageService.save(bookId, pageNumber, resolved);
});
log.info("Saved {} HTML pages to S3 for book {}", parsed.htmlByPage().size(), bookId);
// Step 5: Vision analysis (description + visible text) → embed figure chunks
Map<String, SectionEntity> sectionById = new HashMap<>();
for (SectionEntity s : sections) sectionById.put(s.getId(), s);
for (FigureEntity figure : figures) {
// Prefer caption extracted from the linked section's full text
if (figure.getCaption() == null || figure.getCaption().isBlank()) {
String sectionCaption = extractCaptionFromSection(sectionById.get(figure.getSectionId()));
if (sectionCaption != null) {
figure.setCaption(sectionCaption);
figureRepository.save(figure);
} else {
byte[] imageBytes = figureStorageService.getBytes(figure.getImagePath());
VisionDescriptionService.ImageAnalysis analysis =
visionDescriptionService.analyze(imageBytes, figure.getCaption());
figure.setCaption(analysis.description());
figureRepository.save(figure);
}
}
// Embedding content: description
String embeddingContent = (figure.getCaption() != null ? "\n" + figure.getCaption() : "");
String embeddingId = UUID.randomUUID().toString();
if (!skipEmbedding) {
Document figureDoc = new Document(embeddingId, embeddingContent,
buildFigureMetadata(figure, bookTitle, embeddingId, ""));
vectorStore.add(List.of(figureDoc));
figure.setCaptionEmbeddingId(UUID.fromString(embeddingId));
}
figureRepository.save(figure);
}
log.info("Embedded {} figure chunks for book {}", figures.size(), bookId);
// Step 6: Link text chunks to figures via in-text references
for (SectionEntity section : sections) {
List<Document> sectionChunks = allChunks.stream()
.filter(d -> section.getId().equals(d.getMetadata().get("section_id")))
.toList();
List<FigureEntity> sectionFigures = figures.stream()
.filter(f -> section.getId().equals(f.getSectionId()))
.toList();
chunkFigureRefService.linkChunksToFigures(sectionChunks, sectionFigures, section.getPageStart());
}
book.setStatus(BookStatus.READY); book.setStatus(BookStatus.READY);
book.setPageCount(pageCount); book.setPageCount(parsed.htmlByPage().size());
book.setProcessedAt(java.time.Instant.now()); book.setProcessedAt(Instant.now());
bookRepository.save(book); bookRepository.save(book);
log.info("Finished embedding book {} — {} pages", bookId, pageCount); log.info("Finished embedding book {} — {} pages, {} figures",
bookId, sections.size(), figures.size());
} catch (Exception ex) { } catch (Exception ex) {
log.error("Failed to embed book {}", bookId, ex); log.error("Failed to embed book {}", bookId, ex);
@@ -79,40 +201,114 @@ public class BookEmbeddingService {
} }
} }
private Document enrichDocument(Document doc, String bookId, String bookTitle) { @Transactional
String content = doc.getText();
String chunkType = detectChunkType(content);
doc.getMetadata().put("book_id", bookId);
doc.getMetadata().put("book_title", bookTitle);
doc.getMetadata().put("chunk_type", chunkType);
return doc;
}
private String detectChunkType(String content) {
if (content != null) {
for (String line : content.split("\\r?\\n")) {
if (CAPTION_PATTERN.matcher(line.trim()).find()) {
return "diagram";
}
}
}
return "text";
}
public void deleteBookChunks(UUID bookId) { public void deleteBookChunks(UUID bookId) {
log.info("Deleting vector chunks for book {}", bookId); log.info("Deleting all data for book {}", bookId);
try { try {
List<String> figureIds = figureRepository.findAllByBookId(bookId)
.stream().map(FigureEntity::getId).toList();
if (!figureIds.isEmpty()) {
chunkFigureRefRepository.deleteByFigureIdIn(figureIds);
}
figureRepository.deleteAllByBookId(bookId);
figureStorageService.deleteAll(bookId);
markdownStorageService.deleteAll(bookId);
sectionRepository.deleteAllByBookId(bookId);
chapterRepository.deleteAllByBookId(bookId);
chunkMetadataRepository.deleteByBookId(bookId);
FilterExpressionBuilder b = new FilterExpressionBuilder(); FilterExpressionBuilder b = new FilterExpressionBuilder();
vectorStore.delete(b.eq("book_id", bookId.toString()).build()); vectorStore.delete(b.eq("book_id", bookId.toString()).build());
} catch (Exception ex) { } catch (Exception ex) {
log.warn("Could not delete vector chunks for book {}: {}", bookId, ex.getMessage()); log.warn("Error during cleanup for book {}: {}", bookId, ex.getMessage());
} }
} }
private String truncate(String message, int maxLength) { // --- Private helpers ---
if (message == null) return null;
return message.length() <= maxLength ? message : message.substring(0, maxLength); private List<SectionEntity> buildAndSaveSections(UUID bookId, String bookTitle,
String chapterId,
List<PageResult> pageResults) {
List<SectionEntity> sections = new ArrayList<>();
for (PageResult page : pageResults) {
if (page.orderedText().isBlank()) continue;
String sectionId = bookId + "-p" + page.pageNumber();
String title = truncate(page.headingTitle() != null ? page.headingTitle() : "Page " + page.pageNumber(), 500);
SectionEntity section = new SectionEntity(
sectionId, chapterId, bookId,
String.valueOf(page.pageNumber()),
title,
page.pageNumber(), page.pageNumber(),
page.orderedText());
sections.add(sectionRepository.save(section));
}
return sections;
}
private void embedInBatches(List<Document> docs, UUID bookId) {
int total = docs.size();
for (int i = 0; i < total; i += embeddingBatchSize) {
List<Document> batch = docs.subList(i, Math.min(i + embeddingBatchSize, total));
vectorStore.add(batch);
log.debug("Embedded batch {}/{} for book {}",
i / embeddingBatchSize + 1, (total - 1) / embeddingBatchSize + 1, bookId);
if (i + embeddingBatchSize < total) {
try { Thread.sleep(embeddingBatchDelayMs); }
catch (InterruptedException e) { Thread.currentThread().interrupt(); }
}
}
}
private Map<String, Object> buildFigureMetadata(FigureEntity figure, String bookTitle,
String embeddingId, String imageText) {
Map<String, Object> m = new HashMap<>();
m.put("type", "FIGURE");
m.put("book_id", figure.getBookId().toString());
m.put("book_title", bookTitle);
m.put("chapter_id", figure.getChapterId() != null ? figure.getChapterId() : "");
m.put("section_id", figure.getSectionId() != null ? figure.getSectionId() : "");
m.put("figure_id", figure.getId());
m.put("figure_type", figure.getFigureType().name());
m.put("image_path", figure.getImagePath());
m.put("label", figure.getLabel() != null ? figure.getLabel() : "");
m.put("page", figure.getPage());
m.put("embedding_id", embeddingId);
m.put("image_text", imageText); // verbatim text visible inside the image
return m;
}
/**
* Replaces Marker's {@code src='{blockId}'} image attributes with resolved API URLs.
* Block IDs look like {@code /page/0/Figure/2}.
*/
private String resolveImageSrcs(String html, UUID bookId, Map<String, String> blockIdToFigureId) {
for (Map.Entry<String, String> entry : blockIdToFigureId.entrySet()) {
String blockId = entry.getKey();
String figureId = entry.getValue();
String apiUrl = "/api/v1/figures/" + bookId + "/" + figureId + ".png";
// Marker emits both single and double-quoted src attributes
html = html.replace("src='" + blockId + "'", "src='" + apiUrl + "'");
html = html.replace("src=\"" + blockId + "\"", "src=\"" + apiUrl + "\"");
}
return html;
}
private String extractCaptionFromSection(SectionEntity section) {
if (section == null) return null;
for (String line : section.getFullText().split("\n")) {
String trimmed = line.strip();
if (trimmed.startsWith("Fig.") || trimmed.startsWith("Figure") || trimmed.startsWith("Algorithm")) {
return trimmed;
}
}
return null;
}
private String truncate(String msg, int max) {
if (msg == null) return null;
return msg.length() <= max ? msg : msg.substring(0, max);
} }
} }
@@ -1,11 +1,13 @@
package com.aiteacher.book; package com.aiteacher.book;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import org.springframework.web.multipart.MultipartFile; import org.springframework.web.multipart.MultipartFile;
import java.io.IOException; import java.io.IOException;
import java.nio.file.Files; import java.nio.file.Files;
import java.nio.file.Path; import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.List; import java.util.List;
import java.util.NoSuchElementException; import java.util.NoSuchElementException;
import java.util.UUID; import java.util.UUID;
@@ -15,10 +17,15 @@ public class BookService {
private final BookRepository bookRepository; private final BookRepository bookRepository;
private final BookEmbeddingService bookEmbeddingService; private final BookEmbeddingService bookEmbeddingService;
private final Path bookStoragePath;
public BookService(BookRepository bookRepository, BookEmbeddingService bookEmbeddingService) { public BookService(
BookRepository bookRepository,
BookEmbeddingService bookEmbeddingService,
@Value("${app.figure-storage.base-path:./uploads}") String basePath) {
this.bookRepository = bookRepository; this.bookRepository = bookRepository;
this.bookEmbeddingService = bookEmbeddingService; this.bookEmbeddingService = bookEmbeddingService;
this.bookStoragePath = Paths.get(basePath).toAbsolutePath().normalize().resolve("books");
} }
public Book upload(MultipartFile file) throws IOException { public Book upload(MultipartFile file) throws IOException {
@@ -28,20 +35,35 @@ public class BookService {
} }
String title = deriveTitle(originalFilename); String title = deriveTitle(originalFilename);
Book book = new Book(title, originalFilename, file.getSize()); Book book = new Book(title, originalFilename, file.getSize());
book = bookRepository.save(book); book = bookRepository.save(book);
// Write to a temp file so the async task can read it // Persist PDF in a stable location for potential re-embedding
Path tempFile = Files.createTempFile("aiteacher-", "-" + book.getId() + ".pdf"); Files.createDirectories(bookStoragePath);
file.transferTo(tempFile.toFile()); Path pdfPath = bookStoragePath.resolve(book.getId() + ".pdf");
file.transferTo(pdfPath.toFile());
UUID bookId = book.getId(); UUID bookId = book.getId();
Path pdfPath = tempFile; bookEmbeddingService.embedBook(bookId, title, pdfPath);
String bookTitle = title; return book;
}
bookEmbeddingService.embedBook(bookId, bookTitle, pdfPath); public Book reembed(UUID id) {
Book book = bookRepository.findById(id)
.orElseThrow(() -> new NoSuchElementException("Book not found."));
if (book.getStatus() == BookStatus.PROCESSING) {
throw new IllegalStateException("Book is already being processed.");
}
Path pdfPath = bookStoragePath.resolve(id + ".pdf");
if (!Files.exists(pdfPath)) {
throw new IllegalStateException(
"Original PDF not found. Please re-upload the book before re-embedding.");
}
bookEmbeddingService.deleteBookChunks(id);
bookEmbeddingService.embedBook(id, book.getTitle(), pdfPath);
return book; return book;
} }
@@ -63,14 +85,21 @@ public class BookService {
} }
bookEmbeddingService.deleteBookChunks(id); bookEmbeddingService.deleteBookChunks(id);
// Delete the stored PDF
Path pdfPath = bookStoragePath.resolve(id + ".pdf");
try {
Files.deleteIfExists(pdfPath);
} catch (IOException ex) {
// Non-fatal — log only
}
bookRepository.deleteById(id); bookRepository.deleteById(id);
} }
private String deriveTitle(String filename) { private String deriveTitle(String filename) {
// Strip .pdf extension and replace separators with spaces
String name = filename.replaceAll("(?i)\\.pdf$", ""); String name = filename.replaceAll("(?i)\\.pdf$", "");
name = name.replaceAll("[-_]", " "); name = name.replaceAll("[-_]", " ");
// Capitalise first letter
if (!name.isEmpty()) { if (!name.isEmpty()) {
name = Character.toUpperCase(name.charAt(0)) + name.substring(1); name = Character.toUpperCase(name.charAt(0)) + name.substring(1);
} }
@@ -0,0 +1,12 @@
package com.aiteacher.book;
public record FigureResponse(
String figureId,
String label,
String caption,
String figureType,
int page,
String imageUrl,
String sectionId,
String sectionTitle
) {}
@@ -3,28 +3,21 @@ package com.aiteacher.chat;
import com.aiteacher.book.BookRepository; import com.aiteacher.book.BookRepository;
import com.aiteacher.book.BookStatus; import com.aiteacher.book.BookStatus;
import com.aiteacher.book.NoKnowledgeSourceException; import com.aiteacher.book.NoKnowledgeSourceException;
import org.slf4j.Logger; import com.aiteacher.document.FigureEntity;
import org.slf4j.LoggerFactory; import com.aiteacher.document.SectionEntity;
import com.aiteacher.retrieval.CitationValidatorService;
import com.aiteacher.retrieval.LabelledContext;
import com.aiteacher.retrieval.NeurosurgeryRetriever;
import com.aiteacher.retrieval.QueryExpansionService;
import com.aiteacher.retrieval.RetrievalResult;
import org.springframework.ai.chat.client.ChatClient; import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.document.Document;
import org.springframework.ai.vectorstore.SearchRequest;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import java.util.ArrayList; import java.util.*;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.NoSuchElementException;
import java.util.UUID;
@Service @Service
public class ChatService { public class ChatService {
private static final Logger log = LoggerFactory.getLogger(ChatService.class);
private static final String SYSTEM_PROMPT = """ private static final String SYSTEM_PROMPT = """
You are an expert neurosurgery educator assistant. Answer questions using the You are an expert neurosurgery educator assistant. Answer questions using the
medical textbook content provided to you as context. medical textbook content provided to you as context.
@@ -35,26 +28,34 @@ public class ChatService {
- Build answers from what is present: procedures, conditions, techniques, and descriptions all contribute; combine them into a rich, structured response - Build answers from what is present: procedures, conditions, techniques, and descriptions all contribute; combine them into a rich, structured response
- Use clear structure: headings, bullet points, or numbered steps where appropriate to maximize clarity - Use clear structure: headings, bullet points, or numbered steps where appropriate to maximize clarity
- Only say you cannot answer if the context is entirely unrelated to the question - Only say you cannot answer if the context is entirely unrelated to the question
- Cite sources for each major point (book title and page number from the context metadata) - Cite sources for each major claim using the reference labels from the context (e.g. [S1], [F2]). Prefer these labels over inventing page numbers, but you may also describe the source naturally if needed.
- Figures (labeled [F1], [F2], etc.) are actual images and drawings from the textbook — they will be rendered as inline illustrations in your response. Use them actively to support your explanations: reference a figure when it visually demonstrates anatomy, a surgical step, or a clinical concept you are describing.
- Maintain continuity with the conversation history - Maintain continuity with the conversation history
- Never fabricate clinical information not present in the context - Never fabricate clinical information not present in the context
"""; """;
private final ChatClient chatClient; private final ChatClient chatClient;
private final VectorStore vectorStore;
private final BookRepository bookRepository; private final BookRepository bookRepository;
private final ChatSessionRepository sessionRepository; private final ChatSessionRepository sessionRepository;
private final MessageRepository messageRepository; private final MessageRepository messageRepository;
private final NeurosurgeryRetriever retriever;
private final QueryExpansionService queryExpansionService;
private final CitationValidatorService citationValidatorService;
public ChatService(ChatClient chatClient, VectorStore vectorStore, public ChatService(ChatClient chatClient,
BookRepository bookRepository, BookRepository bookRepository,
ChatSessionRepository sessionRepository, ChatSessionRepository sessionRepository,
MessageRepository messageRepository) { MessageRepository messageRepository,
NeurosurgeryRetriever retriever,
QueryExpansionService queryExpansionService,
CitationValidatorService citationValidatorService) {
this.chatClient = chatClient; this.chatClient = chatClient;
this.vectorStore = vectorStore;
this.bookRepository = bookRepository; this.bookRepository = bookRepository;
this.sessionRepository = sessionRepository; this.sessionRepository = sessionRepository;
this.messageRepository = messageRepository; this.messageRepository = messageRepository;
this.retriever = retriever;
this.queryExpansionService = queryExpansionService;
this.citationValidatorService = citationValidatorService;
} }
public ChatSession createSession(String topicId) { public ChatSession createSession(String topicId) {
@@ -73,7 +74,11 @@ public class ChatService {
ChatSession session = sessionRepository.findById(sessionId) ChatSession session = sessionRepository.findById(sessionId)
.orElseThrow(() -> new NoSuchElementException("Session not found.")); .orElseThrow(() -> new NoSuchElementException("Session not found."));
if (!bookRepository.existsByStatus(BookStatus.READY)) { List<com.aiteacher.book.Book> readyBooks = bookRepository.findAll().stream()
.filter(b -> b.getStatus() == BookStatus.READY)
.toList();
if (readyBooks.isEmpty()) {
throw new NoKnowledgeSourceException("No books are available as knowledge sources."); throw new NoKnowledgeSourceException("No books are available as knowledge sources.");
} }
@@ -81,27 +86,40 @@ public class ChatService {
Message userMessage = new Message(sessionId, MessageRole.USER, userContent); Message userMessage = new Message(sessionId, MessageRole.USER, userContent);
messageRepository.save(userMessage); messageRepository.save(userMessage);
// Build conversation history for context // Build full question with conversation history
List<Message> history = messageRepository.findBySessionIdOrderByCreatedAtAsc(sessionId); List<Message> history = messageRepository.findBySessionIdOrderByCreatedAtAsc(sessionId);
// Build the prompt with full conversation history as context
String fullQuestion = buildQuestionWithHistory(history, userContent, session.getTopicId()); String fullQuestion = buildQuestionWithHistory(history, userContent, session.getTopicId());
var qaAdvisor = QuestionAnswerAdvisor.builder(vectorStore) // Expand only the current user question to clinical terminology for retrieval (US1).
.searchRequest(SearchRequest.builder().similarityThreshold(0.5d).topK(6).build()) // fullQuestion (which includes conversation history) is used for the LLM context prompt,
.build(); // but retrieval should be driven by a concise clinical rewrite of the actual question.
String retrievalQuery = queryExpansionService.expand(userContent).rewritten();
ChatResponse response = chatClient.prompt() // Retrieve context from all ready books using the expanded query
.advisors(qaAdvisor) List<SectionEntity> allSections = new ArrayList<>();
List<FigureEntity> allFigures = new ArrayList<>();
for (com.aiteacher.book.Book book : readyBooks) {
RetrievalResult result = retriever.retrieve(retrievalQuery, book.getId());
allSections.addAll(result.parentSections());
allFigures.addAll(result.figures());
}
// Build labelled context prompt (US2): assigns [S1]/[F1] labels to each source
LabelledContext ctx = buildContextPrompt(fullQuestion, allSections, allFigures);
// Generate answer
String rawContent = chatClient.prompt()
.system(SYSTEM_PROMPT) .system(SYSTEM_PROMPT)
.user(fullQuestion) .user(ctx.promptText())
.call() .call()
.chatResponse(); .content();
String assistantContent = response.getResult().getOutput().getText(); // Strip any citation labels not present in the retrieved context (US2)
List<Map<String, Object>> sources = extractSources(response); String assistantContent = citationValidatorService.validate(rawContent, ctx.allLabels());
// Attach sources with their ref-labels for frontend traceability
List<Map<String, Object>> sources = buildSources(allSections, allFigures);
// Persist assistant message
Message assistantMessage = new Message(sessionId, MessageRole.ASSISTANT, assistantContent); Message assistantMessage = new Message(sessionId, MessageRole.ASSISTANT, assistantContent);
assistantMessage.setSources(sources); assistantMessage.setSources(sources);
return messageRepository.save(assistantMessage); return messageRepository.save(assistantMessage);
@@ -118,24 +136,114 @@ public class ChatService {
sessionRepository.deleteById(sessionId); sessionRepository.deleteById(sessionId);
} }
// -------------------------------------------------------------------------
// Private helpers
// -------------------------------------------------------------------------
/**
* Builds the LLM context prompt, tagging each section as [S1], [S2]… and
* each figure as [F1], [F2]… so the model can cite only known sources.
*/
private LabelledContext buildContextPrompt(String question,
List<SectionEntity> sections,
List<FigureEntity> figures) {
Map<String, SectionEntity> sectionLabels = new LinkedHashMap<>();
Map<String, FigureEntity> figureLabels = new LinkedHashMap<>();
StringBuilder sb = new StringBuilder();
if (!sections.isEmpty()) {
sb.append("CONTEXT:\n\n");
for (int i = 0; i < sections.size(); i++) {
SectionEntity section = sections.get(i);
String label = "S" + (i + 1);
sectionLabels.put(label, section);
sb.append("[").append(label).append("] ")
.append(section.getTitle())
.append(", p.").append(section.getPageStart()).append("\n");
sb.append(section.getFullText()).append("\n\n");
}
}
if (!figures.isEmpty()) {
sb.append("AVAILABLE FIGURES:\n");
for (int i = 0; i < figures.size(); i++) {
FigureEntity figure = figures.get(i);
String label = "F" + (i + 1);
figureLabels.put(label, figure);
sb.append("[").append(label).append("] ")
.append(figure.getLabel() != null ? figure.getLabel() : "Figure")
.append(" (p.").append(figure.getPage()).append("): ")
.append(figure.getCaption() != null ? figure.getCaption() : "")
.append("\n");
}
sb.append("\nWhen referencing diagrams, use their label from the context (e.g. [F1]).\n\n");
}
sb.append("QUESTION:\n").append(question);
return new LabelledContext(sectionLabels, figureLabels, sb.toString());
}
private List<Map<String, Object>> buildSources(List<SectionEntity> sections,
List<FigureEntity> figures) {
List<Map<String, Object>> sources = new ArrayList<>();
for (int i = 0; i < sections.size(); i++) {
SectionEntity section = sections.get(i);
Map<String, Object> source = new LinkedHashMap<>();
source.put("type", "TEXT");
source.put("refLabel", "S" + (i + 1));
source.put("bookId", section.getBookId());
source.put("bookTitle", deriveTitleFromSection(section));
source.put("page", section.getPageStart());
source.put("chunkText", truncate(section.getFullText(), 500));
sources.add(source);
}
for (int i = 0; i < figures.size(); i++) {
FigureEntity figure = figures.get(i);
Map<String, Object> source = new LinkedHashMap<>();
source.put("type", "FIGURE");
source.put("refLabel", "F" + (i + 1));
source.put("bookId", figure.getBookId());
source.put("bookTitle", bookRepository.findById(figure.getBookId())
.map(com.aiteacher.book.Book::getTitle).orElse("Book"));
source.put("page", figure.getPage());
source.put("figureId", figure.getId());
source.put("label", figure.getLabel() != null ? figure.getLabel() : "");
source.put("caption", figure.getCaption() != null ? figure.getCaption() : "");
source.put("figureType", figure.getFigureType().name());
String filename = figure.getImagePath().substring(
figure.getImagePath().lastIndexOf('/') + 1);
source.put("imageUrl", "/api/v1/figures/" + figure.getBookId() + "/" + filename);
sources.add(source);
}
return sources;
}
private String deriveTitleFromSection(SectionEntity section) {
if (section == null) return "Book";
return bookRepository.findById(section.getBookId())
.map(com.aiteacher.book.Book::getTitle)
.orElse("Book");
}
private String buildQuestionWithHistory(List<Message> history, String currentQuestion, private String buildQuestionWithHistory(List<Message> history, String currentQuestion,
String topicId) { String topicId) {
boolean hasTopic = topicId != null && !topicId.equals("free-form"); boolean hasTopic = topicId != null && !topicId.equals("free-form");
if (history.size() <= 1) { if (history.size() <= 1) {
return hasTopic return hasTopic
? String.format("[Context: This is a question about the neurosurgery topic '%s']\n%s", ? String.format("[Context: question about neurosurgery topic '%s']\n%s",
topicId, currentQuestion) topicId, currentQuestion)
: currentQuestion; : currentQuestion;
} }
StringBuilder sb = new StringBuilder(); StringBuilder sb = new StringBuilder();
if (hasTopic) { if (hasTopic) {
sb.append(String.format("[Context: This conversation is about the neurosurgery topic '%s']\n\n", sb.append(String.format("[Context: conversation about '%s']\n\n", topicId));
topicId));
} }
sb.append("Previous conversation:\n"); sb.append("Previous conversation:\n");
// Include all messages except the last (which is the current user message just saved)
for (int i = 0; i < history.size() - 1; i++) { for (int i = 0; i < history.size() - 1; i++) {
Message msg = history.get(i); Message msg = history.get(i);
sb.append(msg.getRole().name()).append(": ").append(msg.getContent()).append("\n"); sb.append(msg.getRole().name()).append(": ").append(msg.getContent()).append("\n");
@@ -144,30 +252,8 @@ public class ChatService {
return sb.toString(); return sb.toString();
} }
private List<Map<String, Object>> extractSources(ChatResponse response) { private String truncate(String text, int maxChars) {
List<Map<String, Object>> sources = new ArrayList<>(); if (text == null) return "";
return text.length() <= maxChars ? text : text.substring(0, maxChars) + "";
if (response.getMetadata() != null) {
Object retrieved = response.getMetadata().get(QuestionAnswerAdvisor.RETRIEVED_DOCUMENTS);
if (retrieved instanceof List<?> docs) {
for (Object docObj : docs) {
if (docObj instanceof Document doc) {
Map<String, Object> metadata = doc.getMetadata();
String bookTitle = (String) metadata.get("book_title");
Object pageObj = metadata.get("page_number");
Integer page = pageObj instanceof Number n ? n.intValue() : null;
if (bookTitle != null) {
Map<String, Object> source = new HashMap<>();
source.put("bookTitle", bookTitle);
source.put("page", page);
source.put("chunkText", doc.getText());
sources.add(source);
}
}
}
}
}
return sources;
} }
} }
@@ -0,0 +1,52 @@
package com.aiteacher.concept;
import com.aiteacher.topic.Topic;
import com.aiteacher.topic.TopicRepository;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;
import java.util.List;
import java.util.Map;
import java.util.NoSuchElementException;
import java.util.UUID;
import java.util.stream.Collectors;
@RestController
@RequestMapping("/api/v1/topics/{id}/concept-reports")
public class ConceptReportController {
private final TopicRepository topicRepository;
private final ConceptReportService conceptReportService;
public ConceptReportController(TopicRepository topicRepository,
ConceptReportService conceptReportService) {
this.topicRepository = topicRepository;
this.conceptReportService = conceptReportService;
}
@PostMapping
public ResponseEntity<ConceptReportResponse> generate(
@PathVariable String id,
@RequestParam(defaultValue = "en") String language) {
Topic topic = topicRepository.findById(id)
.orElseThrow(() -> new NoSuchElementException("Topic not found."));
return ResponseEntity.ok(conceptReportService.generateReport(topic, language));
}
@GetMapping
public ResponseEntity<List<SavedConceptReportItem>> list(@PathVariable String id) {
topicRepository.findById(id)
.orElseThrow(() -> new NoSuchElementException("Topic not found."));
return ResponseEntity.ok(conceptReportService.listReports(id));
}
@GetMapping("/{reportId}")
public ResponseEntity<ConceptReportResponse> get(@PathVariable String id,
@PathVariable UUID reportId) {
topicRepository.findById(id)
.orElseThrow(() -> new NoSuchElementException("Topic not found."));
Map<String, String> topicNames = topicRepository.findAll().stream()
.collect(Collectors.toMap(Topic::getId, Topic::getName, (a, b) -> a));
return ResponseEntity.ok(conceptReportService.getReport(reportId, topicNames));
}
}
@@ -0,0 +1,48 @@
package com.aiteacher.concept;
import jakarta.persistence.*;
import java.time.Instant;
import java.util.UUID;
@Entity
@Table(name = "concept_report")
public class ConceptReportEntity {
@Id
@GeneratedValue(strategy = GenerationType.UUID)
private UUID id;
@Column(name = "topic_id", nullable = false, length = 100)
private String topicId;
@Column(name = "report_number", nullable = false)
private int reportNumber;
@Column(name = "facets_json", nullable = false, columnDefinition = "TEXT")
private String facetsJson;
@Column(name = "sources_json", nullable = false, columnDefinition = "TEXT")
private String sourcesJson;
@Column(name = "generated_at", nullable = false)
private Instant generatedAt;
protected ConceptReportEntity() {}
public ConceptReportEntity(String topicId, int reportNumber, String facetsJson,
String sourcesJson, Instant generatedAt) {
this.topicId = topicId;
this.reportNumber = reportNumber;
this.facetsJson = facetsJson;
this.sourcesJson = sourcesJson;
this.generatedAt = generatedAt;
}
public UUID getId() { return id; }
public String getTopicId() { return topicId; }
public int getReportNumber() { return reportNumber; }
public String getFacetsJson() { return facetsJson; }
public String getSourcesJson() { return sourcesJson; }
public Instant getGeneratedAt() { return generatedAt; }
}
@@ -0,0 +1,13 @@
package com.aiteacher.concept;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.stereotype.Repository;
import java.util.List;
import java.util.UUID;
@Repository
public interface ConceptReportRepository extends JpaRepository<ConceptReportEntity, UUID> {
long countByTopicId(String topicId);
List<ConceptReportEntity> findByTopicIdOrderByReportNumberAsc(String topicId);
}
@@ -0,0 +1,24 @@
package com.aiteacher.concept;
import com.aiteacher.topic.TopicSummaryResponse.SourceReference;
import java.time.Instant;
import java.util.List;
import java.util.UUID;
public record ConceptReportResponse(
UUID id,
int reportNumber,
String topicId,
String topicName,
List<FacetSection> facets,
List<SourceReference> sources,
Instant generatedAt
) {
public record FacetSection(
String facetKey,
String title,
String markdown,
List<String> refLabels
) {}
}
@@ -0,0 +1,299 @@
package com.aiteacher.concept;
import com.aiteacher.book.Book;
import com.aiteacher.book.BookRepository;
import com.aiteacher.book.BookStatus;
import com.aiteacher.book.NoKnowledgeSourceException;
import com.aiteacher.document.FigureEntity;
import com.aiteacher.document.SectionEntity;
import com.aiteacher.enrichment.ConceptFacet;
import com.aiteacher.topic.Topic;
import com.aiteacher.topic.TopicSummaryResponse.SourceReference;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.stereotype.Service;
import java.time.Instant;
import java.util.*;
@Service
public class ConceptReportService {
private static final Logger log = LoggerFactory.getLogger(ConceptReportService.class);
private static final String SYSTEM_PROMPT = """
You are an expert neurosurgery educator. You write focused, facet-specific sections of
a structured concept report for highly experienced neurosurgeons. The audience wants
concise, clinically relevant teaching.
When writing a facet section:
- Stick strictly to the facet you are asked about (e.g. definition, complications).
- Cite claims using ONLY the reference labels provided in the context.
Do not invent page numbers, section titles, or labels not present in CONTEXT.
- Citation format: each citation must be a SINGLE label per bracket — write `[S1], [S2]` or
`[S3] [F2]`. NEVER combine labels inside one bracket (no `[S1 S2]`, `[S1, S2]`, `[S1 2]`).
- Figures ([F#]) are actual images that will be rendered inline — reference them when they
visually support your explanation.
- If CONTEXT is insufficient for the requested facet, write exactly:
"The uploaded books do not contain sufficient information on this aspect."
- Never hallucinate clinical information outside the provided context.
""";
private final ChatClient chatClient;
private final BookRepository bookRepository;
private final ConceptRetriever conceptRetriever;
private final ConceptReportRepository reportRepository;
private final ObjectMapper objectMapper;
public ConceptReportService(ChatClient chatClient,
BookRepository bookRepository,
ConceptRetriever conceptRetriever,
ConceptReportRepository reportRepository,
ObjectMapper objectMapper) {
this.chatClient = chatClient;
this.bookRepository = bookRepository;
this.conceptRetriever = conceptRetriever;
this.reportRepository = reportRepository;
this.objectMapper = objectMapper;
}
public ConceptReportResponse generateReport(Topic topic, String language) {
List<Book> readyBooks = bookRepository.findAll().stream()
.filter(b -> b.getStatus() == BookStatus.READY)
.toList();
if (readyBooks.isEmpty()) {
throw new NoKnowledgeSourceException(
"No books are available as knowledge sources. Please upload and process at least one book.");
}
Map<ConceptFacet, MergedFacet> merged = new EnumMap<>(ConceptFacet.class);
for (Book book : readyBooks) {
ConceptRetrievalResult result = conceptRetriever.retrieveByConcept(topic.getName(), book.getId());
result.byFacet().forEach((facet, bundle) -> merged
.computeIfAbsent(facet, k -> new MergedFacet())
.add(bundle));
}
// Global, deduplicated sources across all facets
List<SectionEntity> globalSections = new ArrayList<>();
Set<String> seenSections = new LinkedHashSet<>();
List<FigureEntity> globalFigures = new ArrayList<>();
Set<String> seenFigures = new LinkedHashSet<>();
for (MergedFacet mf : merged.values()) {
for (SectionEntity s : mf.sections) if (seenSections.add(s.getId())) globalSections.add(s);
for (FigureEntity f : mf.figures) if (seenFigures.add(f.getId())) globalFigures.add(f);
}
// Global label maps: section id -> "S#", figure id -> "F#"
Map<String, String> sectionLabel = new HashMap<>();
for (int i = 0; i < globalSections.size(); i++) {
sectionLabel.put(globalSections.get(i).getId(), "S" + (i + 1));
}
Map<String, String> figureLabel = new HashMap<>();
for (int i = 0; i < globalFigures.size(); i++) {
figureLabel.put(globalFigures.get(i).getId(), "F" + (i + 1));
}
List<ConceptReportResponse.FacetSection> facetSections = new ArrayList<>();
// Preserve enum declaration order for consistent UI rendering
for (ConceptFacet facet : ConceptFacet.values()) {
MergedFacet mf = merged.get(facet);
if (mf == null || mf.isEmpty()) continue;
if (facet == ConceptFacet.OTHER) continue; // skip OTHER bucket in the rendered report
String prompt = buildFacetPrompt(topic, facet, mf, sectionLabel, figureLabel, language);
String markdown = chatClient.prompt()
.system(SYSTEM_PROMPT)
.user(prompt)
.call()
.content();
List<String> refs = collectRefs(mf, sectionLabel, figureLabel);
facetSections.add(new ConceptReportResponse.FacetSection(
facet.name(), facet.displayTitle(), markdown != null ? markdown : "", refs));
}
List<SourceReference> sources = buildSources(globalSections, globalFigures, readyBooks);
Instant generatedAt = Instant.now();
int reportNumber = (int) reportRepository.countByTopicId(topic.getId()) + 1;
ConceptReportEntity entity = new ConceptReportEntity(
topic.getId(), reportNumber,
serialize(facetSections), serialize(sources), generatedAt);
entity = reportRepository.save(entity);
return new ConceptReportResponse(
entity.getId(), reportNumber, topic.getId(), topic.getName(),
facetSections, sources, generatedAt);
}
public List<SavedConceptReportItem> listReports(String topicId) {
return reportRepository.findByTopicIdOrderByReportNumberAsc(topicId).stream()
.map(e -> new SavedConceptReportItem(e.getId(), e.getReportNumber(), e.getGeneratedAt()))
.toList();
}
public ConceptReportResponse getReport(UUID reportId, Map<String, String> topicNamesById) {
ConceptReportEntity entity = reportRepository.findById(reportId)
.orElseThrow(() -> new NoSuchElementException("Concept report not found."));
List<ConceptReportResponse.FacetSection> facets = deserializeFacets(entity.getFacetsJson());
List<SourceReference> sources = deserializeSources(entity.getSourcesJson());
String topicName = topicNamesById.getOrDefault(entity.getTopicId(), entity.getTopicId());
return new ConceptReportResponse(
entity.getId(), entity.getReportNumber(), entity.getTopicId(), topicName,
facets, sources, entity.getGeneratedAt());
}
private String buildFacetPrompt(Topic topic, ConceptFacet facet, MergedFacet mf,
Map<String, String> sectionLabel,
Map<String, String> figureLabel,
String language) {
StringBuilder sb = new StringBuilder();
sb.append("CONCEPT: ").append(topic.getName()).append("\n");
sb.append("FACET: ").append(facet.displayTitle()).append("\n\n");
sb.append("CONTEXT:\n\n");
for (SectionEntity s : mf.sections) {
String label = sectionLabel.get(s.getId());
sb.append("[").append(label).append("] ")
.append(s.getTitle() != null ? s.getTitle() : "")
.append(", p.").append(s.getPageStart()).append("\n");
sb.append(s.getFullText()).append("\n\n");
}
if (!mf.figures.isEmpty()) {
sb.append("AVAILABLE FIGURES:\n");
for (FigureEntity f : mf.figures) {
String label = figureLabel.get(f.getId());
sb.append("[").append(label).append("] ")
.append(f.getLabel() != null ? f.getLabel() : "Figure")
.append(" (p.").append(f.getPage()).append("): ")
.append(f.getCaption() != null ? f.getCaption() : "")
.append("\n");
}
sb.append("\n");
}
sb.append("Write the ").append(facet.displayTitle()).append(" section of a concept report on \"")
.append(topic.getName())
.append("\". Stay strictly within this facet. Use the [S#]/[F#] labels above for citations.");
if ("th".equalsIgnoreCase(language)) {
sb.append("\n\nIMPORTANT: Write the narrative in Thai. ")
.append("Keep all medical, anatomical, surgical, pharmacological, and clinical ")
.append("terminology in English (e.g., cerebellopontine angle, glioblastoma, craniotomy, ")
.append("dexamethasone). Do NOT translate disease names, anatomical structures, drug names, ")
.append("procedures, eponyms, or imaging modalities. Translate only connective prose, ")
.append("explanations, and general descriptions. Citation labels [S#]/[F#] stay unchanged. ")
.append("The sentinel string for insufficient context must remain exactly: ")
.append("\"The uploaded books do not contain sufficient information on this aspect.\"");
}
return sb.toString();
}
private List<String> collectRefs(MergedFacet mf,
Map<String, String> sectionLabel,
Map<String, String> figureLabel) {
List<String> refs = new ArrayList<>();
for (SectionEntity s : mf.sections) {
String l = sectionLabel.get(s.getId());
if (l != null) refs.add(l);
}
for (FigureEntity f : mf.figures) {
String l = figureLabel.get(f.getId());
if (l != null) refs.add(l);
}
return refs;
}
private List<SourceReference> buildSources(List<SectionEntity> sections,
List<FigureEntity> figures,
List<Book> readyBooks) {
List<SourceReference> sources = new ArrayList<>();
for (int i = 0; i < sections.size(); i++) {
SectionEntity s = sections.get(i);
Book book = findBook(readyBooks, s.getBookId());
String title = book != null ? book.getTitle() : "Book";
String bookId = book != null ? book.getId().toString() : null;
sources.add(new SourceReference(
"TEXT", "S" + (i + 1), bookId, title, s.getPageStart(),
truncate(s.getFullText(), 500), null, null, null, null, null));
}
for (int i = 0; i < figures.size(); i++) {
FigureEntity f = figures.get(i);
Book book = findBook(readyBooks, f.getBookId());
String title = book != null ? book.getTitle() : "Book";
String bookId = book != null ? book.getId().toString() : null;
String filename = f.getImagePath().substring(f.getImagePath().lastIndexOf('/') + 1);
String imageUrl = "/api/v1/figures/" + f.getBookId() + "/" + filename;
sources.add(new SourceReference(
"FIGURE", "F" + (i + 1), bookId, title, f.getPage(),
null, f.getId(), f.getLabel(), f.getCaption(),
f.getFigureType().name(), imageUrl));
}
return sources;
}
private Book findBook(List<Book> books, UUID bookId) {
return books.stream().filter(b -> b.getId().equals(bookId)).findFirst().orElse(null);
}
private String serialize(Object value) {
try {
return objectMapper.writeValueAsString(value);
} catch (JsonProcessingException e) {
log.warn("Failed to serialize concept report field", e);
return "[]";
}
}
private List<ConceptReportResponse.FacetSection> deserializeFacets(String json) {
try {
return objectMapper.readValue(json,
objectMapper.getTypeFactory().constructCollectionType(
List.class, ConceptReportResponse.FacetSection.class));
} catch (JsonProcessingException e) {
log.warn("Failed to deserialize facets", e);
return List.of();
}
}
private List<SourceReference> deserializeSources(String json) {
try {
return objectMapper.readValue(json,
objectMapper.getTypeFactory().constructCollectionType(
List.class, SourceReference.class));
} catch (JsonProcessingException e) {
log.warn("Failed to deserialize sources", e);
return List.of();
}
}
private String truncate(String text, int maxChars) {
if (text == null) return "";
return text.length() <= maxChars ? text : text.substring(0, maxChars) + "";
}
private static class MergedFacet {
final List<SectionEntity> sections = new ArrayList<>();
final List<FigureEntity> figures = new ArrayList<>();
final Set<String> sectionIds = new HashSet<>();
final Set<String> figureIds = new HashSet<>();
void add(FacetBundle bundle) {
for (SectionEntity s : bundle.sections()) {
if (sectionIds.add(s.getId())) sections.add(s);
}
for (FigureEntity f : bundle.figures()) {
if (figureIds.add(f.getId())) figures.add(f);
}
}
boolean isEmpty() { return sections.isEmpty() && figures.isEmpty(); }
}
}
@@ -0,0 +1,10 @@
package com.aiteacher.concept;
import com.aiteacher.enrichment.ConceptFacet;
import java.util.Map;
public record ConceptRetrievalResult(
Map<ConceptFacet, FacetBundle> byFacet,
boolean usedFallback
) {}
@@ -0,0 +1,163 @@
package com.aiteacher.concept;
import com.aiteacher.document.*;
import com.aiteacher.enrichment.ChunkMetadataEntity;
import com.aiteacher.enrichment.ChunkMetadataRepository;
import com.aiteacher.enrichment.ConceptFacet;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.ai.document.Document;
import org.springframework.ai.vectorstore.SearchRequest;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.ai.vectorstore.filter.FilterExpressionBuilder;
import org.springframework.stereotype.Service;
import java.util.*;
import java.util.stream.Collectors;
@Service
public class ConceptRetriever {
private static final Logger log = LoggerFactory.getLogger(ConceptRetriever.class);
private static final int FALLBACK_TOP_K = 30;
private static final int FIGURE_TOP_K = 6;
private final ChunkMetadataRepository metadataRepository;
private final VectorStore vectorStore;
private final SectionRepository sectionRepository;
private final FigureRepository figureRepository;
private final ChunkFigureRefRepository chunkFigureRefRepository;
public ConceptRetriever(ChunkMetadataRepository metadataRepository,
VectorStore vectorStore,
SectionRepository sectionRepository,
FigureRepository figureRepository,
ChunkFigureRefRepository chunkFigureRefRepository) {
this.metadataRepository = metadataRepository;
this.vectorStore = vectorStore;
this.sectionRepository = sectionRepository;
this.figureRepository = figureRepository;
this.chunkFigureRefRepository = chunkFigureRefRepository;
}
public ConceptRetrievalResult retrieveByConcept(String conceptKeyword, UUID bookId) {
String canonical = canonicalise(conceptKeyword);
List<ChunkMetadataEntity> hits = metadataRepository
.findByBookIdAndEntityContains(bookId, canonical);
boolean fallback = false;
if (hits.isEmpty()) {
log.debug("Entity match miss for '{}' in book {} — falling back to vector search", canonical, bookId);
fallback = true;
hits = vectorFallback(conceptKeyword, bookId);
}
if (hits.isEmpty()) {
return new ConceptRetrievalResult(Map.of(), fallback);
}
List<FigureEntity> semanticFigures = semanticFigureSearch(conceptKeyword, bookId);
Map<ConceptFacet, List<ChunkMetadataEntity>> grouped = hits.stream()
.collect(Collectors.groupingBy(
ChunkMetadataEntity::getFacet,
LinkedHashMap::new,
Collectors.toList()));
Map<ConceptFacet, FacetBundle> result = new LinkedHashMap<>();
for (Map.Entry<ConceptFacet, List<ChunkMetadataEntity>> entry : grouped.entrySet()) {
result.put(entry.getKey(), hydrate(entry.getValue(), semanticFigures));
}
return new ConceptRetrievalResult(result, fallback);
}
private List<ChunkMetadataEntity> vectorFallback(String query, UUID bookId) {
FilterExpressionBuilder b = new FilterExpressionBuilder();
List<Document> textHits = vectorStore.similaritySearch(
SearchRequest.builder()
.query(query)
.topK(FALLBACK_TOP_K)
.filterExpression(b.and(
b.eq("type", "TEXT"),
b.eq("book_id", bookId.toString())
).build())
.build()
);
List<UUID> chunkIds = textHits.stream()
.map(d -> {
try { return UUID.fromString(d.getId()); }
catch (Exception e) { return null; }
})
.filter(Objects::nonNull)
.toList();
if (chunkIds.isEmpty()) return List.of();
return metadataRepository.findByChunkIdIn(chunkIds);
}
private FacetBundle hydrate(List<ChunkMetadataEntity> chunks, List<FigureEntity> semanticFigures) {
List<String> sectionIds = chunks.stream()
.map(ChunkMetadataEntity::getSectionId)
.distinct()
.toList();
List<SectionEntity> sections = sectionIds.isEmpty()
? List.of()
: sectionRepository.findAllById(sectionIds);
List<UUID> chunkIds = chunks.stream().map(ChunkMetadataEntity::getChunkId).toList();
List<String> linkedFigureIds = chunkFigureRefRepository.findByChunkIdIn(chunkIds)
.stream()
.map(ChunkFigureRefEntity::getFigureId)
.distinct()
.toList();
List<FigureEntity> linkedFigures = linkedFigureIds.isEmpty()
? List.of()
: figureRepository.findAllById(linkedFigureIds);
// Merge caption-semantic-search figures with chunk-linked figures (dedupe by id, linked first)
Map<String, FigureEntity> merged = new LinkedHashMap<>();
linkedFigures.forEach(f -> merged.put(f.getId(), f));
semanticFigures.forEach(f -> merged.putIfAbsent(f.getId(), f));
List<String> summaries = chunks.stream()
.map(ChunkMetadataEntity::getSummary)
.filter(s -> s != null && !s.isBlank())
.distinct()
.toList();
return new FacetBundle(sections, new ArrayList<>(merged.values()), summaries);
}
private List<FigureEntity> semanticFigureSearch(String query, UUID bookId) {
FilterExpressionBuilder b = new FilterExpressionBuilder();
List<Document> figureHits = vectorStore.similaritySearch(
SearchRequest.builder()
.query(query)
.topK(FIGURE_TOP_K)
.filterExpression(b.and(
b.eq("type", "FIGURE"),
b.eq("book_id", bookId.toString())
).build())
.build()
);
List<String> figureIds = figureHits.stream()
.map(d -> (String) d.getMetadata().get("figure_id"))
.filter(Objects::nonNull)
.toList();
return figureIds.isEmpty() ? List.of() : figureRepository.findAllById(figureIds);
}
static String canonicalise(String raw) {
if (raw == null) return "";
String s = raw.trim().toLowerCase(Locale.ROOT);
if (s.endsWith("ies") && s.length() > 3) {
s = s.substring(0, s.length() - 3) + "y";
} else if (s.endsWith("es") && s.length() > 2) {
s = s.substring(0, s.length() - 2);
} else if (s.endsWith("s") && s.length() > 1 && !s.endsWith("ss")) {
s = s.substring(0, s.length() - 1);
}
return s;
}
}
@@ -0,0 +1,12 @@
package com.aiteacher.concept;
import com.aiteacher.document.FigureEntity;
import com.aiteacher.document.SectionEntity;
import java.util.List;
public record FacetBundle(
List<SectionEntity> sections,
List<FigureEntity> figures,
List<String> chunkSummaries
) {}
@@ -0,0 +1,10 @@
package com.aiteacher.concept;
import java.time.Instant;
import java.util.UUID;
public record SavedConceptReportItem(
UUID id,
int reportNumber,
Instant generatedAt
) {}
@@ -0,0 +1,37 @@
package com.aiteacher.config;
import com.aiteacher.figure.FigureStorageService;
import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.server.ResponseStatusException;
import jakarta.servlet.http.HttpServletResponse;
import java.io.IOException;
/**
* Serves figure images by redirecting to a presigned S3 URL.
* The key stored in DB is the full S3 object key, e.g. "figures/{bookId}/{figureId}.png".
*/
@RestController
@RequestMapping("/api/v1/figures")
public class FigureStorageConfig {
private final FigureStorageService figureStorageService;
public FigureStorageConfig(FigureStorageService figureStorageService) {
this.figureStorageService = figureStorageService;
}
@GetMapping("/{bookId}/{filename}")
public void serve(@PathVariable String bookId,
@PathVariable String filename,
HttpServletResponse response) throws IOException {
String key = "figures/" + bookId + "/" + filename;
try {
String url = figureStorageService.presignedUrl(key);
response.sendRedirect(url);
} catch (Exception ex) {
throw new ResponseStatusException(HttpStatus.NOT_FOUND, "Figure not found: " + key);
}
}
}
@@ -0,0 +1,30 @@
package com.aiteacher.config;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.client.JdkClientHttpRequestFactory;
import org.springframework.web.client.RestClient;
import java.net.http.HttpClient;
@Configuration
public class MarkerConfig {
@Value("${app.marker.base-url:http://localhost:8000}")
private String markerBaseUrl;
@Bean
RestClient markerRestClient() {
// Use the JDK HTTP client with no timeout — Marker conversions can take several minutes.
HttpClient httpClient = HttpClient.newBuilder()
.build();
JdkClientHttpRequestFactory factory = new JdkClientHttpRequestFactory(httpClient);
// No read timeout set: JDK HTTP client defaults to no deadline.
return RestClient.builder()
.baseUrl(markerBaseUrl)
.requestFactory(factory)
.build();
}
}
@@ -0,0 +1,92 @@
package com.aiteacher.config;
import org.springframework.aot.hint.MemberCategory;
import org.springframework.aot.hint.RuntimeHints;
import org.springframework.aot.hint.RuntimeHintsRegistrar;
import org.springframework.aot.hint.TypeReference;
import java.util.List;
/**
* GraalVM native-image runtime hints for third-party libraries that use reflection
* or classpath resource scanning not covered by Spring Boot's AOT processor.
*
* Registered via @ImportRuntimeHints on AiTeacherApplication.
*/
public class NativeHintsConfig implements RuntimeHintsRegistrar {
@Override
public void registerHints(RuntimeHints hints, ClassLoader classLoader) {
// PDFBox — font and encoding resources loaded via classpath scanning at runtime
hints.resources().registerPattern("org/apache/pdfbox/resources/*");
hints.resources().registerPattern("org/apache/pdfbox/resources/afm/*");
hints.resources().registerPattern("org/apache/pdfbox/resources/cmap/*");
hints.resources().registerPattern("org/apache/pdfbox/resources/glyphlist/*");
hints.resources().registerPattern("org/apache/pdfbox/resources/icc/*");
hints.resources().registerPattern("org/apache/pdfbox/resources/ttf/*");
hints.resources().registerPattern("org/apache/pdfbox/resources/version.properties");
// PDFBox — font encoding classes instantiated via reflection
hints.reflection().registerType(
org.apache.pdfbox.pdmodel.font.encoding.GlyphList.class,
MemberCategory.INVOKE_PUBLIC_CONSTRUCTORS,
MemberCategory.INVOKE_PUBLIC_METHODS
);
hints.reflection().registerType(
org.apache.pdfbox.pdmodel.font.encoding.WinAnsiEncoding.class,
MemberCategory.INVOKE_PUBLIC_CONSTRUCTORS
);
hints.reflection().registerType(
org.apache.pdfbox.pdmodel.font.encoding.MacRomanEncoding.class,
MemberCategory.INVOKE_PUBLIC_CONSTRUCTORS
);
hints.reflection().registerType(
org.apache.pdfbox.pdmodel.font.encoding.MacExpertEncoding.class,
MemberCategory.INVOKE_PUBLIC_CONSTRUCTORS
);
hints.reflection().registerType(
org.apache.pdfbox.pdmodel.font.encoding.StandardEncoding.class,
MemberCategory.INVOKE_PUBLIC_CONSTRUCTORS
);
// JPA / Hibernate — array types used in entity mappings
hints.reflection().registerType(java.util.UUID[].class, MemberCategory.INVOKE_PUBLIC_CONSTRUCTORS);
// JBoss Logging — message logger implementations generated by annotation processor.
// JBoss Logging uses reflection to look up the generated *_$logger class by name.
registerJBossLogger(hints, "org.hibernate.jpa.internal.JpaLogger_$logger");
registerJBossLogger(hints, "org.hibernate.internal.CoreMessageLogger_$logger");
registerJBossLogger(hints, "org.hibernate.internal.EntityManagerMessageLogger_$logger");
// AWS SDK v2 — HTTP client and SdkPojo serialization
hints.resources().registerPattern("software/amazon/awssdk/global/handlers/execution.interceptors");
hints.resources().registerPattern("software/amazon/awssdk/services/s3/execution.interceptors");
hints.resources().registerPattern("codegen-resources/s3/*");
hints.reflection().registerType(
software.amazon.awssdk.services.s3.S3Client.class,
MemberCategory.INVOKE_PUBLIC_METHODS
);
// Jackson deserialization of records persisted as JSON in DB columns.
// These are reached only via ObjectMapper.readValue in services, so Spring's
// BindingReflectionHintsRegistrar does not auto-discover all accessors.
for (Class<?> type : List.of(
com.aiteacher.topic.TopicSummaryResponse.class,
com.aiteacher.topic.TopicSummaryResponse.SourceReference.class,
com.aiteacher.concept.ConceptReportResponse.class,
com.aiteacher.concept.ConceptReportResponse.FacetSection.class
)) {
hints.reflection().registerType(type,
MemberCategory.INVOKE_DECLARED_CONSTRUCTORS,
MemberCategory.INVOKE_DECLARED_METHODS);
}
}
private void registerJBossLogger(RuntimeHints hints, String className) {
hints.reflection().registerType(
TypeReference.of(className),
MemberCategory.INVOKE_PUBLIC_CONSTRUCTORS,
MemberCategory.INVOKE_PUBLIC_METHODS
);
}
}
@@ -20,7 +20,9 @@ public class SecurityConfig {
@Bean @Bean
public SecurityFilterChain filterChain(HttpSecurity http) throws Exception { public SecurityFilterChain filterChain(HttpSecurity http) throws Exception {
http http
.authorizeHttpRequests(auth -> auth.anyRequest().authenticated()) .authorizeHttpRequests(auth -> auth
.requestMatchers("/api/v1/figures/**").permitAll()
.anyRequest().authenticated())
.httpBasic(Customizer.withDefaults()) .httpBasic(Customizer.withDefaults())
.csrf(AbstractHttpConfigurer::disable); .csrf(AbstractHttpConfigurer::disable);
return http.build(); return http.build();
@@ -28,9 +30,10 @@ public class SecurityConfig {
@Bean @Bean
public UserDetailsService userDetailsService( public UserDetailsService userDetailsService(
@Value("${app.auth.username}") String username,
@Value("${app.auth.password}") String password) { @Value("${app.auth.password}") String password) {
UserDetails user = User.builder() UserDetails user = User.builder()
.username("neurosurgeon") .username(username)
.password("{noop}" + password) .password("{noop}" + password)
.roles("USER") .roles("USER")
.build(); .build();
@@ -0,0 +1,47 @@
package com.aiteacher.document;
import jakarta.persistence.*;
import java.time.Instant;
import java.util.UUID;
@Entity
@Table(name = "chapter")
public class ChapterEntity {
@Id
@Column(name = "id", length = 200)
private String id;
@Column(name = "book_id", nullable = false)
private UUID bookId;
@Column(name = "number", nullable = false)
private int number;
@Column(name = "title", length = 500)
private String title;
@Column(name = "page_start")
private Integer pageStart;
@Column(name = "created_at", nullable = false)
private Instant createdAt;
public ChapterEntity() {}
public ChapterEntity(String id, UUID bookId, int number, String title, Integer pageStart) {
this.id = id;
this.bookId = bookId;
this.number = number;
this.title = title;
this.pageStart = pageStart;
this.createdAt = Instant.now();
}
public String getId() { return id; }
public UUID getBookId() { return bookId; }
public int getNumber() { return number; }
public String getTitle() { return title; }
public Integer getPageStart() { return pageStart; }
public Instant getCreatedAt() { return createdAt; }
}
@@ -0,0 +1,9 @@
package com.aiteacher.document;
import org.springframework.data.jpa.repository.JpaRepository;
import java.util.UUID;
public interface ChapterRepository extends JpaRepository<ChapterEntity, String> {
void deleteAllByBookId(UUID bookId);
}
@@ -0,0 +1,58 @@
package com.aiteacher.document;
import jakarta.persistence.*;
import java.io.Serializable;
import java.util.Objects;
import java.util.UUID;
@Entity
@Table(name = "chunk_figure_ref")
@IdClass(ChunkFigureRefEntity.PK.class)
public class ChunkFigureRefEntity {
@Id
@Column(name = "chunk_id", nullable = false)
private UUID chunkId;
@Id
@Column(name = "figure_id", nullable = false, length = 200)
private String figureId;
@Column(name = "mention_page")
private Integer mentionPage;
public ChunkFigureRefEntity() {}
public ChunkFigureRefEntity(UUID chunkId, String figureId, Integer mentionPage) {
this.chunkId = chunkId;
this.figureId = figureId;
this.mentionPage = mentionPage;
}
public UUID getChunkId() { return chunkId; }
public String getFigureId() { return figureId; }
public Integer getMentionPage() { return mentionPage; }
public static class PK implements Serializable {
private UUID chunkId;
private String figureId;
public PK() {}
public PK(UUID chunkId, String figureId) {
this.chunkId = chunkId;
this.figureId = figureId;
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (!(o instanceof PK pk)) return false;
return Objects.equals(chunkId, pk.chunkId) && Objects.equals(figureId, pk.figureId);
}
@Override
public int hashCode() {
return Objects.hash(chunkId, figureId);
}
}
}
@@ -0,0 +1,18 @@
package com.aiteacher.document;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.Query;
import org.springframework.data.repository.query.Param;
import java.util.List;
import java.util.UUID;
public interface ChunkFigureRefRepository extends JpaRepository<ChunkFigureRefEntity, ChunkFigureRefEntity.PK> {
@Query("SELECT r FROM ChunkFigureRefEntity r WHERE r.chunkId IN :chunkIds")
List<ChunkFigureRefEntity> findByChunkIdIn(@Param("chunkIds") List<UUID> chunkIds);
@Query("DELETE FROM ChunkFigureRefEntity r WHERE r.figureId IN :figureIds")
@org.springframework.data.jpa.repository.Modifying
void deleteByFigureIdIn(@Param("figureIds") List<String> figureIds);
}
@@ -0,0 +1,62 @@
package com.aiteacher.document;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.ai.document.Document;
import org.springframework.stereotype.Service;
import java.util.List;
import java.util.UUID;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
/**
* Scans chunk text for "Fig. X" and "Figure X" references and persists
* ChunkFigureRefEntity rows linking that chunk to its referenced figures.
*/
@Service
public class ChunkFigureRefService {
private static final Logger log = LoggerFactory.getLogger(ChunkFigureRefService.class);
// Matches: "Fig. 12-4", "Fig. 12.4", "Fig 12", "Figure 12-4", etc.
private static final Pattern REF_PATTERN =
Pattern.compile("(?i)\\b(Fig\\.?|Figure)\\s+(\\d+[\\-.\\d]*)");
private final ChunkFigureRefRepository refRepository;
public ChunkFigureRefService(ChunkFigureRefRepository refRepository) {
this.refRepository = refRepository;
}
/**
* For each text chunk, finds figure references and persists ChunkFigureRefEntity rows.
*/
public void linkChunksToFigures(List<Document> chunks, List<FigureEntity> bookFigures,
int pageNum) {
if (bookFigures.isEmpty()) return;
for (Document chunk : chunks) {
String chunkIdStr = chunk.getId();
UUID chunkId;
try {
chunkId = UUID.fromString(chunkIdStr);
} catch (IllegalArgumentException ex) {
log.warn("Chunk has non-UUID id: {}", chunkIdStr);
continue;
}
Matcher m = REF_PATTERN.matcher(chunk.getText());
while (m.find()) {
String refNum = m.group(2).trim();
// Find matching figure by label suffix
for (FigureEntity figure : bookFigures) {
if (figure.getLabel() != null && figure.getLabel().endsWith(refNum)) {
refRepository.save(new ChunkFigureRefEntity(chunkId, figure.getId(), pageNum));
break;
}
}
}
}
}
}
@@ -0,0 +1,82 @@
package com.aiteacher.document;
import jakarta.persistence.*;
import java.time.Instant;
import java.util.UUID;
@Entity
@Table(name = "figure")
public class FigureEntity {
@Id
@Column(name = "id", length = 200)
private String id;
@Column(name = "book_id", nullable = false)
private UUID bookId;
@Column(name = "section_id", length = 200)
private String sectionId;
@Column(name = "chapter_id", length = 200)
private String chapterId;
@Column(name = "label", length = 100)
private String label;
@Column(name = "caption", columnDefinition = "TEXT")
private String caption;
@Enumerated(EnumType.STRING)
@Column(name = "figure_type", nullable = false, length = 50)
private FigureType figureType;
@Column(name = "page", nullable = false)
private int page;
@Column(name = "image_path", nullable = false, length = 1000)
private String imagePath;
@Column(name = "caption_embedding_id")
private UUID captionEmbeddingId;
@Column(name = "created_at", nullable = false)
private Instant createdAt;
public FigureEntity() {}
public FigureEntity(String id, UUID bookId, String sectionId, String chapterId,
String label, String caption, FigureType figureType,
int page, String imagePath) {
this.id = id;
this.bookId = bookId;
this.sectionId = sectionId;
this.chapterId = chapterId;
this.label = label;
this.caption = caption;
this.figureType = figureType;
this.page = page;
this.imagePath = imagePath;
this.createdAt = Instant.now();
}
public String getId() { return id; }
public UUID getBookId() { return bookId; }
public String getSectionId() { return sectionId; }
public String getChapterId() { return chapterId; }
public String getLabel() { return label; }
public String getCaption() { return caption; }
public FigureType getFigureType() { return figureType; }
public int getPage() { return page; }
public String getImagePath() { return imagePath; }
public UUID getCaptionEmbeddingId() { return captionEmbeddingId; }
public Instant getCreatedAt() { return createdAt; }
public void setCaptionEmbeddingId(UUID captionEmbeddingId) {
this.captionEmbeddingId = captionEmbeddingId;
}
public void setCaption(String caption) {
this.caption = caption;
}
}
@@ -0,0 +1,151 @@
package com.aiteacher.document;
import com.aiteacher.figure.FigureStorageService;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import javax.imageio.ImageIO;
import java.awt.image.BufferedImage;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
/**
* Extracts figure images from {@link PageResult.FigureData} entries produced by
* {@link MarkerPageParser}.
*
* <p>Marker returns pre-cropped PNG bytes for each detected figure, so no PDFBox
* page rendering or bounding-box cropping is needed. This service:
* <ol>
* <li>Decodes the PNG bytes to check dimensions (skip images below min size)</li>
* <li>Classifies the figure type from caption and surrounding text keywords</li>
* <li>Persists the image via {@link FigureStorageService}</li>
* <li>Persists a {@link FigureEntity} to the database</li>
* </ol>
*/
@Service
public class FigureExtractionService {
private static final Logger log = LoggerFactory.getLogger(FigureExtractionService.class);
private static final Pattern LABEL_PATTERN =
Pattern.compile("(?i)Fig\\.?\\s*(\\d+[\\-.\\d]*)");
private final FigureStorageService storageService;
private final FigureRepository figureRepository;
private final int minImageSizePx;
public FigureExtractionService(
FigureStorageService storageService,
FigureRepository figureRepository,
@Value("${app.figure-storage.min-image-size-px:100}") int minImageSizePx) {
this.storageService = storageService;
this.figureRepository = figureRepository;
this.minImageSizePx = minImageSizePx;
}
/** Holds the extraction output: persisted figures and a Marker blockId → DB figureId map. */
public record ExtractionResult(List<FigureEntity> figures, Map<String, String> blockIdToFigureId) {}
/**
* Extracts and persists figures for all pages described by {@code pageResults}.
*
* @param bookId owning book
* @param chapterId chapter bucket for these sections
* @param pageResults Marker parse output — each entry's {@code figures} list
* carries pre-cropped PNG bytes for that page
* @return {@link ExtractionResult} with persisted figures and blockId→figureId map
* (used to resolve markdown image placeholders)
*/
public ExtractionResult extract(UUID bookId, String chapterId,
List<PageResult> pageResults) {
List<FigureEntity> figures = new ArrayList<>();
Map<String, String> blockIdToFigureId = new HashMap<>();
int figureCounter = 0;
for (PageResult page : pageResults) {
if (page.figures().isEmpty()) continue;
for (PageResult.FigureData figureData : page.figures()) {
try {
BufferedImage image = decodeImage(figureData.imageBytes());
if (image == null) {
log.debug("Could not decode image on page {} of book {} (block {})",
page.pageNumber(), bookId, figureData.blockId());
continue;
}
if (image.getWidth() < minImageSizePx || image.getHeight() < minImageSizePx) {
log.debug("Skipping small figure on page {} ({}×{})",
page.pageNumber(), image.getWidth(), image.getHeight());
continue;
}
figureCounter++;
String figureId = bookId + "-fig-" + page.pageNumber() + "-" + figureCounter;
String caption = figureData.nearestCaption();
String label = detectLabel(caption, figureCounter);
FigureType type = classifyType(caption, page.orderedText());
String sectionId = bookId + "-p" + page.pageNumber();
String imagePath = storageService.save(bookId, figureId, image);
FigureEntity figure = new FigureEntity(
figureId, bookId, sectionId, chapterId,
label, caption, type, page.pageNumber(), imagePath);
figures.add(figureRepository.save(figure));
blockIdToFigureId.put(figureData.blockId(), figureId);
} catch (Exception ex) {
log.warn("Failed to extract figure on page {} of book {}: {}",
page.pageNumber(), bookId, ex.getMessage());
}
}
}
log.info("Extracted {} figures for book {}", figures.size(), bookId);
return new ExtractionResult(figures, blockIdToFigureId);
}
// --- Private helpers ---
private BufferedImage decodeImage(byte[] imageBytes) {
if (imageBytes == null || imageBytes.length == 0) return null;
try {
return ImageIO.read(new ByteArrayInputStream(imageBytes));
} catch (IOException ex) {
return null;
}
}
private String detectLabel(String caption, int counter) {
if (caption != null) {
Matcher m = LABEL_PATTERN.matcher(caption);
if (m.find()) return "Fig. " + m.group(1).trim();
}
return "Fig. " + counter;
}
private FigureType classifyType(String caption, String pageText) {
String combined = ((caption != null ? caption : "") + " " +
(pageText != null ? pageText : "")).toLowerCase();
if (combined.contains("mri") || combined.contains("ct ") || combined.contains("magnetic")
|| combined.contains("tomography")) return FigureType.MRI_CT_SCAN;
if (combined.contains("intraoperative") || combined.contains("intra-op"))
return FigureType.INTRAOPERATIVE_IMAGE;
if (caption != null && caption.toLowerCase().startsWith("table"))
return FigureType.TABLE;
if (combined.contains("chart") || combined.contains("histogram") || combined.contains("graph"))
return FigureType.CHART;
if (combined.contains("photograph") || combined.contains("photo"))
return FigureType.SURGICAL_PHOTOGRAPH;
return FigureType.ANATOMICAL_DIAGRAM;
}
}
@@ -0,0 +1,11 @@
package com.aiteacher.document;
import org.springframework.data.jpa.repository.JpaRepository;
import java.util.List;
import java.util.UUID;
public interface FigureRepository extends JpaRepository<FigureEntity, String> {
List<FigureEntity> findAllByBookId(UUID bookId);
void deleteAllByBookId(UUID bookId);
}
@@ -0,0 +1,10 @@
package com.aiteacher.document;
public enum FigureType {
ANATOMICAL_DIAGRAM,
SURGICAL_PHOTOGRAPH,
MRI_CT_SCAN,
TABLE,
CHART,
INTRAOPERATIVE_IMAGE
}
@@ -0,0 +1,14 @@
package com.aiteacher.document;
import java.util.UUID;
public interface MarkdownStorageService {
/** Uploads the markdown content and returns the S3 key. */
String save(UUID bookId, int pageNumber, String markdown);
/** Downloads and returns the markdown content for the given book and page. */
String getText(UUID bookId, int pageNumber);
/** Deletes all markdown files for the given book. */
void deleteAll(UUID bookId);
}
@@ -0,0 +1,335 @@
package com.aiteacher.document;
import tools.jackson.databind.JsonNode;
import tools.jackson.databind.ObjectMapper;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.core.io.FileSystemResource;
import org.springframework.http.MediaType;
import org.springframework.stereotype.Service;
import org.springframework.util.LinkedMultiValueMap;
import org.springframework.util.MultiValueMap;
import org.springframework.web.client.RestClient;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.*;
/**
* Parses a PDF with a single call to the Marker server using {@code output_format=json}.
*
* <p>The JSON response contains an {@code output} field that is itself a JSON string with a
* tree structure: the root has a {@code children} array where each item is a {@code Page} block.
* Each block carries an {@code html} field with {@code <content-ref src='blockId'>} placeholders
* that reference its {@code children} by ID.
*
* <p>{@link #jsonToHtml} mirrors the Marker Python {@code json_to_html} utility: it walks the
* tree recursively and resolves every {@code content-ref} with the rendered HTML of the
* referenced child block.
*
* <p>Returns a {@link ParsedBook} with:
* <ul>
* <li>{@code pages} — one {@link PageResult} per non-empty page (drives embeddings)</li>
* <li>{@code htmlByPage} — full resolved HTML per page (saved to S3 for the reader)</li>
* </ul>
*/
@Service
public class MarkerPageParser {
private static final Logger log = LoggerFactory.getLogger(MarkerPageParser.class);
private static final Set<String> TEXT_BLOCK_TYPES = Set.of(
"Text", "TextInlineMath", "ListItem", "Table", "TableOfContents", "Code", "Equation",
"Footnote", "Caption", "PageHeader", "PageFooter", "Handwriting"
);
private static final Set<String> FIGURE_BLOCK_TYPES = Set.of("Figure", "Picture", "FigureGroup", "PictureGroup");
private static final int CHUNK_SIZE = 100;
private static final ObjectMapper MAPPER = new ObjectMapper();
private final RestClient restClient;
private final PdfSplitterService pdfSplitterService;
public MarkerPageParser(@Qualifier("markerRestClient") RestClient restClient,
PdfSplitterService pdfSplitterService) {
this.restClient = restClient;
this.pdfSplitterService = pdfSplitterService;
}
/**
* Parses a PDF by splitting it into {@value #CHUNK_SIZE}-page chunks, submitting each
* chunk to Marker individually, and merging the results into a single {@link ParsedBook}.
* Page numbers in the merged result are absolute (1-based across the whole document).
*/
public ParsedBook parse(Path pdfPath) throws IOException {
List<PdfSplitterService.PdfChunk> chunks = pdfSplitterService.split(pdfPath, CHUNK_SIZE);
log.info("Processing {} chunk(s) for {}", chunks.size(), pdfPath.getFileName());
List<PageResult> allPages = new ArrayList<>();
Map<Integer, String> allHtml = new LinkedHashMap<>();
try {
for (int c = 0; c < chunks.size(); c++) {
PdfSplitterService.PdfChunk chunk = chunks.get(c);
log.info("Submitting chunk {}/{} to Marker (page offset {})", c + 1, chunks.size(), chunk.pageOffset());
ParsedBook chunkResult = submitChunk(chunk.tempFile());
// Rebase page numbers from chunk-relative to document-absolute
for (PageResult page : chunkResult.pages()) {
int absolutePage = chunk.pageOffset() + page.pageNumber();
allPages.add(new PageResult(absolutePage, page.orderedText(), page.headingTitle(), page.figures()));
}
chunkResult.htmlByPage().forEach((chunkPage, html) ->
allHtml.put(chunk.pageOffset() + chunkPage, html));
}
} finally {
// Delete temporary chunk files (skip if the chunk is the original PDF)
for (PdfSplitterService.PdfChunk chunk : chunks) {
if (!chunk.tempFile().equals(pdfPath)) {
try { Files.deleteIfExists(chunk.tempFile()); }
catch (IOException e) { log.warn("Could not delete temp chunk {}", chunk.tempFile()); }
}
}
}
log.info("Marker produced {} non-empty pages from {} chunk(s) of {}",
allPages.size(), chunks.size(), pdfPath.getFileName());
return new ParsedBook(allPages, allHtml);
}
/** Submits a single PDF file to Marker and returns the parsed result with chunk-relative page numbers. */
private ParsedBook submitChunk(Path chunkPath) {
MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
body.add("file", new FileSystemResource(chunkPath));
body.add("output_format", "json");
JsonNode response = restClient.post()
.uri("/marker/upload")
.contentType(MediaType.MULTIPART_FORM_DATA)
.body(body)
.retrieve()
.body(JsonNode.class);
try {
Files.writeString(Path.of("/tmp/marker-response-json.json"), response.toPrettyString());
} catch (IOException e) {
log.warn("Could not save Marker response to /tmp/marker-response-json.json", e);
}
List<JsonNode> pageNodes = extractPages(response);
if (pageNodes.isEmpty()) {
log.warn("Marker returned no pages for chunk {}", chunkPath.getFileName());
return new ParsedBook(List.of(), Map.of());
}
List<PageResult> pages = new ArrayList<>();
Map<Integer, String> htmlByPage = new LinkedHashMap<>();
for (int i = 0; i < pageNodes.size(); i++) {
JsonNode pageNode = pageNodes.get(i);
int pageNumber = i + 1; // 1-based, chunk-relative
PageResult result = buildPageResult(pageNode, pageNumber);
String html = jsonToHtml(pageNode);
// Always save HTML so the reader can navigate to every page
htmlByPage.put(pageNumber, html);
// Only queue for embedding if the page has extractable content
if (!result.orderedText().isBlank() || !result.figures().isEmpty()) {
pages.add(result);
}
}
return new ParsedBook(pages, htmlByPage);
}
// ── Page extraction ───────────────────────────────────────────────────────
/**
* Parses the {@code output} JSON string and returns the list of page nodes
* (the top-level {@code children} of the document root).
*/
private List<JsonNode> extractPages(JsonNode response) {
if (response == null) return List.of();
JsonNode outputNode = response.path("output");
if (outputNode.isMissingNode()) {
log.warn("Marker response has no 'output' field");
return List.of();
}
try {
JsonNode root = MAPPER.readTree(outputNode.stringValue());
JsonNode children = root.path("children");
if (children.isMissingNode() || !children.isArray()) {
log.warn("Marker output root has no 'children' array");
return List.of();
}
List<JsonNode> result = new ArrayList<>();
children.forEach(result::add);
return result;
} catch (Exception e) {
log.warn("Could not parse Marker 'output' string as JSON: {}", e.getMessage());
return List.of();
}
}
// ── HTML rendering ────────────────────────────────────────────────────────
/**
* Java equivalent of the Marker Python {@code json_to_html} utility.
*
* <p>Algorithm:
* <ol>
* <li>If the block has no children, return its {@code html} as-is (leaf node).</li>
* <li>Otherwise recursively render each child, then replace every
* {@code <content-ref src='childId'>} placeholder in the block's own {@code html}
* with the rendered child HTML.</li>
* </ol>
*/
String jsonToHtml(JsonNode block) {
String html = str(block.path("html"));
// If the block carries image data, inject <img> data-URI tags.
// Marker stores base64 image bytes in block.images keyed by block ID.
// Picture/Figure leaf blocks have empty html, so this is the only way to
// get the image into the rendered output.
JsonNode images = block.path("images");
if (!images.isMissingNode() && !images.isNull() && !images.isEmpty()) {
StringBuilder imgTags = new StringBuilder();
images.properties().forEach(entry -> {
String base64 = str(entry.getValue());
if (!base64.isEmpty()) {
String mime = detectImageMime(base64);
imgTags.append("<img src=\"data:").append(mime)
.append(";base64,").append(base64).append("\">");
}
});
if (!imgTags.isEmpty()) {
html = html + imgTags;
}
}
JsonNode children = block.path("children");
if (children.isMissingNode() || children.isNull() || !children.isArray() || children.isEmpty()) {
return html; // leaf node
}
// Build id → rendered-html map for all direct children
Map<String, String> childHtml = new LinkedHashMap<>();
for (JsonNode child : children) {
String id = str(child.path("id"));
childHtml.put(id, jsonToHtml(child));
}
// Replace every <content-ref src='id'></content-ref> with the child's HTML
for (Map.Entry<String, String> entry : childHtml.entrySet()) {
String ref = "<content-ref src='" + entry.getKey() + "'></content-ref>";
html = html.replace(ref, entry.getValue());
}
return html;
}
// ── PageResult (text + figures for embeddings) ────────────────────────────
private PageResult buildPageResult(JsonNode pageBlock, int pageNumber) {
StringBuilder text = new StringBuilder();
String[] headingTitle = {null};
List<PageResult.FigureData> figures = new ArrayList<>();
walkBlock(pageBlock, text, headingTitle, figures);
return new PageResult(pageNumber, text.toString().strip(), headingTitle[0], figures);
}
/** Recursively walks the block tree, collecting text and figures in reading order. */
private void walkBlock(JsonNode block, StringBuilder text, String[] headingTitle,
List<PageResult.FigureData> figures) {
String type = str(block.path("block_type"));
if ("SectionHeader".equals(type)) {
String heading = stripHtml(str(block.path("html"))).strip();
if (!heading.isEmpty() && headingTitle[0] == null) headingTitle[0] = heading;
appendText(text, heading);
} else if (TEXT_BLOCK_TYPES.contains(type)) {
appendText(text, stripHtml(str(block.path("html"))));
} else if (FIGURE_BLOCK_TYPES.contains(type)) {
String caption = findCaption(block);
extractFigures(block, caption, figures);
}
// Recurse into children (content-ref ordering is implicit via tree order)
JsonNode children = block.path("children");
if (!children.isMissingNode() && !children.isNull() && children.isArray()) {
for (JsonNode child : children) {
walkBlock(child, text, headingTitle, figures);
}
}
}
/** Finds the first Caption child inside a figure block, if any. */
private String findCaption(JsonNode figureBlock) {
JsonNode children = figureBlock.path("children");
if (children.isMissingNode() || !children.isArray()) return null;
for (JsonNode child : children) {
if ("Caption".equals(str(child.path("block_type")))) {
String caption = stripHtml(str(child.path("html"))).strip();
return caption.isEmpty() ? null : caption;
}
}
return null;
}
private void extractFigures(JsonNode block, String caption, List<PageResult.FigureData> out) {
JsonNode images = block.path("images");
if (images.isMissingNode() || images.isEmpty()) return;
images.properties().forEach(entry -> {
String blockId = entry.getKey();
String base64 = str(entry.getValue());
if (base64.isEmpty()) return;
try {
byte[] bytes = Base64.getDecoder().decode(base64);
out.add(new PageResult.FigureData(bytes, caption, blockId));
} catch (IllegalArgumentException ex) {
log.warn("Could not decode base64 image for block {}: {}", blockId, ex.getMessage());
}
});
}
// ── Utilities ─────────────────────────────────────────────────────────────
private void appendText(StringBuilder sb, String text) {
if (text == null) return;
String stripped = text.strip();
if (stripped.isEmpty()) return;
if (sb.length() > 0) sb.append("\n\n");
sb.append(stripped);
}
private String stripHtml(String html) {
if (html == null || html.isEmpty()) return "";
return html.replaceAll("<[^>]*>", "").replaceAll("\\s{2,}", " ").strip();
}
/** Detects MIME type from the first characters of a base64-encoded image. */
private static String detectImageMime(String base64) {
if (base64.startsWith("/9j/")) return "image/jpeg";
if (base64.startsWith("iVBOR")) return "image/png";
if (base64.startsWith("R0lGO")) return "image/gif";
if (base64.startsWith("UklGR")) return "image/webp";
return "image/png"; // safe fallback
}
/** Null-safe string extraction from a JsonNode (Jackson 3: stringValue() returns null for non-strings). */
private static String str(JsonNode node) {
String v = node.stringValue();
return v != null ? v : "";
}
}
@@ -0,0 +1,25 @@
package com.aiteacher.document;
import java.util.List;
/**
* Internal DTO produced by MarkerPageParser for one PDF page.
* Decouples the Marker HTTP API from downstream services.
*/
public record PageResult(
int pageNumber, // 1-based, derived from Marker page block index
String orderedText, // full page text in correct reading order (blocks joined by \n\n)
String headingTitle, // first SectionHeader block on page, or null
List<FigureData> figures // extracted figure images (may be empty)
) {
/**
* A figure extracted from the page.
* Image bytes are PNG data decoded from the Marker JSON {@code images} map.
*/
public record FigureData(
byte[] imageBytes, // PNG image data (base64-decoded from Marker response)
String nearestCaption, // text of the adjacent Caption block, or null
String blockId // Marker block ID (e.g. "/page/0/Figure/2") for traceability
) {}
}
@@ -0,0 +1,16 @@
package com.aiteacher.document;
import java.util.List;
import java.util.Map;
/**
* Result of a full Marker parse: structured page data (from JSON) plus
* native per-page markdown (from the separate Markdown API call).
*
* @param pages one entry per non-empty page, derived from the chunks response
* @param htmlByPage concatenated block HTML keyed by 1-based page number
*/
public record ParsedBook(
List<PageResult> pages,
Map<Integer, String> htmlByPage
) {}
@@ -0,0 +1,72 @@
package com.aiteacher.document;
import org.apache.pdfbox.io.RandomAccessReadBufferedFile;
import org.apache.pdfbox.multipdf.Splitter;
import org.apache.pdfbox.pdfparser.PDFParser;
import org.apache.pdfbox.pdmodel.PDDocument;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Service;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
/**
* Splits a PDF file into fixed-size chunks using PDFBox.
* Each chunk is saved as a temporary file so it can be submitted independently to Marker.
*/
@Service
public class PdfSplitterService {
private static final Logger log = LoggerFactory.getLogger(PdfSplitterService.class);
/**
* A chunk of a split PDF.
*
* @param tempFile path to the temporary PDF file (caller must delete when done)
* @param pageOffset 0-based index of the first page in this chunk within the original document
*/
public record PdfChunk(Path tempFile, int pageOffset) {}
/**
* Splits {@code pdfPath} into chunks of at most {@code maxPagesPerChunk} pages.
* Returns a single-element list when the document fits in one chunk.
*
* @param pdfPath source PDF
* @param maxPagesPerChunk maximum pages per chunk
* @return ordered list of chunks; caller is responsible for deleting {@code tempFile}s
*/
public List<PdfChunk> split(Path pdfPath, int maxPagesPerChunk) throws IOException {
try (PDDocument doc = new PDFParser(new RandomAccessReadBufferedFile(pdfPath.toFile())).parse()) {
int totalPages = doc.getNumberOfPages();
log.info("PDF {} has {} pages, splitting into chunks of {}", pdfPath.getFileName(), totalPages, maxPagesPerChunk);
if (totalPages <= maxPagesPerChunk) {
// No split needed — return the original file as a single virtual chunk
return List.of(new PdfChunk(pdfPath, 0));
}
Splitter splitter = new Splitter();
splitter.setSplitAtPage(maxPagesPerChunk);
List<PDDocument> parts = splitter.split(doc);
List<PdfChunk> chunks = new ArrayList<>(parts.size());
int offset = 0;
for (PDDocument part : parts) {
try {
Path tmp = Files.createTempFile("marker-chunk-", ".pdf");
part.save(tmp.toFile());
chunks.add(new PdfChunk(tmp, offset));
log.debug("Created chunk at {} (page offset {})", tmp, offset);
offset += part.getNumberOfPages();
} finally {
part.close();
}
}
return chunks;
}
}
}
@@ -0,0 +1,114 @@
package com.aiteacher.document;
import org.apache.pdfbox.Loader;
import org.apache.pdfbox.pdmodel.PDDocument;
import org.apache.pdfbox.pdmodel.PDPage;
import org.apache.pdfbox.pdmodel.common.PDRectangle;
import org.apache.pdfbox.text.PDFTextStripperByArea;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.awt.Rectangle;
import java.io.IOException;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
import java.util.UUID;
/**
* Parses a PDF into page-level SectionEntity records stored in Postgres.
* Uses column-aware extraction via PDFTextStripperByArea: for two-column pages,
* left column is extracted first then right, preserving correct reading order.
* Text is also normalized (collapsed whitespace) before storage.
*/
@Service
public class PdfStructureParser {
private static final Logger log = LoggerFactory.getLogger(PdfStructureParser.class);
// Right column is considered empty (single-column page) if it has < 20% of left column's content
private static final double TWO_COLUMN_THRESHOLD = 0.2;
private final ChapterRepository chapterRepository;
private final SectionRepository sectionRepository;
public PdfStructureParser(ChapterRepository chapterRepository,
SectionRepository sectionRepository) {
this.chapterRepository = chapterRepository;
this.sectionRepository = sectionRepository;
}
@Transactional
public List<SectionEntity> parse(UUID bookId, String bookTitle, Path pdfPath) {
log.info("Parsing PDF structure for book {}", bookId);
String chapterId = bookId + "-ch1";
ChapterEntity chapter = new ChapterEntity(chapterId, bookId, 1, bookTitle, 1);
chapterRepository.save(chapter);
List<SectionEntity> sections = new ArrayList<>();
try (PDDocument doc = Loader.loadPDF(pdfPath.toFile())) {
List<PDPage> pages = new ArrayList<>();
doc.getPages().forEach(pages::add);
for (int i = 0; i < 25; i++) {
int pageNum = i + 1;
String text = normalizeWhitespace(extractPageText(pages.get(i)));
if (text.isBlank()) continue;
String sectionId = bookId + "-p" + pageNum;
SectionEntity section = new SectionEntity(
sectionId, chapterId, bookId,
String.valueOf(pageNum),
"Page " + pageNum,
pageNum, pageNum,
text
);
sections.add(sectionRepository.save(section));
}
} catch (IOException e) {
throw new RuntimeException("Failed to parse PDF for book " + bookId, e);
}
log.info("Parsed {} sections for book {}", sections.size(), bookId);
return sections;
}
/**
* Extracts text from a single page using column-aware region extraction.
* Splits the page at the horizontal midpoint. If the right region has fewer
* than 20% of the characters of the left region, treats the page as single-column.
*/
private String extractPageText(PDPage page) throws IOException {
PDRectangle mediaBox = page.getMediaBox();
int width = (int) mediaBox.getWidth();
int height = (int) mediaBox.getHeight();
int mid = width / 2;
PDFTextStripperByArea stripper = new PDFTextStripperByArea();
stripper.setSortByPosition(true);
stripper.addRegion("left", new Rectangle(0, 0, mid, height));
stripper.addRegion("right", new Rectangle(mid, 0, width - mid, height));
stripper.extractRegions(page);
String left = stripper.getTextForRegion("left").strip();
String right = stripper.getTextForRegion("right").strip();
if (right.length() < left.length() * TWO_COLUMN_THRESHOLD) {
// Single-column page — left holds all (or nearly all) content
return left.isEmpty() ? right : left;
}
return left + "\n\n" + right;
}
/** Collapses multi-space/tab runs and excessive blank lines. */
private String normalizeWhitespace(String text) {
return text
.replaceAll("[ \t]{2,}", " ")
.replaceAll("\n{3,}", "\n\n")
.trim();
}
}
@@ -0,0 +1,97 @@
package com.aiteacher.document;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.S3Configuration;
import software.amazon.awssdk.services.s3.model.*;
import java.net.URI;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.UUID;
@Service
public class S3MarkdownStorageService implements MarkdownStorageService {
private static final Logger log = LoggerFactory.getLogger(S3MarkdownStorageService.class);
private final S3Client s3;
private final String bucket;
public S3MarkdownStorageService(
@Value("${app.figure-storage.endpoint}") String endpoint,
@Value("${app.figure-storage.region}") String region,
@Value("${app.figure-storage.bucket}") String bucket,
@Value("${app.figure-storage.access-key-id}") String accessKeyId,
@Value("${app.figure-storage.secret-access-key}") String secretKey) {
this.bucket = bucket;
URI endpointUri = URI.create(endpoint);
StaticCredentialsProvider credentials = StaticCredentialsProvider.create(
AwsBasicCredentials.create(accessKeyId, secretKey));
Region awsRegion = Region.of(region);
S3Configuration s3Config = S3Configuration.builder().pathStyleAccessEnabled(true).build();
this.s3 = S3Client.builder()
.endpointOverride(endpointUri)
.region(awsRegion)
.credentialsProvider(credentials)
.serviceConfiguration(s3Config)
.build();
}
@Override
public String save(UUID bookId, int pageNumber, String markdown) {
String key = key(bookId, pageNumber);
byte[] bytes = markdown.getBytes(StandardCharsets.UTF_8);
s3.putObject(
PutObjectRequest.builder().bucket(bucket).key(key)
.contentType("text/html; charset=utf-8")
.contentLength((long) bytes.length).build(),
RequestBody.fromBytes(bytes));
return key;
}
@Override
public String getText(UUID bookId, int pageNumber) {
byte[] bytes = s3.getObjectAsBytes(
GetObjectRequest.builder().bucket(bucket).key(key(bookId, pageNumber)).build()
).asByteArray();
return new String(bytes, StandardCharsets.UTF_8);
}
@Override
public void deleteAll(UUID bookId) {
String prefix = "html/" + bookId + "/";
try {
List<ObjectIdentifier> toDelete = new ArrayList<>();
s3.listObjectsV2Paginator(ListObjectsV2Request.builder()
.bucket(bucket).prefix(prefix).build()).stream()
.flatMap(page -> page.contents().stream())
.map(S3Object::key)
.map(k -> ObjectIdentifier.builder().key(k).build())
.forEach(toDelete::add);
if (toDelete.isEmpty()) return;
s3.deleteObjects(DeleteObjectsRequest.builder()
.bucket(bucket)
.delete(Delete.builder().objects(toDelete).build())
.build());
log.info("Deleted {} markdown files from S3 for book {}", toDelete.size(), bookId);
} catch (S3Exception ex) {
log.warn("Could not fully delete markdown for book {} from S3: {}", bookId, ex.getMessage());
}
}
private static String key(UUID bookId, int pageNumber) {
return "html/" + bookId + "/page-" + pageNumber + ".html";
}
}
@@ -0,0 +1,63 @@
package com.aiteacher.document;
import jakarta.persistence.*;
import java.time.Instant;
import java.util.UUID;
@Entity
@Table(name = "section")
public class SectionEntity {
@Id
@Column(name = "id", length = 200)
private String id;
@Column(name = "chapter_id", nullable = false, length = 200)
private String chapterId;
@Column(name = "book_id", nullable = false)
private UUID bookId;
@Column(name = "number", length = 50)
private String number;
@Column(name = "title", length = 500)
private String title;
@Column(name = "page_start", nullable = false)
private int pageStart;
@Column(name = "page_end", nullable = false)
private int pageEnd;
@Column(name = "full_text", nullable = false, columnDefinition = "TEXT")
private String fullText;
@Column(name = "created_at", nullable = false)
private Instant createdAt;
public SectionEntity() {}
public SectionEntity(String id, String chapterId, UUID bookId, String number,
String title, int pageStart, int pageEnd, String fullText) {
this.id = id;
this.chapterId = chapterId;
this.bookId = bookId;
this.number = number;
this.title = title;
this.pageStart = pageStart;
this.pageEnd = pageEnd;
this.fullText = fullText;
this.createdAt = Instant.now();
}
public String getId() { return id; }
public String getChapterId() { return chapterId; }
public UUID getBookId() { return bookId; }
public String getNumber() { return number; }
public String getTitle() { return title; }
public int getPageStart() { return pageStart; }
public int getPageEnd() { return pageEnd; }
public String getFullText() { return fullText; }
public Instant getCreatedAt() { return createdAt; }
}
@@ -0,0 +1,19 @@
package com.aiteacher.document;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.Query;
import org.springframework.data.repository.query.Param;
import java.util.List;
import java.util.UUID;
public interface SectionRepository extends JpaRepository<SectionEntity, String> {
List<SectionEntity> findAllByBookId(UUID bookId);
void deleteAllByBookId(UUID bookId);
@Query("SELECT s FROM SectionEntity s WHERE s.bookId = :bookId AND s.pageStart <= :windowEnd AND s.pageEnd >= :windowStart ORDER BY s.pageStart")
List<SectionEntity> findByBookIdAndPageOverlap(
@Param("bookId") UUID bookId,
@Param("windowStart") int windowStart,
@Param("windowEnd") int windowEnd);
}
@@ -0,0 +1,103 @@
package com.aiteacher.document;
import org.springframework.ai.document.Document;
import org.springframework.stereotype.Service;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.UUID;
/**
* Splits a SectionEntity's full text into overlapping chunks for vector embedding.
* Target size: ~1800 characters (~450 tokens); overlap: 200 characters.
*/
@Service
public class TextChunkingService {
private static final int TARGET_CHARS = 1800;
private static final int OVERLAP_CHARS = 200;
public List<Document> chunk(SectionEntity section, String bookTitle) {
String text = section.getFullText();
if (text == null || text.isBlank()) return List.of();
List<String> windows = split(text);
List<Document> documents = new ArrayList<>();
for (int i = 0; i < windows.size(); i++) {
String chunkId = UUID.randomUUID().toString();
Map<String, Object> metadata = buildMetadata(section, bookTitle, i, windows.size(), chunkId);
documents.add(new Document(chunkId, windows.get(i), metadata));
}
return documents;
}
private List<String> split(String text) {
List<String> windows = new ArrayList<>();
int start = 0;
while (start < text.length()) {
int hardEnd = Math.min(start + TARGET_CHARS, text.length());
if (hardEnd == text.length()) {
String last = text.substring(start).strip();
if (!last.isEmpty()) windows.add(last);
break;
}
int splitAt = findSplitPoint(text, start, hardEnd);
String chunk = text.substring(start, splitAt).strip();
if (!chunk.isEmpty()) windows.add(chunk);
// Overlap: back up from split point, align to a word start
int overlapStart = Math.max(start + 1, splitAt - OVERLAP_CHARS);
while (overlapStart < splitAt && text.charAt(overlapStart) != ' ') overlapStart++;
start = overlapStart < splitAt ? overlapStart + 1 : splitAt;
}
return windows;
}
/**
* Finds the best split point at or before hardEnd, preferring (in order):
* paragraph boundary, sentence boundary, word boundary, hard cut.
*/
private int findSplitPoint(String text, int start, int hardEnd) {
int lookback = Math.min(400, (hardEnd - start) / 2);
// 1. Paragraph boundary
int paraIdx = text.lastIndexOf("\n\n", hardEnd);
if (paraIdx > hardEnd - lookback && paraIdx > start) return paraIdx + 2;
// 2. Sentence boundary (. ! ?) followed by space or newline
for (int i = hardEnd - 1; i > hardEnd - lookback && i > start; i--) {
char c = text.charAt(i);
if ((c == '.' || c == '!' || c == '?') && i + 1 < text.length()) {
char next = text.charAt(i + 1);
if (next == ' ' || next == '\n') return i + 1;
}
}
// 3. Word boundary
for (int i = hardEnd - 1; i > hardEnd - 100 && i > start; i--) {
if (text.charAt(i) == ' ') return i + 1;
}
// 4. Hard cut
return hardEnd;
}
private Map<String, Object> buildMetadata(SectionEntity section, String bookTitle,
int index, int total, String chunkId) {
Map<String, Object> m = new HashMap<>();
m.put("type", "TEXT");
m.put("book_id", section.getBookId().toString());
m.put("book_title", bookTitle);
m.put("chapter_id", section.getChapterId());
m.put("section_id", section.getId());
m.put("section_title", section.getTitle() != null ? section.getTitle() : "");
m.put("page_start", section.getPageStart());
m.put("page_end", section.getPageEnd());
m.put("chunk_index", index);
m.put("total_chunks", total);
m.put("chunk_id", chunkId);
return m;
}
}
@@ -0,0 +1,108 @@
package com.aiteacher.document;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.core.io.ByteArrayResource;
import org.springframework.stereotype.Service;
import org.springframework.util.MimeTypeUtils;
/**
* Analyses an extracted figure image using the OpenAI vision model.
*
* <p>Returns an {@link ImageAnalysis} record containing:
* <ul>
* <li>{@code description} — 2-3 sentence clinical description of the image</li>
* <li>{@code imageText} — all visible text, labels, and annotations copied verbatim
* from the image (empty string when none present)</li>
* </ul>
*
* <p>Both fields are stored: {@code description} drives the embedding; {@code imageText}
* is added to chunk metadata so queries can match exact labels (e.g., "Circle of Willis").
*/
@Service
public class VisionDescriptionService {
private static final Logger log = LoggerFactory.getLogger(VisionDescriptionService.class);
private static final String PROMPT = """
You are a neurosurgery educator analysing a medical image.
Respond in EXACTLY this format — no other text, no markdown:
DESCRIPTION: <2-3 sentence clinical description focusing on anatomical structures, surgical landmarks, and clinical significance>
IMAGE_TEXT: <all visible text, labels, measurements, and annotations copied verbatim, comma-separated; write NONE if no text visible>
""";
/** Minimum ms between vision API calls. Configurable via app.vision.min-interval-ms. */
private final long minIntervalMs;
private final ChatClient chatClient;
private volatile long lastCallAt = 0;
public VisionDescriptionService(
ChatClient chatClient,
@Value("${app.vision.min-interval-ms:2000}") long minIntervalMs) {
this.chatClient = chatClient;
this.minIntervalMs = minIntervalMs;
}
/**
* Holds the structured output of a vision model call on one figure image.
*
* @param description clinical description of the image content
* @param imageText verbatim text visible inside the image; empty string if none
*/
public record ImageAnalysis(String description, String imageText) {}
/**
* Analyses the image bytes and returns an {@link ImageAnalysis}.
* Falls back gracefully: if the vision call fails, the caption is used as description
* and imageText is left empty.
*
* @param imageBytes PNG bytes of the extracted figure
* @param captionFallback caption detected from surrounding text, may be null
*/
public ImageAnalysis analyze(byte[] imageBytes, String captionFallback) {
throttle();
try {
String raw = chatClient.prompt()
.user(u -> u
.text(PROMPT)
.media(MimeTypeUtils.IMAGE_PNG, new ByteArrayResource(imageBytes)))
.call()
.content();
return parse(raw, captionFallback);
} catch (Exception ex) {
log.warn("Vision analysis failed: {} — using caption as fallback", ex.getMessage());
return new ImageAnalysis(
captionFallback != null ? captionFallback : "Figure",
"");
}
}
private synchronized void throttle() {
long now = System.currentTimeMillis();
long wait = minIntervalMs - (now - lastCallAt);
if (wait > 0) {
try { Thread.sleep(wait); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
}
lastCallAt = System.currentTimeMillis();
}
private ImageAnalysis parse(String raw, String captionFallback) {
String description = captionFallback != null ? captionFallback : "Figure";
String imageText = "";
if (raw != null) {
for (String line : raw.split("\n")) {
if (line.startsWith("DESCRIPTION:")) {
String val = line.substring("DESCRIPTION:".length()).strip();
if (!val.isEmpty()) description = val;
} else if (line.startsWith("IMAGE_TEXT:")) {
String val = line.substring("IMAGE_TEXT:".length()).strip();
if (!val.isEmpty() && !"NONE".equalsIgnoreCase(val)) imageText = val;
}
}
}
return new ImageAnalysis(description, imageText);
}
}
@@ -0,0 +1,75 @@
package com.aiteacher.enrichment;
import com.aiteacher.document.SectionEntity;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.ai.document.Document;
import org.springframework.stereotype.Service;
import java.time.Instant;
import java.util.List;
import java.util.Map;
import java.util.UUID;
@Service
public class ChunkEnrichmentPipeline {
private static final Logger log = LoggerFactory.getLogger(ChunkEnrichmentPipeline.class);
private final ChunkEnrichmentService enrichmentService;
private final ChunkMetadataRepository metadataRepository;
public ChunkEnrichmentPipeline(ChunkEnrichmentService enrichmentService,
ChunkMetadataRepository metadataRepository) {
this.enrichmentService = enrichmentService;
this.metadataRepository = metadataRepository;
}
public void enrichAndPersist(List<Document> chunks,
Map<String, SectionEntity> sectionsById,
String bookTitle) {
int total = chunks.size();
int done = 0;
for (Document chunk : chunks) {
String sectionId = (String) chunk.getMetadata().get("section_id");
SectionEntity section = sectionId != null ? sectionsById.get(sectionId) : null;
UUID chunkId;
try {
chunkId = UUID.fromString(chunk.getId());
} catch (IllegalArgumentException ex) {
log.warn("Skipping chunk with non-UUID id '{}'", chunk.getId());
continue;
}
UUID bookId = extractBookId(chunk);
if (bookId == null || sectionId == null) {
log.warn("Skipping chunk {} missing book_id or section_id metadata", chunkId);
continue;
}
try {
ChunkEnrichmentResult result = enrichmentService.enrich(chunk.getText(), section, bookTitle);
ChunkMetadataEntity entity = new ChunkMetadataEntity(
chunkId, bookId, sectionId,
result.facet(), result.entities(), result.summary(),
ChunkEnrichmentService.MODEL_VERSION, Instant.now());
metadataRepository.save(entity);
} catch (Exception ex) {
log.warn("Enrichment failed for chunk {}: {}", chunkId, ex.getMessage());
}
done++;
if (done % 25 == 0) {
log.info("Enrichment progress: {}/{} chunks", done, total);
}
}
log.info("Enrichment complete: {}/{} chunks enriched", done, total);
}
private UUID extractBookId(Document chunk) {
Object raw = chunk.getMetadata().get("book_id");
if (raw == null) return null;
try {
return UUID.fromString(raw.toString());
} catch (IllegalArgumentException ex) {
return null;
}
}
}
@@ -0,0 +1,9 @@
package com.aiteacher.enrichment;
import java.util.List;
public record ChunkEnrichmentResult(
List<String> entities,
ConceptFacet facet,
String summary
) {}
@@ -0,0 +1,135 @@
package com.aiteacher.enrichment;
import com.aiteacher.document.SectionEntity;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.stereotype.Service;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
@Service
public class ChunkEnrichmentService {
public static final String MODEL_VERSION = "v1";
private static final int MAX_ENTITIES = 8;
private static final Logger log = LoggerFactory.getLogger(ChunkEnrichmentService.class);
private static final String SYSTEM_PROMPT = """
You are a medical indexing assistant that classifies neurosurgery textbook excerpts.
For each excerpt you receive, extract three fields:
- entities: the medical concepts, conditions, procedures, tools, or anatomical
structures the excerpt is ABOUT. Normalise each to lowercase, singular canonical
English form. Expand abbreviations (e.g. "SAH" -> "subarachnoid hemorrhage").
Avoid generic words ("patient", "technique"). Cap at %d entities.
- facet: exactly one of the following. Pick the SINGLE best fit based on the
excerpt's PRIMARY teaching purpose. Use OTHER only when nothing else applies.
DEFINITION — defines the entity / syndrome / concept ("what is X").
ANATOMY — neuroanatomy, vascular/tract relationships, operative
landmarks, anatomical variants.
PATHOPHYSIOLOGY — mechanism of disease, etiology, natural history,
molecular/cellular basis.
EPIDEMIOLOGY — incidence, prevalence, demographics, risk factors.
CLINICAL_PRESENTATION — symptoms, signs, neurological exam findings, syndromes
as they present in patients.
IMAGING — CT / MRI / angiography / DSA / ultrasound features and
interpretation. If the excerpt describes HOW something
looks on imaging, use IMAGING.
CLASSIFICATION — named grading scales, staging systems, subtype
taxonomies (Hunt-Hess, WFNS, Fisher, Spetzler-Martin,
GCS, Karnofsky, mRS, Simpson, etc.). If the excerpt
defines or applies a named scale, use CLASSIFICATION
even if it is grounded in imaging or clinical exam.
INDICATIONS — when to operate / treat / observe; patient selection
criteria; contraindications.
SURGICAL_TECHNIQUE — operative approach, positioning, steps, landmarks,
instruments, implants, intraoperative monitoring.
NONSURGICAL_MANAGEMENT — medical therapy, endovascular treatment, stereotactic
radiosurgery, conservative / observational management.
COMPLICATIONS — intra- or postoperative complications, adverse events.
OUTCOMES_FOLLOWUP — prognosis, morbidity/mortality rates, recurrence,
surveillance schedules, follow-up care.
OTHER — history, philosophy, ethics, or anything not covered.
Disambiguation rules:
* A named grading scale => CLASSIFICATION (even when grounded in imaging/exam).
* Tools and implants described as part of an operation => SURGICAL_TECHNIQUE,
not a standalone facet.
* Illustrative case reports => CLINICAL_PRESENTATION.
* Imaging findings of complications => COMPLICATIONS, not IMAGING.
- summary: one or two sentences describing what the excerpt teaches.
Respond with the structured JSON requested. Do not fabricate content not present in
the excerpt.
""".formatted(MAX_ENTITIES);
private final ChatClient chatClient;
public ChunkEnrichmentService(ChatClient chatClient) {
this.chatClient = chatClient;
}
public ChunkEnrichmentResult enrich(String chunkText, SectionEntity section, String bookTitle) {
String userPrompt = buildUserPrompt(chunkText, section, bookTitle);
LlmOutput raw = chatClient.prompt()
.system(SYSTEM_PROMPT)
.user(userPrompt)
.call()
.entity(LlmOutput.class);
if (raw == null) {
log.warn("LLM returned null enrichment; defaulting to OTHER");
return new ChunkEnrichmentResult(List.of(), ConceptFacet.OTHER, "");
}
List<String> entities = normaliseEntities(raw.entities());
ConceptFacet facet = parseFacet(raw.facet());
String summary = raw.summary() != null ? raw.summary().strip() : "";
return new ChunkEnrichmentResult(entities, facet, summary);
}
private String buildUserPrompt(String chunkText, SectionEntity section, String bookTitle) {
String sectionTitle = section != null && section.getTitle() != null ? section.getTitle() : "";
return """
BOOK: %s
SECTION: %s
EXCERPT:
---
%s
---
""".formatted(bookTitle, sectionTitle, chunkText);
}
private List<String> normaliseEntities(List<String> raw) {
if (raw == null) return List.of();
List<String> out = new ArrayList<>();
for (String e : raw) {
if (e == null) continue;
String canonical = e.trim().toLowerCase(Locale.ROOT);
if (canonical.isEmpty()) continue;
if (!out.contains(canonical)) out.add(canonical);
if (out.size() >= MAX_ENTITIES) break;
}
return out;
}
private ConceptFacet parseFacet(String raw) {
if (raw == null) return ConceptFacet.OTHER;
try {
return ConceptFacet.valueOf(raw.trim().toUpperCase(Locale.ROOT));
} catch (IllegalArgumentException ex) {
log.warn("LLM returned unknown facet '{}', defaulting to OTHER", raw);
return ConceptFacet.OTHER;
}
}
// DTO for Spring AI structured output; facet is read as String so we can defend against bad values
public record LlmOutput(List<String> entities, String facet, String summary) {}
}
@@ -0,0 +1,71 @@
package com.aiteacher.enrichment;
import jakarta.persistence.*;
import org.hibernate.annotations.JdbcTypeCode;
import org.hibernate.type.SqlTypes;
import java.time.Instant;
import java.util.List;
import java.util.UUID;
@Entity
@Table(name = "chunk_metadata")
@org.hibernate.annotations.Check(
name = "chunk_metadata_facet_check",
constraints = "facet IN ('DEFINITION','ANATOMY','PATHOPHYSIOLOGY','EPIDEMIOLOGY'," +
"'CLINICAL_PRESENTATION','IMAGING','CLASSIFICATION','INDICATIONS'," +
"'SURGICAL_TECHNIQUE','NONSURGICAL_MANAGEMENT','COMPLICATIONS'," +
"'OUTCOMES_FOLLOWUP','OTHER')")
public class ChunkMetadataEntity {
@Id
@Column(name = "chunk_id", nullable = false)
private UUID chunkId;
@Column(name = "book_id", nullable = false)
private UUID bookId;
@Column(name = "section_id", nullable = false, length = 200)
private String sectionId;
@Enumerated(EnumType.STRING)
@Column(name = "facet", nullable = false, length = 32)
private ConceptFacet facet;
@JdbcTypeCode(SqlTypes.JSON)
@Column(name = "entities", nullable = false, columnDefinition = "jsonb")
private List<String> entities;
@Column(name = "summary", nullable = false, columnDefinition = "TEXT")
private String summary;
@Column(name = "model_version", nullable = false, length = 32)
private String modelVersion;
@Column(name = "enriched_at", nullable = false)
private Instant enrichedAt;
protected ChunkMetadataEntity() {}
public ChunkMetadataEntity(UUID chunkId, UUID bookId, String sectionId,
ConceptFacet facet, List<String> entities, String summary,
String modelVersion, Instant enrichedAt) {
this.chunkId = chunkId;
this.bookId = bookId;
this.sectionId = sectionId;
this.facet = facet;
this.entities = entities;
this.summary = summary;
this.modelVersion = modelVersion;
this.enrichedAt = enrichedAt;
}
public UUID getChunkId() { return chunkId; }
public UUID getBookId() { return bookId; }
public String getSectionId() { return sectionId; }
public ConceptFacet getFacet() { return facet; }
public List<String> getEntities() { return entities; }
public String getSummary() { return summary; }
public String getModelVersion() { return modelVersion; }
public Instant getEnrichedAt() { return enrichedAt; }
}
@@ -0,0 +1,36 @@
package com.aiteacher.enrichment;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.Query;
import org.springframework.data.repository.query.Param;
import org.springframework.stereotype.Repository;
import org.springframework.transaction.annotation.Transactional;
import java.util.Collection;
import java.util.List;
import java.util.UUID;
@Repository
public interface ChunkMetadataRepository extends JpaRepository<ChunkMetadataEntity, UUID> {
long countByBookId(UUID bookId);
@Query(value = """
SELECT * FROM chunk_metadata
WHERE book_id = :bookId
AND entities @> to_jsonb(CAST(:entity AS text))
""", nativeQuery = true)
List<ChunkMetadataEntity> findByBookIdAndEntityContains(@Param("bookId") UUID bookId,
@Param("entity") String entity);
@Query(value = """
SELECT * FROM chunk_metadata
WHERE entities @> to_jsonb(CAST(:entity AS text))
""", nativeQuery = true)
List<ChunkMetadataEntity> findByEntityContains(@Param("entity") String entity);
List<ChunkMetadataEntity> findByChunkIdIn(Collection<UUID> chunkIds);
@Transactional
void deleteByBookId(UUID bookId);
}
@@ -0,0 +1,27 @@
package com.aiteacher.enrichment;
public enum ConceptFacet {
DEFINITION("Definition & Overview"),
ANATOMY("Anatomy"),
PATHOPHYSIOLOGY("Pathophysiology"),
EPIDEMIOLOGY("Epidemiology"),
CLINICAL_PRESENTATION("Clinical Presentation"),
IMAGING("Imaging"),
CLASSIFICATION("Classification & Grading"),
INDICATIONS("Indications & Patient Selection"),
SURGICAL_TECHNIQUE("Surgical Technique"),
NONSURGICAL_MANAGEMENT("Non-surgical Management"),
COMPLICATIONS("Complications"),
OUTCOMES_FOLLOWUP("Outcomes & Follow-up"),
OTHER("Other");
private final String displayTitle;
ConceptFacet(String displayTitle) {
this.displayTitle = displayTitle;
}
public String displayTitle() {
return displayTitle;
}
}
@@ -0,0 +1,138 @@
package com.aiteacher.enrichment;
import com.aiteacher.document.SectionEntity;
import com.aiteacher.document.SectionRepository;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.ai.document.Document;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Service;
import java.time.Instant;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.UUID;
import java.util.concurrent.ConcurrentHashMap;
@Service
public class EnrichmentBackfillService {
private static final Logger log = LoggerFactory.getLogger(EnrichmentBackfillService.class);
private final JdbcTemplate jdbcTemplate;
private final ChunkEnrichmentService enrichmentService;
private final ChunkMetadataRepository metadataRepository;
private final SectionRepository sectionRepository;
private final ObjectMapper objectMapper;
private final Map<UUID, BackfillProgress> progressByBook = new ConcurrentHashMap<>();
public EnrichmentBackfillService(JdbcTemplate jdbcTemplate,
ChunkEnrichmentService enrichmentService,
ChunkMetadataRepository metadataRepository,
SectionRepository sectionRepository,
ObjectMapper objectMapper) {
this.jdbcTemplate = jdbcTemplate;
this.enrichmentService = enrichmentService;
this.metadataRepository = metadataRepository;
this.sectionRepository = sectionRepository;
this.objectMapper = objectMapper;
}
public BackfillProgress getProgress(UUID bookId) {
return progressByBook.getOrDefault(bookId, BackfillProgress.idle());
}
@Async
public void backfillBook(UUID bookId, String bookTitle) {
List<Document> pending = listUnenrichedChunks(bookId);
int total = pending.size();
progressByBook.put(bookId, new BackfillProgress("RUNNING", total, 0, null));
log.info("Backfill starting for book {} — {} chunks pending", bookId, total);
int done = 0;
Map<String, SectionEntity> sectionCache = new HashMap<>();
for (Document chunk : pending) {
try {
String sectionId = (String) chunk.getMetadata().get("section_id");
SectionEntity section = sectionId != null
? sectionCache.computeIfAbsent(sectionId,
id -> sectionRepository.findById(id).orElse(null))
: null;
ChunkEnrichmentResult result = enrichmentService.enrich(chunk.getText(), section, bookTitle);
UUID chunkId = UUID.fromString(chunk.getId());
metadataRepository.save(new ChunkMetadataEntity(
chunkId, bookId, sectionId != null ? sectionId : "",
result.facet(), result.entities(), result.summary(),
ChunkEnrichmentService.MODEL_VERSION, Instant.now()));
} catch (Exception ex) {
log.warn("Backfill failed for chunk {} of book {}: {}", chunk.getId(), bookId, ex.getMessage());
}
done++;
progressByBook.put(bookId, new BackfillProgress("RUNNING", total, done, null));
}
progressByBook.put(bookId, new BackfillProgress("COMPLETED", total, done, null));
log.info("Backfill finished for book {} — {}/{} enriched", bookId, done, total);
}
private List<Document> listUnenrichedChunks(UUID bookId) {
// Left anti-join against chunk_metadata so re-runs are cheap.
String sql = """
SELECT vs.id, vs.content, vs.metadata::text AS metadata_text
FROM vector_store vs
LEFT JOIN chunk_metadata cm ON cm.chunk_id = vs.id
WHERE vs.metadata->>'book_id' = ?
AND vs.metadata->>'type' = 'TEXT'
AND cm.chunk_id IS NULL
""";
return jdbcTemplate.query(sql, (rs, rowNum) -> {
String id = rs.getString("id");
String content = rs.getString("content");
String metaJson = rs.getString("metadata_text");
Map<String, Object> meta = parseMetadata(metaJson);
return new Document(id, content != null ? content : "", meta);
}, bookId.toString());
}
private Map<String, Object> parseMetadata(String json) {
if (json == null || json.isBlank()) return Map.of();
try {
JsonNode node = objectMapper.readTree(json);
Map<String, Object> out = new HashMap<>();
node.properties().forEach(e -> {
JsonNode v = e.getValue();
if (v.isTextual()) out.put(e.getKey(), v.asText());
else if (v.isInt()) out.put(e.getKey(), v.asInt());
else if (v.isLong()) out.put(e.getKey(), v.asLong());
else if (v.isBoolean()) out.put(e.getKey(), v.asBoolean());
else out.put(e.getKey(), v.toString());
});
return out;
} catch (JsonProcessingException ex) {
log.warn("Failed to parse vector_store metadata JSON: {}", ex.getMessage());
return Map.of();
}
}
public Optional<Integer> countEnrichedChunks(UUID bookId) {
return Optional.of((int) metadataRepository.countByBookId(bookId));
}
public int countTotalTextChunks(UUID bookId) {
Integer n = jdbcTemplate.queryForObject(
"SELECT COUNT(*) FROM vector_store WHERE metadata->>'book_id' = ? AND metadata->>'type' = 'TEXT'",
Integer.class, bookId.toString());
return n != null ? n : 0;
}
public record BackfillProgress(String status, int chunksTotal, int chunksEnriched, String errorMessage) {
public static BackfillProgress idle() {
return new BackfillProgress("IDLE", 0, 0, null);
}
}
}
@@ -0,0 +1,50 @@
package com.aiteacher.enrichment;
import com.aiteacher.book.Book;
import com.aiteacher.book.BookRepository;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;
import java.util.NoSuchElementException;
import java.util.UUID;
@RestController
@RequestMapping("/api/v1/admin/books/{id}/enrich")
public class EnrichmentController {
private final BookRepository bookRepository;
private final EnrichmentBackfillService backfillService;
public EnrichmentController(BookRepository bookRepository,
EnrichmentBackfillService backfillService) {
this.bookRepository = bookRepository;
this.backfillService = backfillService;
}
@PostMapping
public ResponseEntity<EnrichmentBackfillService.BackfillProgress> start(@PathVariable UUID id) {
Book book = bookRepository.findById(id)
.orElseThrow(() -> new NoSuchElementException("Book not found."));
backfillService.backfillBook(id, book.getTitle());
int total = backfillService.countTotalTextChunks(id);
int enriched = backfillService.countEnrichedChunks(id).orElse(0);
return ResponseEntity.status(HttpStatus.ACCEPTED)
.body(new EnrichmentBackfillService.BackfillProgress("RUNNING", total, enriched, null));
}
@GetMapping
public ResponseEntity<EnrichmentBackfillService.BackfillProgress> status(@PathVariable UUID id) {
bookRepository.findById(id)
.orElseThrow(() -> new NoSuchElementException("Book not found."));
EnrichmentBackfillService.BackfillProgress progress = backfillService.getProgress(id);
if ("IDLE".equals(progress.status())) {
int total = backfillService.countTotalTextChunks(id);
int enriched = backfillService.countEnrichedChunks(id).orElse(0);
progress = new EnrichmentBackfillService.BackfillProgress(
enriched >= total && total > 0 ? "COMPLETED" : "IDLE",
total, enriched, null);
}
return ResponseEntity.ok(progress);
}
}
@@ -0,0 +1,27 @@
package com.aiteacher.figure;
import java.awt.image.BufferedImage;
import java.util.UUID;
public interface FigureStorageService {
/**
* Saves an extracted image to S3 and returns the object key stored in the database.
*/
String save(UUID bookId, String figureId, BufferedImage image);
/**
* Downloads the image bytes for the given S3 object key.
*/
byte[] getBytes(String key);
/**
* Returns a presigned GET URL valid for 1 hour for the given S3 object key.
*/
String presignedUrl(String key);
/**
* Deletes all figure objects for the given book.
*/
void deleteAll(UUID bookId);
}
@@ -0,0 +1,132 @@
package com.aiteacher.figure;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.S3Configuration;
import software.amazon.awssdk.services.s3.model.*;
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
import software.amazon.awssdk.services.s3.presigner.model.GetObjectPresignRequest;
import software.amazon.awssdk.services.s3.model.S3Object;
import javax.imageio.ImageIO;
import java.awt.image.BufferedImage;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.net.URI;
import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import java.util.UUID;
@Service
public class S3FigureStorageService implements FigureStorageService {
private static final Logger log = LoggerFactory.getLogger(S3FigureStorageService.class);
private final S3Client s3;
private final S3Presigner presigner;
private final String bucket;
public S3FigureStorageService(
@Value("${app.figure-storage.endpoint}") String endpoint,
@Value("${app.figure-storage.region}") String region,
@Value("${app.figure-storage.bucket}") String bucket,
@Value("${app.figure-storage.access-key-id}") String accessKeyId,
@Value("${app.figure-storage.secret-access-key}") String secretKey) {
this.bucket = bucket;
URI endpointUri = URI.create(endpoint);
StaticCredentialsProvider credentials = StaticCredentialsProvider.create(
AwsBasicCredentials.create(accessKeyId, secretKey));
Region awsRegion = Region.of(region);
S3Configuration s3Config = S3Configuration.builder()
.pathStyleAccessEnabled(true)
.build();
this.s3 = S3Client.builder()
.endpointOverride(endpointUri)
.region(awsRegion)
.credentialsProvider(credentials)
.serviceConfiguration(s3Config)
.build();
this.presigner = S3Presigner.builder()
.endpointOverride(endpointUri)
.region(awsRegion)
.credentialsProvider(credentials)
.serviceConfiguration(s3Config)
.build();
}
@Override
public String save(UUID bookId, String figureId, BufferedImage image) {
String key = "figures/" + bookId + "/" + figureId + ".png";
try {
ByteArrayOutputStream out = new ByteArrayOutputStream();
ImageIO.write(image, "PNG", out);
byte[] bytes = out.toByteArray();
s3.putObject(
PutObjectRequest.builder().bucket(bucket).key(key)
.contentType("image/png").contentLength((long) bytes.length).build(),
RequestBody.fromBytes(bytes));
return key;
} catch (IOException ex) {
throw new RuntimeException("Failed to encode figure " + figureId, ex);
} catch (S3Exception ex) {
throw new RuntimeException("Failed to upload figure " + figureId + " to S3", ex);
}
}
@Override
public byte[] getBytes(String key) {
try {
return s3.getObjectAsBytes(
GetObjectRequest.builder().bucket(bucket).key(key).build()).asByteArray();
} catch (S3Exception ex) {
throw new RuntimeException("Failed to download figure from S3: " + key, ex);
}
}
@Override
public String presignedUrl(String key) {
GetObjectPresignRequest request = GetObjectPresignRequest.builder()
.signatureDuration(Duration.ofHours(1))
.getObjectRequest(r -> r.bucket(bucket).key(key))
.build();
return presigner.presignGetObject(request).url().toString();
}
@Override
public void deleteAll(UUID bookId) {
String prefix = "figures/" + bookId + "/";
try {
List<ObjectIdentifier> toDelete = new ArrayList<>();
ListObjectsV2Request listRequest = ListObjectsV2Request.builder()
.bucket(bucket).prefix(prefix).build();
s3.listObjectsV2Paginator(listRequest).stream()
.flatMap(page -> page.contents().stream())
.map(S3Object::key)
.map(k -> ObjectIdentifier.builder().key(k).build())
.forEach(toDelete::add);
if (toDelete.isEmpty()) return;
s3.deleteObjects(DeleteObjectsRequest.builder()
.bucket(bucket)
.delete(Delete.builder().objects(toDelete).build())
.build());
log.info("Deleted {} figures from S3 for book {}", toDelete.size(), bookId);
} catch (S3Exception ex) {
log.warn("Could not fully delete figures for book {} from S3: {}", bookId, ex.getMessage());
}
}
}
@@ -0,0 +1,59 @@
package com.aiteacher.retrieval;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Service;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
/**
* Post-processes generated answers to strip citation labels that do not
* correspond to any passage retrieved for the current query, preventing
* hallucinated source references from reaching the user.
*/
@Service
public class CitationValidatorService {
private static final Logger log = LoggerFactory.getLogger(CitationValidatorService.class);
/** Matches citation labels of the form [S1], [F2], [S12], etc. */
private static final Pattern CITATION_PATTERN = Pattern.compile("\\[(S|F)\\d+\\]");
/**
* Removes any {@code [Sx]} / {@code [Fx]} citation in {@code generatedAnswer}
* whose label is not contained in {@code validLabels}.
*
* @param generatedAnswer raw model output
* @param validLabels set of labels present in the retrieved context
* @return cleaned answer text with hallucinated citations removed
*/
public String validate(String generatedAnswer, Set<String> validLabels) {
if (generatedAnswer == null) return "";
Matcher matcher = CITATION_PATTERN.matcher(generatedAnswer);
List<String> removed = new ArrayList<>();
StringBuffer sb = new StringBuffer();
while (matcher.find()) {
String label = matcher.group();
String inner = label.substring(1, label.length() - 1); // strip [ ]
if (validLabels.contains(inner)) {
matcher.appendReplacement(sb, Matcher.quoteReplacement(label));
} else {
removed.add(inner);
matcher.appendReplacement(sb, "");
}
}
matcher.appendTail(sb);
if (!removed.isEmpty()) {
log.warn("Stripped hallucinated citations: {}", removed);
}
return sb.toString();
}
}
@@ -0,0 +1,7 @@
package com.aiteacher.retrieval;
/**
* Value object holding the original user query alongside its clinically
* rewritten variant used for vector-store retrieval.
*/
public record ExpandedQuery(String original, String rewritten) {}
@@ -0,0 +1,27 @@
package com.aiteacher.retrieval;
import com.aiteacher.document.FigureEntity;
import com.aiteacher.document.SectionEntity;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
/**
* Value object produced when building the LLM context prompt.
* Maps short ref-labels (S1, S2… / F1, F2…) to their source entities
* and carries the fully formatted prompt text.
*/
public record LabelledContext(
Map<String, SectionEntity> sectionLabels,
Map<String, FigureEntity> figureLabels,
String promptText) {
/** Returns the union of all valid citation labels for this context. */
public Set<String> allLabels() {
Set<String> labels = new HashSet<>();
labels.addAll(sectionLabels.keySet());
labels.addAll(figureLabels.keySet());
return labels;
}
}
@@ -0,0 +1,111 @@
package com.aiteacher.retrieval;
import com.aiteacher.document.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.ai.document.Document;
import org.springframework.ai.vectorstore.SearchRequest;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.ai.vectorstore.filter.FilterExpressionBuilder;
import org.springframework.stereotype.Service;
import java.util.*;
/**
* Dual-modality retriever: searches text chunks and figure captions independently,
* then expands text hits to their parent sections and merges linked figures.
*/
@Service
public class NeurosurgeryRetriever {
private static final Logger log = LoggerFactory.getLogger(NeurosurgeryRetriever.class);
private static final int TEXT_TOP_K = 5;
private static final int FIGURE_TOP_K = 3;
private final VectorStore vectorStore;
private final SectionRepository sectionRepository;
private final FigureRepository figureRepository;
private final ChunkFigureRefRepository chunkFigureRefRepository;
public NeurosurgeryRetriever(VectorStore vectorStore,
SectionRepository sectionRepository,
FigureRepository figureRepository,
ChunkFigureRefRepository chunkFigureRefRepository) {
this.vectorStore = vectorStore;
this.sectionRepository = sectionRepository;
this.figureRepository = figureRepository;
this.chunkFigureRefRepository = chunkFigureRefRepository;
}
public RetrievalResult retrieve(String query, UUID bookId) {
FilterExpressionBuilder b = new FilterExpressionBuilder();
// 1. Text chunk search
List<Document> textHits = vectorStore.similaritySearch(
SearchRequest.builder()
.query(query)
.topK(TEXT_TOP_K)
.filterExpression(b.and(
b.eq("type", "TEXT"),
b.eq("book_id", bookId.toString())
).build())
.build()
);
// 2. Figure caption search (independent topK)
List<Document> figureHits = vectorStore.similaritySearch(
SearchRequest.builder()
.query(query)
.topK(FIGURE_TOP_K)
.filterExpression(b.and(
b.eq("type", "FIGURE"),
b.eq("book_id", bookId.toString())
).build())
.build()
);
// 3. Expand text chunks to parent sections from Postgres
List<String> sectionIds = textHits.stream()
.map(d -> (String) d.getMetadata().get("section_id"))
.filter(Objects::nonNull)
.distinct()
.toList();
List<SectionEntity> sections = sectionIds.isEmpty()
? List.of()
: sectionRepository.findAllById(sectionIds);
// 4. Fetch figures explicitly linked to retrieved chunks
List<UUID> chunkIds = textHits.stream()
.map(d -> {
try { return UUID.fromString(d.getId()); }
catch (Exception e) { return null; }
})
.filter(Objects::nonNull)
.toList();
List<String> linkedFigureIds = chunkIds.isEmpty()
? List.of()
: chunkFigureRefRepository.findByChunkIdIn(chunkIds)
.stream().map(ChunkFigureRefEntity::getFigureId).distinct().toList();
List<FigureEntity> linkedFigures = linkedFigureIds.isEmpty()
? List.of()
: figureRepository.findAllById(linkedFigureIds);
// 5. Collect figures from semantic figure search
List<String> semanticFigureIds = figureHits.stream()
.map(d -> (String) d.getMetadata().get("figure_id"))
.filter(Objects::nonNull)
.toList();
List<FigureEntity> semanticFigures = semanticFigureIds.isEmpty()
? List.of()
: figureRepository.findAllById(semanticFigureIds);
// 6. Merge and deduplicate figures by figureId (linked figures take precedence)
Map<String, FigureEntity> merged = new LinkedHashMap<>();
linkedFigures.forEach(f -> merged.put(f.getId(), f));
semanticFigures.forEach(f -> merged.putIfAbsent(f.getId(), f));
log.debug("Retrieved {} sections, {} figures for query", sections.size(), merged.size());
return new RetrievalResult(sections, new ArrayList<>(merged.values()));
}
}
@@ -0,0 +1,47 @@
package com.aiteacher.retrieval;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.stereotype.Service;
/**
* Rewrites a user query into precise clinical/surgical terminology so that
* vector-store retrieval can match textbook language even when the user's
* phrasing differs from the documentation vocabulary.
*/
@Service
public class QueryExpansionService {
private static final Logger log = LoggerFactory.getLogger(QueryExpansionService.class);
private static final String EXPANSION_PROMPT = """
Rewrite the following question using precise medical and surgical terminology \
as it would appear in a neurosurgery textbook index. \
Output only the rewritten question, nothing else.
Question: %s""";
private final ChatClient chatClient;
public QueryExpansionService(ChatClient chatClient) {
this.chatClient = chatClient;
}
/**
* Returns an {@link ExpandedQuery} whose {@code rewritten} field contains
* the clinically rephrased version of {@code query}.
*/
public ExpandedQuery expand(String query) {
String rewritten = chatClient.prompt()
.user(EXPANSION_PROMPT.formatted(query))
.call()
.content();
if (rewritten == null || rewritten.isBlank()) {
rewritten = query;
}
log.debug("Query expanded: '{}' → '{}'", query, rewritten);
return new ExpandedQuery(query, rewritten);
}
}
@@ -0,0 +1,11 @@
package com.aiteacher.retrieval;
import com.aiteacher.document.FigureEntity;
import com.aiteacher.document.SectionEntity;
import java.util.List;
public record RetrievalResult(
List<SectionEntity> parentSections,
List<FigureEntity> figures
) {}
@@ -0,0 +1,7 @@
package com.aiteacher.topic;
import java.time.Instant;
import java.util.UUID;
public record SavedSummaryItem(UUID id, int summaryNumber, Instant generatedAt) {
}
@@ -5,6 +5,7 @@ import org.springframework.web.bind.annotation.*;
import java.util.List; import java.util.List;
import java.util.NoSuchElementException; import java.util.NoSuchElementException;
import java.util.UUID;
@RestController @RestController
@RequestMapping("/api/v1/topics") @RequestMapping("/api/v1/topics")
@@ -25,11 +26,30 @@ public class TopicController {
} }
@PostMapping("/{id}/summary") @PostMapping("/{id}/summary")
public ResponseEntity<TopicSummaryResponse> generateSummary(@PathVariable String id) { public ResponseEntity<TopicSummaryResponse> generateSummary(
@PathVariable String id,
@RequestParam(defaultValue = "en") String language) {
Topic topic = topicRepository.findById(id) Topic topic = topicRepository.findById(id)
.orElseThrow(() -> new NoSuchElementException("Topic not found.")); .orElseThrow(() -> new NoSuchElementException("Topic not found."));
TopicSummaryResponse response = topicSummaryService.generateSummary(topic); TopicSummaryResponse response = topicSummaryService.generateSummary(topic, language);
return ResponseEntity.ok(response); return ResponseEntity.ok(response);
} }
@GetMapping("/{id}/summaries")
public ResponseEntity<List<SavedSummaryItem>> listSummaries(@PathVariable String id) {
topicRepository.findById(id)
.orElseThrow(() -> new NoSuchElementException("Topic not found."));
return ResponseEntity.ok(topicSummaryService.listSummaries(id));
}
@GetMapping("/{id}/summaries/{summaryId}")
public ResponseEntity<TopicSummaryResponse> getSummary(@PathVariable String id,
@PathVariable UUID summaryId) {
topicRepository.findById(id)
.orElseThrow(() -> new NoSuchElementException("Topic not found."));
return ResponseEntity.ok(topicSummaryService.getSummary(summaryId));
}
} }
@@ -0,0 +1,53 @@
package com.aiteacher.topic;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
import java.time.Instant;
import java.util.UUID;
@Entity
@Table(name = "topic_summary")
public class TopicSummaryEntity {
@Id
@GeneratedValue(strategy = GenerationType.UUID)
private UUID id;
@Column(name = "topic_id", nullable = false)
private String topicId;
@Column(name = "summary_number", nullable = false)
private int summaryNumber;
@Column(nullable = false, columnDefinition = "TEXT")
private String summary;
@Column(name = "sources_json", nullable = false, columnDefinition = "TEXT")
private String sourcesJson;
@Column(name = "generated_at", nullable = false)
private Instant generatedAt;
protected TopicSummaryEntity() {}
public TopicSummaryEntity(String topicId, int summaryNumber, String summary,
String sourcesJson, Instant generatedAt) {
this.topicId = topicId;
this.summaryNumber = summaryNumber;
this.summary = summary;
this.sourcesJson = sourcesJson;
this.generatedAt = generatedAt;
}
public UUID getId() { return id; }
public String getTopicId() { return topicId; }
public int getSummaryNumber() { return summaryNumber; }
public String getSummary() { return summary; }
public String getSourcesJson() { return sourcesJson; }
public Instant getGeneratedAt() { return generatedAt; }
}
@@ -0,0 +1,13 @@
package com.aiteacher.topic;
import org.springframework.data.jpa.repository.JpaRepository;
import java.util.List;
import java.util.UUID;
public interface TopicSummaryRepository extends JpaRepository<TopicSummaryEntity, UUID> {
List<TopicSummaryEntity> findByTopicIdOrderBySummaryNumberAsc(String topicId);
long countByTopicId(String topicId);
}
@@ -2,8 +2,11 @@ package com.aiteacher.topic;
import java.time.Instant; import java.time.Instant;
import java.util.List; import java.util.List;
import java.util.UUID;
public record TopicSummaryResponse( public record TopicSummaryResponse(
UUID id,
int summaryNumber,
String topicId, String topicId,
String topicName, String topicName,
String summary, String summary,
@@ -11,8 +14,17 @@ public record TopicSummaryResponse(
Instant generatedAt Instant generatedAt
) { ) {
public record SourceReference( public record SourceReference(
String type,
String refLabel,
String bookId,
String bookTitle, String bookTitle,
Integer page Integer page,
String chunkText,
String figureId,
String label,
String caption,
String figureType,
String imageUrl
) { ) {
} }
} }
@@ -1,21 +1,25 @@
package com.aiteacher.topic; package com.aiteacher.topic;
import com.aiteacher.book.Book;
import com.aiteacher.book.BookRepository; import com.aiteacher.book.BookRepository;
import com.aiteacher.book.BookStatus; import com.aiteacher.book.BookStatus;
import com.aiteacher.book.NoKnowledgeSourceException; import com.aiteacher.book.NoKnowledgeSourceException;
import com.aiteacher.document.FigureEntity;
import com.aiteacher.document.SectionEntity;
import com.aiteacher.retrieval.NeurosurgeryRetriever;
import com.aiteacher.retrieval.RetrievalResult;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.slf4j.Logger; import org.slf4j.Logger;
import org.slf4j.LoggerFactory; import org.slf4j.LoggerFactory;
import org.springframework.ai.chat.client.ChatClient; import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.document.Document;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import java.time.Instant; import java.time.Instant;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.List; import java.util.List;
import java.util.Map; import java.util.NoSuchElementException;
import java.util.UUID;
@Service @Service
public class TopicSummaryService { public class TopicSummaryService {
@@ -23,86 +27,222 @@ public class TopicSummaryService {
private static final Logger log = LoggerFactory.getLogger(TopicSummaryService.class); private static final Logger log = LoggerFactory.getLogger(TopicSummaryService.class);
private static final String SYSTEM_PROMPT = """ private static final String SYSTEM_PROMPT = """
You are an expert neurosurgery educator. Your role is to provide accurate, You are an expert neurosurgery educator. Your role is to provide accurate, detailed but synthetically concise educational reports on neurosurgery topics, based on the content retrieved from the uploaded medical textbooks. Your audience is highly experienced neurosurgeons, who are looking for a comprehensive yet digestible overview of a specific topic.
clinically relevant summaries based ONLY on the content retrieved from the When generating reports, your primary goal is to distill the most important and clinically relevant information about the topic. This includes key concepts, anatomical details, surgical techniques, clinical considerations, and any other information that would be essential for a neurosurgeon to understand the topic thoroughly.
uploaded medical textbooks. Do not use any knowledge outside the provided context. Base your reports on uploaded medical textbooks. Do not use any knowledge outside the provided context.
When answering: When answering:
- Structure your response clearly with key points - Structure your response clearly with key points
- If the context mentions specific book titles and page numbers, reference them - Cite claims using ONLY the reference labels provided in the context (e.g. [S1], [F2]).
Do not invent page numbers, section titles, or labels not present in the CONTEXT block.
- Figures (labeled [F1], [F2], etc.) are actual images and drawings from the textbook — they will be rendered as inline illustrations in your response. Use them actively to support your explanations: reference a figure when it visually demonstrates anatomy, a surgical step, or a clinical concept you are describing.
- If the retrieved context does not contain sufficient information on the topic, - If the retrieved context does not contain sufficient information on the topic,
explicitly state: "The uploaded books do not contain sufficient information on this topic." explicitly state: "The uploaded books do not contain sufficient information on this topic."
- Never hallucinate or fabricate clinical information - Never hallucinate or fabricate clinical information
"""; """;
private final ChatClient chatClient; private final ChatClient chatClient;
private final VectorStore vectorStore;
private final BookRepository bookRepository; private final BookRepository bookRepository;
private final NeurosurgeryRetriever retriever;
private final TopicSummaryRepository summaryRepository;
private final ObjectMapper objectMapper;
public TopicSummaryService(ChatClient chatClient, VectorStore vectorStore, public TopicSummaryService(ChatClient chatClient,
BookRepository bookRepository) { BookRepository bookRepository,
NeurosurgeryRetriever retriever,
TopicSummaryRepository summaryRepository,
ObjectMapper objectMapper) {
this.chatClient = chatClient; this.chatClient = chatClient;
this.vectorStore = vectorStore;
this.bookRepository = bookRepository; this.bookRepository = bookRepository;
this.retriever = retriever;
this.summaryRepository = summaryRepository;
this.objectMapper = objectMapper;
} }
public TopicSummaryResponse generateSummary(Topic topic) { public TopicSummaryResponse generateSummary(Topic topic, String language) {
if (!bookRepository.existsByStatus(BookStatus.READY)) { List<Book> readyBooks = bookRepository.findAll().stream()
.filter(b -> b.getStatus() == BookStatus.READY)
.toList();
if (readyBooks.isEmpty()) {
throw new NoKnowledgeSourceException( throw new NoKnowledgeSourceException(
"No books are available as knowledge sources. Please upload and process at least one book."); "No books are available as knowledge sources. Please upload and process at least one book.");
} }
String question = buildQuestion(topic); String question = buildQuestion(topic);
ChatResponse response = chatClient.prompt() List<SectionEntity> allSections = new ArrayList<>();
.system(SYSTEM_PROMPT) List<FigureEntity> allFigures = new ArrayList<>();
.advisors(QuestionAnswerAdvisor.builder(vectorStore).build()) for (Book book : readyBooks) {
.user(question) RetrievalResult result = retriever.retrieve(question, book.getId());
.call() allSections.addAll(result.parentSections());
.chatResponse(); allFigures.addAll(result.figures());
}
String summary = response.getResult().getOutput().getText(); log.debug("Topic reports for '{}': {} sections, {} figures retrieved",
List<TopicSummaryResponse.SourceReference> sources = extractSources(response); topic.getName(), allSections.size(), allFigures.size());
String contextPrompt = buildContextPrompt(question, allSections, allFigures, language);
String summary = chatClient.prompt()
.system(SYSTEM_PROMPT)
.user(contextPrompt)
.call()
.content();
List<TopicSummaryResponse.SourceReference> sources = buildSources(allSections, allFigures, readyBooks);
Instant generatedAt = Instant.now();
int summaryNumber = (int) summaryRepository.countByTopicId(topic.getId()) + 1;
String sourcesJson = serializeSources(sources);
TopicSummaryEntity entity = new TopicSummaryEntity(
topic.getId(), summaryNumber, summary, sourcesJson, generatedAt);
entity = summaryRepository.save(entity);
return new TopicSummaryResponse( return new TopicSummaryResponse(
entity.getId(),
summaryNumber,
topic.getId(), topic.getId(),
topic.getName(), topic.getName(),
summary, summary,
sources, sources,
Instant.now() generatedAt
);
}
public List<SavedSummaryItem> listSummaries(String topicId) {
return summaryRepository.findByTopicIdOrderBySummaryNumberAsc(topicId).stream()
.map(e -> new SavedSummaryItem(e.getId(), e.getSummaryNumber(), e.getGeneratedAt()))
.toList();
}
public TopicSummaryResponse getSummary(UUID summaryId) {
TopicSummaryEntity entity = summaryRepository.findById(summaryId)
.orElseThrow(() -> new NoSuchElementException("Summary not found."));
List<TopicSummaryResponse.SourceReference> sources = deserializeSources(entity.getSourcesJson());
return new TopicSummaryResponse(
entity.getId(),
entity.getSummaryNumber(),
entity.getTopicId(),
entity.getTopicId(),
entity.getSummary(),
sources,
entity.getGeneratedAt()
); );
} }
private String buildQuestion(Topic topic) { private String buildQuestion(Topic topic) {
return String.format( return String.format(
"Please provide a comprehensive educational summary of the following neurosurgery topic: " + "Provide a comprehensive educational report of the following neurosurgery topic: " +
"%s. Topic description: %s. " + "%s. Topic description: %s. ",
"Include key concepts, clinical considerations, and important details that a neurosurgeon should know.",
topic.getName(), topic.getDescription() topic.getName(), topic.getDescription()
); );
} }
private List<TopicSummaryResponse.SourceReference> extractSources(ChatResponse response) { private String buildContextPrompt(String question,
List<TopicSummaryResponse.SourceReference> sources = new ArrayList<>(); List<SectionEntity> sections,
List<FigureEntity> figures,
String language) {
StringBuilder sb = new StringBuilder();
if (response.getMetadata() != null) { if (!sections.isEmpty()) {
Object retrieved = response.getMetadata().get(QuestionAnswerAdvisor.RETRIEVED_DOCUMENTS); sb.append("CONTEXT:\n\n");
if (retrieved instanceof List<?> docs) { for (int i = 0; i < sections.size(); i++) {
for (Object docObj : docs) { SectionEntity s = sections.get(i);
if (docObj instanceof Document doc) { sb.append("[S").append(i + 1).append("] ")
Map<String, Object> metadata = doc.getMetadata(); .append(s.getTitle()).append(", p.").append(s.getPageStart()).append("\n");
String bookTitle = (String) metadata.get("book_title"); sb.append(s.getFullText()).append("\n\n");
Object pageObj = metadata.get("page_number");
Integer page = pageObj instanceof Number n ? n.intValue() : null;
if (bookTitle != null) {
sources.add(new TopicSummaryResponse.SourceReference(bookTitle, page));
}
}
}
} }
} }
// Deduplicate by bookTitle + page if (!figures.isEmpty()) {
return sources.stream().distinct().toList(); sb.append("AVAILABLE FIGURES:\n");
for (int i = 0; i < figures.size(); i++) {
FigureEntity f = figures.get(i);
sb.append("[F").append(i + 1).append("] ")
.append(f.getLabel() != null ? f.getLabel() : "Figure")
.append(" (p.").append(f.getPage()).append("): ")
.append(f.getCaption() != null ? f.getCaption() : "")
.append("\n");
}
sb.append("\nWhen referencing diagrams, use their label from the context (e.g. [F1]).\n\n");
}
sb.append("QUESTION:\n").append(question);
if ("th".equalsIgnoreCase(language)) {
sb.append("\n\nIMPORTANT: Write the narrative in Thai. ")
.append("Keep all medical, anatomical, surgical, pharmacological, and clinical ")
.append("terminology in English (e.g., cerebellopontine angle, glioblastoma, craniotomy, ")
.append("dexamethasone). Do NOT translate disease names, anatomical structures, drug names, ")
.append("procedures, eponyms, or imaging modalities. Translate only connective prose, ")
.append("explanations, and general descriptions. Citation labels [S#]/[F#] stay unchanged. ")
.append("The sentinel string for insufficient context must remain exactly: ")
.append("\"The uploaded books do not contain sufficient information on this topic.\"");
}
return sb.toString();
}
private List<TopicSummaryResponse.SourceReference> buildSources(List<SectionEntity> sections,
List<FigureEntity> figures,
List<Book> readyBooks) {
List<TopicSummaryResponse.SourceReference> sources = new ArrayList<>();
for (int i = 0; i < sections.size(); i++) {
SectionEntity s = sections.get(i);
Book book = readyBooks.stream()
.filter(b -> b.getId().equals(s.getBookId()))
.findFirst()
.orElse(null);
String title = book != null ? book.getTitle() : "Book";
String bookId = book != null ? book.getId().toString() : null;
sources.add(new TopicSummaryResponse.SourceReference(
"TEXT", "S" + (i + 1), bookId, title, s.getPageStart(),
truncate(s.getFullText(), 500), null, null, null, null, null));
}
for (int i = 0; i < figures.size(); i++) {
FigureEntity f = figures.get(i);
Book book = readyBooks.stream()
.filter(b -> b.getId().equals(f.getBookId()))
.findFirst()
.orElse(null);
String title = book != null ? book.getTitle() : "Book";
String bookId = book != null ? book.getId().toString() : null;
String filename = f.getImagePath().substring(f.getImagePath().lastIndexOf('/') + 1);
String imageUrl = "/api/v1/figures/" + f.getBookId() + "/" + filename;
sources.add(new TopicSummaryResponse.SourceReference(
"FIGURE", "F" + (i + 1), bookId, title, f.getPage(),
null, f.getId(), f.getLabel(), f.getCaption(),
f.getFigureType().name(), imageUrl));
}
return sources;
}
private String serializeSources(List<TopicSummaryResponse.SourceReference> sources) {
try {
return objectMapper.writeValueAsString(sources);
} catch (JsonProcessingException e) {
log.warn("Failed to serialize sources, storing empty array", e);
return "[]";
}
}
private String truncate(String text, int maxChars) {
if (text == null) return "";
return text.length() <= maxChars ? text : text.substring(0, maxChars) + "";
}
private List<TopicSummaryResponse.SourceReference> deserializeSources(String json) {
try {
return objectMapper.readValue(json,
objectMapper.getTypeFactory().constructCollectionType(
List.class, TopicSummaryResponse.SourceReference.class));
} catch (JsonProcessingException e) {
log.warn("Failed to deserialize sources from stored JSON", e);
return List.of();
}
} }
} }
+27 -3
View File
@@ -7,7 +7,7 @@ spring:
jpa: jpa:
hibernate: hibernate:
ddl-auto: update ddl-auto: none
show-sql: false show-sql: false
properties: properties:
hibernate: hibernate:
@@ -27,10 +27,11 @@ spring:
index-type: HNSW index-type: HNSW
initialize-schema: false initialize-schema: false
openai: openai:
api-key: ${OPENAI_API_KEY} api-key: ${OPENAI_API_KEY:}
chat: chat:
options: options:
model: gpt-4o model: o4-mini
reasoning-effort: high
embedding: embedding:
options: options:
model: "text-embedding-3-small" model: "text-embedding-3-small"
@@ -47,6 +48,29 @@ spring:
max-size: 8 max-size: 8
queue-capacity: 50 queue-capacity: 50
logging:
level:
"[org.apache.pdfbox]": ERROR
app: app:
features:
upload-enabled: ${UPLOAD_ENABLED:true}
delete-enabled: ${DELETE_ENABLED:true}
auth: auth:
username: ${APP_AUTH_USERNAME:neurosurgeon}
password: ${APP_PASSWORD:changeme} password: ${APP_PASSWORD:changeme}
figure-storage:
endpoint: ${S3_ENDPOINT:https://s3.immich-ad.ovh}
region: ${S3_REGION:garage}
bucket: ${S3_BUCKET:aiteacher}
access-key-id: ${S3_ACCESS_KEY_ID:}
secret-access-key: ${S3_SECRET_ACCESS_KEY:}
min-image-size-px: 100
embedding:
batch-size: 20
batch-delay-ms: 2000
skip-embedding: false
marker:
base-url: ${MARKER_BASE_URL:http://192.168.1.105:8000}
vision:
min-interval-ms: ${VISION_MIN_INTERVAL_MS:2000}
@@ -0,0 +1,28 @@
-- ============================================================
-- V4: Document hierarchy — chapter and section tables
-- Supports parent-child retrieval pattern for RAG precision.
-- ============================================================
CREATE TABLE IF NOT EXISTS chapter (
id VARCHAR(200) PRIMARY KEY,
book_id UUID NOT NULL REFERENCES book(id) ON DELETE CASCADE,
number INT NOT NULL DEFAULT 1,
title VARCHAR(500),
page_start INT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS section (
id VARCHAR(200) PRIMARY KEY,
chapter_id VARCHAR(200) NOT NULL REFERENCES chapter(id) ON DELETE CASCADE,
book_id UUID NOT NULL REFERENCES book(id) ON DELETE CASCADE,
number VARCHAR(50),
title VARCHAR(500),
page_start INT NOT NULL,
page_end INT NOT NULL,
full_text TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_section_book ON section(book_id);
CREATE INDEX IF NOT EXISTS idx_section_chapter ON section(chapter_id);
@@ -0,0 +1,29 @@
-- ============================================================
-- V5: Figures and chunk-to-figure reference table
-- figure: metadata + file path for each extracted image
-- chunk_figure_ref: links vector-store chunks to figures
-- ============================================================
CREATE TABLE IF NOT EXISTS figure (
id VARCHAR(200) PRIMARY KEY,
book_id UUID NOT NULL REFERENCES book(id) ON DELETE CASCADE,
section_id VARCHAR(200) REFERENCES section(id) ON DELETE SET NULL,
chapter_id VARCHAR(200) REFERENCES chapter(id) ON DELETE SET NULL,
label VARCHAR(100),
caption TEXT,
figure_type VARCHAR(50) NOT NULL,
page INT NOT NULL,
image_path VARCHAR(1000) NOT NULL,
caption_embedding_id UUID,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS chunk_figure_ref (
chunk_id UUID NOT NULL,
figure_id VARCHAR(200) NOT NULL REFERENCES figure(id) ON DELETE CASCADE,
mention_page INT,
PRIMARY KEY (chunk_id, figure_id)
);
CREATE INDEX IF NOT EXISTS idx_figure_book ON figure(book_id);
CREATE INDEX IF NOT EXISTS idx_cfr_chunk ON chunk_figure_ref(chunk_id);
@@ -0,0 +1,10 @@
CREATE TABLE topic_summary (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
topic_id VARCHAR(100) NOT NULL,
summary_number INT NOT NULL,
summary TEXT NOT NULL,
sources_json TEXT NOT NULL,
generated_at TIMESTAMPTZ NOT NULL
);
CREATE INDEX idx_topic_summary_topic_id ON topic_summary(topic_id, summary_number);
@@ -0,0 +1,14 @@
CREATE TABLE chunk_metadata (
chunk_id UUID PRIMARY KEY,
book_id UUID NOT NULL,
section_id VARCHAR(200) NOT NULL,
facet VARCHAR(32) NOT NULL,
entities JSONB NOT NULL,
summary TEXT NOT NULL,
model_version VARCHAR(32) NOT NULL,
enriched_at TIMESTAMPTZ NOT NULL
);
CREATE INDEX idx_chunk_metadata_book ON chunk_metadata(book_id);
CREATE INDEX idx_chunk_metadata_book_facet ON chunk_metadata(book_id, facet);
CREATE INDEX idx_chunk_metadata_entities_gin ON chunk_metadata USING GIN (entities jsonb_path_ops);
@@ -0,0 +1,11 @@
CREATE TABLE concept_report (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
topic_id VARCHAR(100) NOT NULL,
report_number INT NOT NULL,
facets_json TEXT NOT NULL,
sources_json TEXT NOT NULL,
generated_at TIMESTAMPTZ NOT NULL,
UNIQUE (topic_id, report_number)
);
CREATE INDEX idx_concept_report_topic ON concept_report(topic_id, report_number);
@@ -0,0 +1,19 @@
ALTER TABLE chunk_metadata DROP CONSTRAINT IF EXISTS chunk_metadata_facet_check;
ALTER TABLE chunk_metadata
ADD CONSTRAINT chunk_metadata_facet_check
CHECK (facet IN (
'DEFINITION',
'ANATOMY',
'PATHOPHYSIOLOGY',
'EPIDEMIOLOGY',
'CLINICAL_PRESENTATION',
'IMAGING',
'CLASSIFICATION',
'INDICATIONS',
'SURGICAL_TECHNIQUE',
'NONSURGICAL_MANAGEMENT',
'COMPLICATIONS',
'OUTCOMES_FOLLOWUP',
'OTHER'
));
+172
View File
@@ -0,0 +1,172 @@
# Concept Retrieval via Indexing-Time Chunk Enrichment
## Context
Vector similarity alone can't answer "tell me everything about aneurysms." It surfaces the chunks most *linguistically* similar to the query, not the set of all chunks that *concern* the concept — and it has no notion of whether each chunk is a definition, a case, a technique, or a complication.
The unlock is to move intelligence from query time to indexing time: for every text chunk, use an LLM to extract **structured metadata** (entities, facet, summary). At retrieval time, concept lookup becomes an SQL filter (`entities @> ['aneurysm']`) bucketed by facet — deterministic, exhaustive, and organized by default. Vector search remains as a fallback for typos / synonyms and for ranking within a facet.
This plan covers: (1) defining the metadata schema, (2) enriching chunks during new book ingestion, (3) back-filling the already-embedded corpus via an admin endpoint, (4) a new concept retrieval path, and (5) a Topics-page UI to surface the result.
## Approach
### 1. Data model — new `chunk_metadata` table
Flyway migration `backend/src/main/resources/db/migration/V7__chunk_metadata.sql`:
```sql
CREATE TABLE chunk_metadata (
chunk_id VARCHAR(64) PRIMARY KEY, -- same UUID that TextChunkingService issues and stores in vectorstore
book_id UUID NOT NULL,
section_id VARCHAR(255) NOT NULL,
facet VARCHAR(32) NOT NULL, -- enum (see ConceptFacet)
entities JSONB NOT NULL, -- canonical lowercase string[]
summary TEXT NOT NULL,
model_version VARCHAR(32) NOT NULL, -- records which LLM/prompt version tagged this chunk
enriched_at TIMESTAMPTZ NOT NULL
);
CREATE INDEX idx_chunk_metadata_book ON chunk_metadata(book_id);
CREATE INDEX idx_chunk_metadata_book_facet ON chunk_metadata(book_id, facet);
CREATE INDEX idx_chunk_metadata_entities_gin ON chunk_metadata USING GIN (entities jsonb_path_ops);
```
Why `chunk_id` is the natural key: `TextChunkingService` already generates a UUID per chunk, uses it as the pgvector Document id, stores it in metadata, and it's the key in `ChunkFigureRefEntity` — so the table joins cleanly to everything already in place.
### 2. Enrichment service & facet taxonomy
New package `com.aiteacher.enrichment`:
- `ConceptFacet` enum — 13 values tailored to neurosurgery textbooks: `DEFINITION, ANATOMY, PATHOPHYSIOLOGY, EPIDEMIOLOGY, CLINICAL_PRESENTATION, IMAGING, CLASSIFICATION, INDICATIONS, SURGICAL_TECHNIQUE, NONSURGICAL_MANAGEMENT, COMPLICATIONS, OUTCOMES_FOLLOWUP, OTHER`. `OTHER` is mandatory so the LLM always has an out (no hallucinated bucketing). The prompt carries explicit disambiguation rules (named grading scales → `CLASSIFICATION`; imaging of a complication → `COMPLICATIONS`; tools inside an operation → `SURGICAL_TECHNIQUE`).
- `ChunkEnrichmentResult` — record `(List<String> entities, ConceptFacet facet, String summary)`
- `ChunkEnrichmentService` — single method `enrich(String chunkText, SectionEntity section, String bookTitle) → ChunkEnrichmentResult`. Uses Spring AI `ChatClient.prompt().call().entity(Class)` for structured output. The prompt gives: book title, section title, chunk text, the fixed facet enum list, and instructs the model to return JSON with entities normalised to lowercase singular canonical form (e.g. "aneurysms" → "aneurysm"; "SAH" → "subarachnoid hemorrhage"). Caps entities at ~8 per chunk.
- `ChunkMetadataEntity` + `ChunkMetadataRepository` — JPA entity/repo mirroring the table.
Model version string (e.g. `"v1"`) lives on the service and is stamped into each row so a future prompt rev can be rolled out by filtering `model_version <> 'v2'` in the backfill job.
### 3. Hook into new book ingestion
Modify `BookEmbeddingService.embedBook`:
```java
// Step 3: Chunk and embed text
List<Document> allChunks = new ArrayList<>();
for (SectionEntity section : sections) {
allChunks.addAll(textChunkingService.chunk(section, bookTitle));
}
if (skipEmbedding) { ... } else {
embedInBatches(allChunks, bookId);
chunkEnrichmentPipeline.enrichAndPersist(allChunks, sectionsById, bookTitle); // NEW
}
```
- `ChunkEnrichmentPipeline` — new orchestrator that iterates chunks, calls `ChunkEnrichmentService.enrich(...)` per chunk, saves `ChunkMetadataEntity` rows in batches, with the same throttle pattern as `embedInBatches`.
- Runs *after* embedding, not in place of it, so a failure in enrichment doesn't corrupt the vector store. On failure, log and continue — the backfill endpoint is the universal recovery path.
- Extend `deleteBookChunks` to also delete `chunk_metadata` rows so deletion stays consistent.
### 4. Backfill endpoint for already-embedded books
New `EnrichmentController` in `com.aiteacher.enrichment`:
- `POST /api/v1/admin/books/{id}/enrich` → kicks off async backfill, returns 202 with `{status, chunksTotal, chunksEnriched}`
- `GET /api/v1/admin/books/{id}/enrich` → returns progress
Backfill flow (`EnrichmentBackfillService.backfillBook(UUID bookId)`):
1. Query the pgvector storage table directly via `JdbcTemplate` for all chunks of the book:
```sql
SELECT id, content, metadata
FROM vector_store
WHERE metadata->>'book_id' = ? AND metadata->>'type' = 'TEXT'
```
2. Left-anti-join against `chunk_metadata` to skip already-enriched chunks → idempotent, resumable.
3. For each missing chunk: look up its `SectionEntity` via `section_id` in metadata, call `ChunkEnrichmentService.enrich`, write a `ChunkMetadataEntity` row.
4. Progress tracked in an in-memory `ConcurrentHashMap<UUID, BackfillProgress>` (POC scope — no cross-restart resumability needed because the left-anti-join makes re-runs free).
5. `@Async` on the backfill method using the same executor as `embedBook`.
### 5. Concept retrieval path
New `com.aiteacher.concept.ConceptRetriever`:
```java
public ConceptRetrievalResult retrieveByConcept(String conceptKeyword, UUID bookId) {
String canonical = canonicalise(conceptKeyword); // lowercase, trim, simple plural strip
// 5a. Primary: SQL entity match, grouped by facet
List<ChunkMetadataEntity> hits = chunkMetadataRepository
.findByBookIdAndEntityContains(bookId, canonical); // WHERE entities @> to_jsonb(?::text)
if (hits.isEmpty()) {
// 5b. Fallback: vector search, then enrich-join + facet-group
List<Document> vectorHits = vectorStore.similaritySearch(/* TEXT filter, book_id filter, topK=30 */);
List<String> chunkIds = vectorHits.stream().map(Document::getId).toList();
hits = chunkMetadataRepository.findByChunkIdIn(chunkIds);
}
Map<ConceptFacet, List<ChunkMetadataEntity>> byFacet = hits.stream()
.collect(groupingBy(ChunkMetadataEntity::getFacet, LinkedHashMap::new, toList()));
// Hydrate: load SectionEntity for each chunk's section_id; load linked figures
// via ChunkFigureRefRepository.findByChunkIdIn(chunkIds) — reuses existing linkage.
return assemble(byFacet, ...);
}
```
`ConceptRetrievalResult` = `Map<ConceptFacet, FacetBundle>` where each `FacetBundle` holds the parent sections, linked figures, and the per-chunk `summary` strings.
Cross-book aggregation: caller loops over READY books and merges bundles by facet.
### 6. Concept Report service & controller
New `ConceptReportService` in `com.aiteacher.concept` — mirrors the shape of `TopicSummaryService`, but:
- Calls `ConceptRetriever.retrieveByConcept(topic.getName(), bookId)` per book.
- For each facet that has hits, sends **one** LLM synthesis call with the chunks/figures of that facet — producing a structured, facet-labelled report.
- Persists in a new `concept_report` table:
```sql
CREATE TABLE concept_report (
id UUID PRIMARY KEY,
topic_id VARCHAR(255) NOT NULL REFERENCES topic(id),
report_number INT NOT NULL,
facets_json JSONB NOT NULL, -- [{facetKey,title,markdown,refLabels[]}, ...]
sources_json JSONB NOT NULL, -- deduplicated SourceReference[]
generated_at TIMESTAMPTZ NOT NULL,
UNIQUE (topic_id, report_number)
);
```
Controller `ConceptReportController` exposes three endpoints under `/api/v1/topics/{id}/concept-reports` (POST generate, GET list, GET `/{reportId}`).
Reuses `TopicSummaryResponse.SourceReference` verbatim.
### 7. Frontend
- `frontend/src/stores/topicStore.ts`: add parallel state `conceptReportList`, `activeConceptReport`, `conceptReportLoading`, and actions mirroring the existing summary ones.
- `frontend/src/views/TopicsView.vue`: add a **Summary / Concept Report** tab toggle at the top of the topic panel. Concept Report reuses the history-chips + Generate button UI. Report body renders each `FacetSection` as `<h3>{title}</h3>` + markdown.
- Loading hint: update the "up to 30 seconds" copy to "up to 60 seconds".
### 8. README update
Add an **Indexing Pipeline** diagram showing: PDF → parse → chunk → embed → **enrich (new)** → chunk_metadata. Plus a **Concept Retrieval** sequence diagram: query → entity-match SQL → facet-grouped bundle → synthesis → report.
## Decisions & trade-offs
- **Storage as separate Postgres table, not vectorstore JSON**: vectorstore has no metadata-only update API, backfill would require delete+reinsert (re-embedding cost). A dedicated table joins cleanly on `chunk_id` and is GIN-indexed.
- **Entity-match primary, vector fallback**: deterministic for the main use case, robust against typos/synonyms. Vector search stays the default for normal chat retrieval — this feature is additive.
- **Enrichment runs *after* embedding, not before**: keeps the two failure modes independent. The backfill endpoint is the universal recovery lever.
- **Fixed 9-value facet enum** (incl. `OTHER`): constrains LLM outputs; `OTHER` prevents forced mis-bucketing.
- **Direct `JdbcTemplate` read against `vector_store` for backfill**: Spring AI exposes no listing API. Acceptable for a POC, isolated behind one method.
- **Synchronous (sequential) LLM calls**: simplest; parallelism is a later optimisation if needed.
- **`model_version` column**: cheap insurance. If the prompt or facet taxonomy changes, backfill can re-enrich only stale rows.
## Verification
1. Migration applies V7 and V8. Tables and indexes created.
2. New book ingestion: upload PDF → `chunk_metadata` populated with plausible entities/facets/summaries.
3. Backfill: POST `/api/v1/admin/books/{id}/enrich` → idempotent, completes, re-run is a no-op.
4. Concept retrieval primary path: POST `/api/v1/topics/aneurysm/concept-reports` → 200 with facets populated.
5. Fallback path: misspelled topic still returns results via vector fallback.
6. Frontend: Concept Report tab renders facet-labelled markdown + sources + inline figures; persists across reloads.
7. Deletion: removing a book cascades to `chunk_metadata` rows.
8. Regression: existing chat and summary flows still work.
9. Lint & tests pass.
+37
View File
@@ -0,0 +1,37 @@
version: '3.9'
services:
postgres:
image: pgvector/pgvector:pg16
container_name: aiteacher-postgres-native
environment:
POSTGRES_DB: aiteacher
POSTGRES_USER: aiteacher
POSTGRES_PASSWORD: aiteacher
ports:
- "5432:5432"
volumes:
- pgdata_native:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U aiteacher -d aiteacher"]
interval: 10s
timeout: 5s
retries: 5
backend:
image: ai-teacher-backend:latest
container_name: aiteacher-backend-native
env_file:
- .env
environment:
SPRING_DATASOURCE_URL: jdbc:postgresql://postgres:5432/aiteacher
SPRING_DATASOURCE_USERNAME: aiteacher
SPRING_DATASOURCE_PASSWORD: aiteacher
ports:
- "8080:8080"
depends_on:
postgres:
condition: service_healthy
volumes:
pgdata_native:
+9 -2
View File
@@ -3,5 +3,12 @@
# In production point it directly at the backend, e.g. https://api.example.com/api/v1 # In production point it directly at the backend, e.g. https://api.example.com/api/v1
VITE_API_URL=/api/v1 VITE_API_URL=/api/v1
# Shared password for HTTP Basic auth (must match APP_PASSWORD on the backend). # Credentials are no longer configured here. Users enter their username and
VITE_APP_PASSWORD=changeme # password via the login form. The backend validates them via HTTP Basic Auth.
# Configure the backend credentials with APP_AUTH_USERNAME and APP_PASSWORD.
# Set to 'false' to hide the upload UI (frontend). Also set UPLOAD_ENABLED=false on the backend to block the endpoint.
VITE_UPLOAD_ENABLED=true
# Set to 'false' to hide the delete button (frontend). Also set DELETE_ENABLED=false on the backend to block the endpoint.
VITE_DELETE_ENABLED=true
+5 -3
View File
@@ -1,5 +1,5 @@
# ---- Build stage ---- # ---- Build stage ----
FROM node:20-alpine AS build FROM docker.io/library/node:20-alpine AS build
WORKDIR /app WORKDIR /app
COPY package*.json ./ COPY package*.json ./
RUN npm ci RUN npm ci
@@ -7,8 +7,10 @@ COPY . .
RUN npm run build RUN npm run build
# ---- Runtime stage (nginx) ---- # ---- Runtime stage (nginx) ----
FROM nginx:alpine FROM docker.io/library/nginx:alpine
COPY --from=build /app/dist /usr/share/nginx/html COPY --from=build /app/dist /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf COPY nginx.conf /etc/nginx/conf.d/default.conf
COPY docker-entrypoint.sh /docker-entrypoint.sh
RUN chmod +x /docker-entrypoint.sh
EXPOSE 80 EXPOSE 80
CMD ["nginx", "-g", "daemon off;"] ENTRYPOINT ["/docker-entrypoint.sh"]
+16
View File
@@ -0,0 +1,16 @@
#!/bin/sh
set -e
# Write runtime env vars into a JS file loaded before the app bundle.
# Any VITE_* variable passed via `docker run -e` will be available as
# window.__env__.VITE_* inside the browser.
cat > /usr/share/nginx/html/env-config.js <<EOF
window.__env__ = {
VITE_API_URL: "${VITE_API_URL:-}",
VITE_APP_PASSWORD: "${VITE_APP_PASSWORD:-}",
VITE_UPLOAD_ENABLED: "${VITE_UPLOAD_ENABLED:-}",
VITE_DELETE_ENABLED: "${VITE_DELETE_ENABLED:-}"
};
EOF
exec nginx -g "daemon off;"
+1
View File
@@ -8,6 +8,7 @@
</head> </head>
<body> <body>
<div id="app"></div> <div id="app"></div>
<script src="/env-config.js"></script>
<script type="module" src="/src/main.ts"></script> <script type="module" src="/src/main.ts"></script>
</body> </body>
</html> </html>
+156 -21
View File
@@ -6,23 +6,31 @@
<span class="brand-name">AI Teacher</span> <span class="brand-name">AI Teacher</span>
<span class="brand-subtitle">Neurosurgeon Learning Platform</span> <span class="brand-subtitle">Neurosurgeon Learning Platform</span>
</div> </div>
<ul class="navbar-links"> <template v-if="authStore.isAuthenticated">
<li> <button class="burger" :class="{ open: menuOpen }" @click="menuOpen = !menuOpen" aria-label="Menu">
<RouterLink to="/" :class="{ active: $route.path === '/' }"> <span></span><span></span><span></span>
<span class="nav-icon">📚</span> Library </button>
</RouterLink> <div class="nav-drawer" :class="{ open: menuOpen }" @click="menuOpen = false">
</li> <ul class="navbar-links">
<li> <li>
<RouterLink to="/topics" :class="{ active: $route.path === '/topics' }"> <RouterLink to="/" :class="{ active: $route.path === '/' }">
<span class="nav-icon">🗂</span> Topics <span class="nav-icon">📚</span> Library
</RouterLink> </RouterLink>
</li> </li>
<li> <li>
<RouterLink to="/chat" :class="{ active: $route.path === '/chat' }"> <RouterLink to="/topics" :class="{ active: $route.path === '/topics' }">
<span class="nav-icon">💬</span> Chat <span class="nav-icon">🗂</span> Topics
</RouterLink> </RouterLink>
</li> </li>
</ul> <li>
<RouterLink to="/chat" :class="{ active: $route.path === '/chat' }">
<span class="nav-icon">💬</span> Chat
</RouterLink>
</li>
</ul>
<button class="btn btn-logout" @click.stop="logout">Sign out</button>
</div>
</template>
</nav> </nav>
<main class="main-content"> <main class="main-content">
@@ -35,12 +43,26 @@
</template> </template>
<script setup lang="ts"> <script setup lang="ts">
import { ref, provide } from 'vue' import { ref, provide, watch } from 'vue'
import { RouterLink, RouterView } from 'vue-router' import { RouterLink, RouterView, useRouter, useRoute } from 'vue-router'
import { useAuthStore } from '@/stores/authStore'
const authStore = useAuthStore()
const router = useRouter()
const route = useRoute()
const menuOpen = ref(false)
const toastMessage = ref('') const toastMessage = ref('')
const toastType = ref<'toast-error' | 'toast-success'>('toast-error') const toastType = ref<'toast-error' | 'toast-success'>('toast-error')
// Close menu on navigation
watch(() => route.path, () => { menuOpen.value = false })
function logout() {
authStore.clearCredentials()
router.push({ name: 'login' })
}
function showToast(message: string, type: 'error' | 'success' = 'error') { function showToast(message: string, type: 'error' | 'success' = 'error') {
toastMessage.value = message toastMessage.value = message
toastType.value = type === 'error' ? 'toast-error' : 'toast-success' toastType.value = type === 'error' ? 'toast-error' : 'toast-success'
@@ -64,11 +86,11 @@ body {
Ubuntu, Cantarell, 'Fira Sans', 'Droid Sans', 'Helvetica Neue', sans-serif; Ubuntu, Cantarell, 'Fira Sans', 'Droid Sans', 'Helvetica Neue', sans-serif;
background: #f0f4f8; background: #f0f4f8;
color: #2d3748; color: #2d3748;
min-height: 100vh; height: 100vh;
} }
#app { #app {
min-height: 100vh; height: 100vh;
display: flex; display: flex;
flex-direction: column; flex-direction: column;
} }
@@ -82,6 +104,9 @@ body {
justify-content: space-between; justify-content: space-between;
height: 64px; height: 64px;
box-shadow: 0 2px 8px rgba(0, 0, 0, 0.3); box-shadow: 0 2px 8px rgba(0, 0, 0, 0.3);
position: sticky;
top: 0;
z-index: 100;
} }
.navbar-brand { .navbar-brand {
@@ -106,6 +131,13 @@ body {
margin-left: 0.25rem; margin-left: 0.25rem;
} }
/* Desktop: links inline */
.nav-drawer {
display: flex;
align-items: center;
gap: 0.5rem;
}
.navbar-links { .navbar-links {
list-style: none; list-style: none;
display: flex; display: flex;
@@ -131,8 +163,38 @@ body {
color: white; color: white;
} }
/* Burger button — hidden on desktop */
.burger {
display: none;
flex-direction: column;
justify-content: center;
gap: 5px;
width: 36px;
height: 36px;
background: transparent;
border: none;
cursor: pointer;
padding: 4px;
border-radius: 6px;
}
.burger span {
display: block;
height: 2px;
background: #bee3f8;
border-radius: 2px;
transition: transform 0.2s, opacity 0.2s;
}
.burger.open span:nth-child(1) { transform: translateY(7px) rotate(45deg); }
.burger.open span:nth-child(2) { opacity: 0; }
.burger.open span:nth-child(3) { transform: translateY(-7px) rotate(-45deg); }
.main-content { .main-content {
flex: 1; flex: 1;
min-height: 0;
display: flex;
flex-direction: column;
padding: 2rem; padding: 2rem;
max-width: 1200px; max-width: 1200px;
margin: 0 auto; margin: 0 auto;
@@ -224,6 +286,20 @@ body {
background: #cbd5e0; background: #cbd5e0;
} }
.btn-logout {
background: transparent;
color: #bee3f8;
border: 1px solid #4a90b8;
font-size: 0.85rem;
padding: 0.4rem 0.9rem;
margin-left: 1rem;
}
.btn-logout:hover {
background: #2b6cb0;
color: white;
}
.spinner { .spinner {
display: inline-block; display: inline-block;
width: 20px; width: 20px;
@@ -284,4 +360,63 @@ body {
font-size: 0.9rem; font-size: 0.9rem;
margin-top: 0.5rem; margin-top: 0.5rem;
} }
@media (max-width: 768px) {
.navbar {
padding: 0 1rem;
}
.brand-subtitle {
display: none;
}
/* Show burger, hide desktop drawer */
.burger {
display: flex;
}
.nav-drawer {
display: none;
position: absolute;
top: 64px;
right: 0;
left: 0;
background: #1a365d;
flex-direction: column;
align-items: stretch;
padding: 0.5rem 0 1rem;
box-shadow: 0 4px 12px rgba(0, 0, 0, 0.3);
z-index: 99;
}
.nav-drawer.open {
display: flex;
}
.navbar-links {
flex-direction: column;
gap: 0;
}
.navbar-links a {
padding: 0.85rem 1.5rem;
border-radius: 0;
font-size: 1rem;
}
.navbar-links a:hover,
.navbar-links a.active {
background: #2b6cb0;
}
.btn-logout {
margin: 0.5rem 1.5rem 0;
width: calc(100% - 3rem);
justify-content: center;
}
.main-content {
padding: 1rem;
}
}
</style> </style>
+80 -2
View File
@@ -32,8 +32,32 @@
<span>{{ book.status === 'PENDING' ? 'Queued for processing...' : 'Embedding in progress...' }}</span> <span>{{ book.status === 'PENDING' ? 'Queued for processing...' : 'Embedding in progress...' }}</span>
</div> </div>
<div v-if="enrichProgress && enrichProgress.status === 'RUNNING'" class="processing-indicator">
<div class="spinner spinner-dark"></div>
<span>Enriching chunks {{ enrichProgress.chunksEnriched }} / {{ enrichProgress.chunksTotal }}</span>
</div>
<div v-if="enrichFeedback" class="enrich-feedback">{{ enrichFeedback }}</div>
<div class="book-actions"> <div class="book-actions">
<router-link
v-if="book.status === 'READY'"
:to="{ name: 'book-reader', params: { id: book.id } }"
class="btn btn-secondary"
>
Read
</router-link>
<button <button
v-if="book.status === 'READY' && uploadEnabled"
class="btn btn-secondary"
:disabled="enrichRunning"
@click="handleEnrich"
title="Enrich chunks with concept metadata"
>
{{ enrichRunning ? 'Enriching...' : 'Enrich' }}
</button>
<button
v-if="deleteEnabled"
class="btn btn-danger" class="btn btn-danger"
:disabled="book.status === 'PROCESSING' || deleting" :disabled="book.status === 'PROCESSING' || deleting"
@click="$emit('delete', book.id)" @click="$emit('delete', book.id)"
@@ -46,18 +70,62 @@
</template> </template>
<script setup lang="ts"> <script setup lang="ts">
import { computed } from 'vue' import { computed, onUnmounted, ref } from 'vue'
import type { Book } from '@/stores/bookStore' import type { Book, EnrichmentProgress } from '@/stores/bookStore'
import { useBookStore } from '@/stores/bookStore'
import { env } from '@/env';
const props = defineProps<{ const props = defineProps<{
book: Book book: Book
deleting?: boolean deleting?: boolean
deleteEnabled?: boolean
}>() }>()
defineEmits<{ defineEmits<{
(e: 'delete', id: string): void (e: 'delete', id: string): void
}>() }>()
const bookStore = useBookStore()
const enrichProgress = ref<EnrichmentProgress | null>(null)
const enrichFeedback = ref<string | null>(null)
let pollTimer: ReturnType<typeof setInterval> | null = null
const enrichRunning = computed(() => enrichProgress.value?.status === 'RUNNING')
const uploadEnabled = env('VITE_UPLOAD_ENABLED') !== 'false'
async function handleEnrich() {
enrichFeedback.value = null
const started = await bookStore.startEnrichment(props.book.id)
if (!started) {
enrichFeedback.value = bookStore.error ?? 'Enrichment failed to start.'
return
}
enrichProgress.value = started
startPolling()
}
function startPolling() {
stopPolling()
pollTimer = setInterval(async () => {
const status = await bookStore.fetchEnrichmentStatus(props.book.id)
if (!status) return
enrichProgress.value = status
if (status.status === 'COMPLETED') {
stopPolling()
enrichFeedback.value = `Enriched ${status.chunksEnriched} / ${status.chunksTotal} chunks.`
}
}, 2000)
}
function stopPolling() {
if (pollTimer != null) {
clearInterval(pollTimer)
pollTimer = null
}
}
onUnmounted(stopPolling)
const statusClass = computed(() => { const statusClass = computed(() => {
switch (props.book.status) { switch (props.book.status) {
case 'READY': case 'READY':
@@ -181,6 +249,16 @@ function formatDate(iso: string): string {
.book-actions { .book-actions {
display: flex; display: flex;
justify-content: flex-end; justify-content: flex-end;
gap: 0.5rem;
margin-top: 0.25rem; margin-top: 0.25rem;
} }
.enrich-feedback {
font-size: 0.8rem;
color: #22543d;
background: #f0fff4;
border: 1px solid #c6f6d5;
border-radius: 6px;
padding: 0.4rem 0.6rem;
}
</style> </style>
+239
View File
@@ -0,0 +1,239 @@
<template>
<div class="book-panel">
<div class="book-panel-header">
<span class="book-panel-title">{{ bookTitle || 'Book' }} p.&nbsp;{{ page }}</span>
<div class="book-panel-nav">
<button class="nav-btn" :disabled="page <= 1" @click="emit('navigate', page - 1)">&#8592;</button>
<button class="nav-btn" @click="emit('navigate', page + 1)">&#8594;</button>
</div>
<button class="close-btn" @click="emit('close')" title="Close">&#x2715;</button>
</div>
<div class="book-panel-body">
<div v-if="loading" class="panel-loading">
<div class="spinner spinner-dark" style="width:24px;height:24px;margin:0 auto 0.5rem;"></div>
<p>Loading page {{ page }}</p>
</div>
<div v-else-if="error" class="panel-error">{{ error }}</div>
<div v-else class="markdown-body" v-html="renderedHtml"></div>
</div>
</div>
</template>
<script setup lang="ts">
import { ref, watch, onMounted, onUnmounted } from 'vue'
import { api } from '@/services/api'
const props = defineProps<{
bookId: string
page: number
bookTitle?: string
}>()
const emit = defineEmits<{
close: []
navigate: [page: number]
}>()
const loading = ref(false)
const error = ref<string | null>(null)
const renderedHtml = ref('')
let activeBlobUrls: string[] = []
onMounted(() => loadPage(props.page))
watch(() => [props.bookId, props.page], () => loadPage(props.page))
onUnmounted(() => {
activeBlobUrls.forEach(u => URL.revokeObjectURL(u))
})
async function loadPage(page: number) {
loading.value = true
error.value = null
renderedHtml.value = ''
activeBlobUrls.forEach(u => URL.revokeObjectURL(u))
activeBlobUrls = []
try {
const res = await api.get<string>(`/books/${props.bookId}/pages/${page}/html`, {
headers: { Accept: 'text/html' },
responseType: 'text'
})
renderedHtml.value = await resolveImages(res.data)
} catch (e: any) {
error.value = e.message ?? 'Failed to load page.'
} finally {
loading.value = false
}
}
async function resolveImages(html: string): Promise<string> {
const srcPattern = /src="(\/api\/v1\/figures\/[^"]+)"/g
const matches = [...html.matchAll(srcPattern)]
if (matches.length === 0) return html
const unique = [...new Set(matches.map(m => m[1]))]
const blobMap: Record<string, string> = {}
await Promise.all(
unique.map(async (src) => {
try {
const res = await api.get(src.replace(/^\/api\/v1/, ''), { responseType: 'blob' })
const blobUrl = URL.createObjectURL(res.data)
activeBlobUrls.push(blobUrl)
blobMap[src] = blobUrl
} catch {
// leave original src
}
})
)
return html.replace(/src="(\/api\/v1\/figures\/[^"]+)"/g, (_, src) =>
blobMap[src] ? `src="${blobMap[src]}"` : `src="${src}"`
)
}
</script>
<style scoped>
.book-panel {
display: flex;
flex-direction: column;
height: 100%;
background: white;
border-left: 1px solid #e2e8f0;
border-radius: 0 10px 10px 0;
overflow: hidden;
}
.book-panel-header {
display: flex;
align-items: center;
gap: 0.5rem;
padding: 0.6rem 0.75rem;
background: #f7fafc;
border-bottom: 1px solid #e2e8f0;
flex-shrink: 0;
}
.book-panel-title {
flex: 1;
font-size: 0.8rem;
font-weight: 600;
color: #2b6cb0;
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
}
.book-panel-nav {
display: flex;
gap: 0.25rem;
}
.nav-btn {
width: 1.75rem;
height: 1.75rem;
border: 1px solid #cbd5e0;
border-radius: 5px;
background: white;
cursor: pointer;
font-size: 0.85rem;
display: flex;
align-items: center;
justify-content: center;
transition: background 0.15s;
}
.nav-btn:hover:not(:disabled) { background: #ebf8ff; border-color: #3182ce; }
.nav-btn:disabled { opacity: 0.4; cursor: not-allowed; }
.close-btn {
width: 1.75rem;
height: 1.75rem;
border: none;
border-radius: 5px;
background: none;
cursor: pointer;
font-size: 1rem;
color: #718096;
display: flex;
align-items: center;
justify-content: center;
transition: background 0.15s, color 0.15s;
}
.close-btn:hover { background: #fed7d7; color: #742a2a; }
.book-panel-body {
flex: 1;
overflow-y: auto;
padding: 1rem 1.25rem;
}
.panel-loading {
text-align: center;
padding: 2rem;
color: #718096;
font-size: 0.875rem;
}
.panel-error {
padding: 1rem;
background: #fff5f5;
border: 1px solid #fed7d7;
color: #742a2a;
border-radius: 6px;
font-size: 0.875rem;
}
.markdown-body {
font-size: 0.9rem;
line-height: 1.75;
color: #2d3748;
}
.markdown-body :deep(h1),
.markdown-body :deep(h2),
.markdown-body :deep(h3) {
color: #1a365d;
font-weight: 600;
margin: 1.25rem 0 0.5rem;
}
.markdown-body :deep(h2) { font-size: 1.05rem; border-bottom: 1px solid #e2e8f0; padding-bottom: 0.3rem; }
.markdown-body :deep(h3) { font-size: 0.95rem; }
.markdown-body :deep(p) { margin: 0.6rem 0; }
.markdown-body :deep(img) {
max-width: 100%;
border-radius: 6px;
display: block;
margin: 0.75rem auto;
box-shadow: 0 1px 4px rgba(0,0,0,0.12);
}
.markdown-body :deep(ul),
.markdown-body :deep(ol) { padding-left: 1.4rem; margin: 0.5rem 0; }
.markdown-body :deep(code) {
background: #f7fafc;
border: 1px solid #e2e8f0;
border-radius: 3px;
padding: 0.1em 0.3em;
font-size: 0.85em;
}
.markdown-body :deep(blockquote) {
border-left: 3px solid #3182ce;
padding-left: 0.75rem;
color: #4a5568;
margin: 0.5rem 0;
}
.markdown-body :deep(table) {
width: 100%;
border-collapse: collapse;
font-size: 0.875em;
margin: 0.75rem 0;
}
.markdown-body :deep(th),
.markdown-body :deep(td) {
border: 1px solid #e2e8f0;
padding: 0.35rem 0.6rem;
text-align: left;
}
.markdown-body :deep(th) { background: #f7fafc; font-weight: 600; }
</style>
+206 -32
View File
@@ -3,25 +3,17 @@
<div class="message-bubble" :class="isUser ? 'bubble-user' : 'bubble-assistant'"> <div class="message-bubble" :class="isUser ? 'bubble-user' : 'bubble-assistant'">
<div class="message-role">{{ isUser ? 'You' : 'AI Teacher' }}</div> <div class="message-role">{{ isUser ? 'You' : 'AI Teacher' }}</div>
<div v-if="isUser" class="message-content">{{ message.content }}</div> <div v-if="isUser" class="message-content">{{ message.content }}</div>
<div v-else class="message-content message-content--markdown" v-html="renderedContent"></div> <div v-else class="message-content message-content--markdown" v-html="renderedWithBadges" @click="onContentClick"></div>
<!-- Source chips for assistant messages --> <!-- Sources for assistant messages -->
<div v-if="!isUser && message.sources && message.sources.length > 0" class="message-sources"> <div v-if="!isUser && message.sources && message.sources.length > 0" class="message-sources">
<div class="sources-label">Sources:</div> <div class="sources-label">Sources:</div>
<div class="source-list"> <SourceList
<div ref="sourceListEl"
v-for="(source, idx) in message.sources" :sources="message.sources"
:key="idx" :active-ref="activeRef"
class="source-item" @open-source="(bookId: string, page: number) => emit('open-source', bookId, page)"
> />
<div class="source-chip">
<span class="source-book-icon">📖</span>
<span class="source-book-title">{{ source.bookTitle }}</span>
<span v-if="source.page" class="source-page">p.&nbsp;{{ source.page }}</span>
</div>
<div v-if="source.chunkText" class="source-chunk">{{ source.chunkText }}</div>
</div>
</div>
</div> </div>
<div class="message-timestamp">{{ formatTime(message.createdAt) }}</div> <div class="message-timestamp">{{ formatTime(message.createdAt) }}</div>
@@ -30,16 +22,72 @@
</template> </template>
<script setup lang="ts"> <script setup lang="ts">
import { computed } from 'vue' import { computed, ref } from 'vue'
import { marked } from 'marked' import { marked } from 'marked'
import type { ChatMessage } from '@/stores/chatStore' import type { ChatMessage, ChatSource } from '@/stores/chatStore'
import SourceList from '@/components/SourceList.vue'
const props = defineProps<{ const props = defineProps<{
message: ChatMessage message: ChatMessage
}>() }>()
const emit = defineEmits<{
'open-source': [bookId: string, page: number]
}>()
const isUser = computed(() => props.message.role === 'USER') const isUser = computed(() => props.message.role === 'USER')
const renderedContent = computed(() => marked.parse(props.message.content) as string) const activeRef = ref<string | null>(null)
const sourceListEl = ref<InstanceType<typeof SourceList> | null>(null)
function escapeHtml(s: string): string {
return s.replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;').replace(/"/g, '&quot;')
}
const renderedWithBadges = computed(() => {
const html = marked.parse(props.message.content) as string
const figureMap = new Map<string, ChatSource>()
for (const src of (props.message.sources ?? [])) {
if (src.type === 'FIGURE' && src.refLabel) {
figureMap.set(src.refLabel, src)
}
}
return html.replace(/\[(S|F)\d+\]/g, (match) => {
const inner = match.slice(1, -1)
const badge = `<span class="citation-badge" data-ref="${inner}" title="Jump to source ${inner}">${match}</span>`
const fig = figureMap.get(inner)
if (fig?.imageUrl) {
const alt = escapeHtml(fig.caption || fig.label || 'Figure')
const captionText = [fig.label, fig.caption].filter(Boolean).map(escapeHtml).join(' — ')
const captionHtml = captionText
? `<figcaption class="inline-figure-caption">${captionText}</figcaption>`
: ''
return `${badge}<figure class="inline-figure"><img src="${fig.imageUrl}" alt="${alt}" class="inline-figure-img" loading="lazy" onerror="this.parentElement.style.display='none'" />${captionHtml}</figure>`
}
return badge
})
})
function onContentClick(e: MouseEvent) {
const target = e.target as HTMLElement
if (!target.classList.contains('citation-badge')) return
const label = target.getAttribute('data-ref')
if (!label) return
activeRef.value = activeRef.value === label ? null : label
const sourceEl = sourceListEl.value?.$el?.querySelector(`[data-ref-label="${label}"]`) as HTMLElement | null
sourceEl?.scrollIntoView({ behavior: 'smooth', block: 'start' })
const source = (props.message.sources ?? []).find((s: ChatSource) => s.refLabel === label)
if (source?.bookId && source.page) {
emit('open-source', source.bookId, source.page)
}
}
function formatTime(iso: string): string { function formatTime(iso: string): string {
return new Date(iso).toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' }) return new Date(iso).toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' })
@@ -182,6 +230,71 @@ function formatTime(iso: string): string {
gap: 0.25rem; gap: 0.25rem;
} }
.source-item--figure {
gap: 0.4rem;
}
.source-chip {
display: inline-flex;
align-items: center;
gap: 0.25rem;
border-radius: 4px;
padding: 0.2rem 0.5rem;
font-size: 0.78rem;
}
.source-chip--text {
background: #ebf8ff;
border: 1px solid #bee3f8;
}
.source-chip--clickable {
cursor: pointer;
transition: background 0.15s, border-color 0.15s;
}
.source-chip--clickable:hover {
background: #bee3f8;
border-color: #90cdf4;
}
.source-open-hint {
font-size: 0.75rem;
color: #3182ce;
margin-left: 0.1rem;
}
.source-chip--figure {
background: #f0fff4;
border: 1px solid #9ae6b4;
}
.source-icon {
font-size: 0.8rem;
}
.source-book-title {
color: #2b6cb0;
font-weight: 500;
}
.source-figure-label {
color: #276749;
font-weight: 600;
}
.source-figure-type {
color: #718096;
font-size: 0.72rem;
background: #e2e8f0;
border-radius: 3px;
padding: 0 0.3rem;
}
.source-page {
color: #718096;
}
.source-chunk { .source-chunk {
font-size: 0.78rem; font-size: 0.78rem;
color: #4a5568; color: #4a5568;
@@ -194,28 +307,89 @@ function formatTime(iso: string): string {
line-height: 1.5; line-height: 1.5;
} }
.source-chip { .source-caption {
display: inline-flex;
align-items: center;
gap: 0.25rem;
background: #ebf8ff;
border: 1px solid #bee3f8;
border-radius: 4px;
padding: 0.2rem 0.5rem;
font-size: 0.78rem; font-size: 0.78rem;
color: #4a5568;
font-style: italic;
} }
.source-book-icon { .source-figure-image {
font-size: 0.8rem; max-width: 100%;
} }
.source-book-title { .figure-img {
max-width: 100%;
max-height: 300px;
border-radius: 6px;
border: 1px solid #e2e8f0;
object-fit: contain;
}
.figure-missing {
font-size: 0.78rem;
color: #a0aec0;
font-style: italic;
}
.message-content--markdown :deep(.citation-badge) {
display: inline-block;
background: #ebf8ff;
border: 1px solid #90cdf4;
border-radius: 3px;
padding: 0 0.3em;
font-size: 0.78em;
font-weight: 600;
color: #2b6cb0; color: #2b6cb0;
font-weight: 500; cursor: pointer;
user-select: none;
transition: background 0.15s;
} }
.source-page { .message-content--markdown :deep(.citation-badge:hover) {
background: #bee3f8;
}
.source-item--active {
outline: 2px solid #4299e1;
border-radius: 6px;
}
.source-ref-label {
font-size: 0.72rem;
font-weight: 700;
background: #bee3f8;
color: #2b6cb0;
border-radius: 3px;
padding: 0 0.3rem;
}
.source-ref-label--figure {
background: #9ae6b4;
color: #276749;
}
.message-content--markdown :deep(.inline-figure) {
display: block;
margin: 0.75rem 0;
text-align: center;
}
.message-content--markdown :deep(.inline-figure-img) {
max-width: 100%;
max-height: 400px;
border-radius: 6px;
border: 1px solid #e2e8f0;
object-fit: contain;
display: block;
margin: 0 auto;
}
.message-content--markdown :deep(.inline-figure-caption) {
font-size: 0.78rem;
color: #718096; color: #718096;
font-style: italic;
margin-top: 0.3rem;
text-align: center;
} }
.message-timestamp { .message-timestamp {
+298
View File
@@ -0,0 +1,298 @@
<template>
<div class="source-list">
<!-- TEXT sources -->
<div
v-for="(source, idx) in textSources"
:key="'text-' + idx"
class="source-item"
:class="{ 'source-item--active': activeRef === source.refLabel }"
:data-ref-label="source.refLabel"
>
<div class="source-chip-wrapper">
<div
class="source-chip source-chip--text"
:class="{ 'source-chip--clickable': source.bookId && source.page }"
@click="source.bookId && source.page ? emit('open-source', source.bookId, source.page) : undefined"
>
<span class="source-icon">📖</span>
<span v-if="source.refLabel" class="source-ref-label">{{ source.refLabel }}</span>
<span class="source-book-title">{{ source.bookTitle }}</span>
<span v-if="source.page" class="source-page">p.&nbsp;{{ source.page }}</span>
<span v-if="source.bookId && source.page" class="source-open-hint"></span>
</div>
<div v-if="source.chunkText" class="tooltip tooltip--text">
<p class="tooltip-chunk">{{ source.chunkText }}</p>
</div>
</div>
</div>
<!-- FIGURE sources -->
<div
v-for="(source, idx) in figureSources"
:key="'fig-' + idx"
class="source-item source-item--figure"
:class="{ 'source-item--active': activeRef === source.refLabel }"
:data-ref-label="source.refLabel"
>
<div class="source-chip-wrapper">
<div
class="source-chip source-chip--figure"
:class="{ 'source-chip--clickable': source.bookId && source.page }"
@click="source.bookId && source.page ? emit('open-source', source.bookId, source.page) : undefined"
>
<span class="source-icon">🖼</span>
<span v-if="source.refLabel" class="source-ref-label source-ref-label--figure">{{ source.refLabel }}</span>
<span class="source-figure-label">{{ source.label || 'Figure' }}</span>
<span v-if="source.page" class="source-page">p.&nbsp;{{ source.page }}</span>
<span v-if="source.figureType" class="source-figure-type">{{ formatFigureType(source.figureType) }}</span>
<span v-if="source.bookId && source.page" class="source-open-hint"></span>
</div>
<div v-if="source.imageUrl || source.caption" class="tooltip tooltip--figure">
<img
v-if="source.imageUrl"
:src="source.imageUrl"
:alt="source.caption || source.label || 'Figure'"
class="tooltip-figure-img"
loading="lazy"
@error="onImageError"
/>
<p v-if="source.caption" class="tooltip-caption">{{ source.caption }}</p>
</div>
</div>
</div>
</div>
</template>
<script setup lang="ts">
import { computed } from 'vue'
export interface SourceItem {
type?: 'TEXT' | 'FIGURE'
refLabel?: string
bookId?: string | null
bookTitle: string
page?: number | null
chunkText?: string
figureId?: string
label?: string
caption?: string
figureType?: string
imageUrl?: string
}
const props = defineProps<{
sources: SourceItem[]
activeRef?: string | null
}>()
const emit = defineEmits<{
'open-source': [bookId: string, page: number]
}>()
const textSources = computed(() =>
props.sources.filter(s => s.type === 'TEXT' || !s.type)
)
const figureSources = computed(() =>
props.sources.filter(s => s.type === 'FIGURE')
)
function formatFigureType(type: string): string {
const labels: Record<string, string> = {
ANATOMICAL_DIAGRAM: 'Anatomical Diagram',
SURGICAL_PHOTOGRAPH: 'Surgical Photo',
MRI_CT_SCAN: 'MRI / CT',
TABLE: 'Table',
CHART: 'Chart',
INTRAOPERATIVE_IMAGE: 'Intraoperative'
}
return labels[type] ?? type
}
function onImageError(e: Event) {
const img = e.target as HTMLImageElement
img.style.display = 'none'
const wrapper = img.parentElement
if (wrapper) {
const missing = document.createElement('span')
missing.className = 'figure-missing'
missing.textContent = 'Image unavailable'
wrapper.appendChild(missing)
}
}
</script>
<style scoped>
.source-list {
display: flex;
flex-direction: column;
gap: 0.5rem;
}
.source-item {
display: flex;
flex-direction: column;
gap: 0.25rem;
}
.source-item--active {
outline: 2px solid #4299e1;
border-radius: 6px;
}
/* Wrapper provides the positioning context for the tooltip */
.source-chip-wrapper {
position: relative;
display: inline-block;
}
/* ── Chip base ── */
.source-chip {
display: inline-flex;
align-items: center;
gap: 0.25rem;
border-radius: 4px;
padding: 0.2rem 0.5rem;
font-size: 0.78rem;
}
.source-chip--text {
background: #ebf8ff;
border: 1px solid #bee3f8;
}
.source-chip--figure {
background: #f0fff4;
border: 1px solid #9ae6b4;
}
.source-chip--clickable {
cursor: pointer;
transition: background 0.15s, border-color 0.15s;
}
.source-chip--clickable:hover {
background: #bee3f8;
border-color: #90cdf4;
}
.source-chip--figure.source-chip--clickable:hover {
background: #c6f6d5;
border-color: #68d391;
}
/* ── Tooltip ── */
.tooltip {
display: none;
position: absolute;
left: 0;
top: calc(100% + 6px);
z-index: 100;
background: #1a202c;
border-radius: 6px;
padding: 0.6rem 0.75rem;
box-shadow: 0 4px 16px rgba(0, 0, 0, 0.2);
/* Keep it from overflowing too far */
max-width: min(340px, 80vw);
pointer-events: none;
}
/* Show on chip hover */
.source-chip-wrapper:hover .tooltip {
display: block;
}
/* Small arrow pointing up */
.tooltip::before {
content: '';
position: absolute;
top: -5px;
left: 14px;
border-left: 5px solid transparent;
border-right: 5px solid transparent;
border-bottom: 5px solid #1a202c;
}
.tooltip--text .tooltip-chunk {
margin: 0;
font-size: 0.78rem;
color: #e2e8f0;
line-height: 1.5;
white-space: pre-wrap;
word-break: break-word;
}
.tooltip--figure {
max-width: min(300px, 80vw);
}
.tooltip-figure-img {
display: block;
max-width: 100%;
max-height: 220px;
border-radius: 4px;
object-fit: contain;
margin-bottom: 0.4rem;
}
.tooltip-caption {
margin: 0;
font-size: 0.75rem;
color: #cbd5e0;
font-style: italic;
line-height: 1.4;
}
/* ── Chip internals ── */
.source-icon {
font-size: 0.8rem;
}
.source-ref-label {
font-size: 0.72rem;
font-weight: 700;
background: #bee3f8;
color: #2b6cb0;
border-radius: 3px;
padding: 0 0.3rem;
}
.source-ref-label--figure {
background: #9ae6b4;
color: #276749;
}
.source-book-title {
color: #2b6cb0;
font-weight: 500;
}
.source-figure-label {
color: #276749;
font-weight: 600;
}
.source-figure-type {
color: #718096;
font-size: 0.72rem;
background: #e2e8f0;
border-radius: 3px;
padding: 0 0.3rem;
}
.source-page {
color: #718096;
}
.source-open-hint {
font-size: 0.75rem;
color: #3182ce;
margin-left: 0.1rem;
}
.figure-missing {
font-size: 0.78rem;
color: #a0aec0;
font-style: italic;
}
</style>
+2
View File
@@ -3,6 +3,8 @@
interface ImportMetaEnv { interface ImportMetaEnv {
readonly VITE_API_URL: string readonly VITE_API_URL: string
readonly VITE_APP_PASSWORD: string readonly VITE_APP_PASSWORD: string
readonly VITE_UPLOAD_ENABLED: string
readonly VITE_DELETE_ENABLED: string
} }
interface ImportMeta { interface ImportMeta {
+10
View File
@@ -0,0 +1,10 @@
/**
* Read a VITE_ env variable.
* At runtime in Docker, values come from window.__env__ (injected by docker-entrypoint.sh).
* At build time (dev / CI), values come from import.meta.env.
*/
export function env(key: string): string | undefined {
const runtime = (window as Record<string, any>).__env__?.[key]
if (runtime) return runtime
return (import.meta as any).env?.[key]
}
+16 -1
View File
@@ -4,6 +4,21 @@ import App from './App.vue'
import router from './router' import router from './router'
const app = createApp(App) const app = createApp(App)
app.use(createPinia()) const pinia = createPinia()
app.use(pinia)
app.use(router) app.use(router)
// Verify any session restored from sessionStorage is still valid.
// If the backend rejects the credentials (e.g. password changed), clear them
// before the router guard fires so the user lands on /login cleanly.
import { useAuthStore } from '@/stores/authStore'
import { api } from '@/services/api'
const auth = useAuthStore()
if (auth.isAuthenticated) {
api.get('/auth/check').catch(() => {
auth.clearCredentials()
})
}
app.mount('#app') app.mount('#app')
+20
View File
@@ -1,11 +1,19 @@
import { createRouter, createWebHistory } from 'vue-router' import { createRouter, createWebHistory } from 'vue-router'
import { useAuthStore } from '@/stores/authStore'
import LoginView from '@/views/LoginView.vue'
import UploadView from '@/views/UploadView.vue' import UploadView from '@/views/UploadView.vue'
import TopicsView from '@/views/TopicsView.vue' import TopicsView from '@/views/TopicsView.vue'
import ChatView from '@/views/ChatView.vue' import ChatView from '@/views/ChatView.vue'
import BookReaderView from '@/views/BookReaderView.vue'
const router = createRouter({ const router = createRouter({
history: createWebHistory(import.meta.env.BASE_URL), history: createWebHistory(import.meta.env.BASE_URL),
routes: [ routes: [
{
path: '/login',
name: 'login',
component: LoginView
},
{ {
path: '/', path: '/',
name: 'upload', name: 'upload',
@@ -20,8 +28,20 @@ const router = createRouter({
path: '/chat', path: '/chat',
name: 'chat', name: 'chat',
component: ChatView component: ChatView
},
{
path: '/books/:id/read',
name: 'book-reader',
component: BookReaderView
} }
] ]
}) })
router.beforeEach((to) => {
const auth = useAuthStore()
if (to.name !== 'login' && !auth.isAuthenticated) {
return { name: 'login' }
}
})
export default router export default router
+16 -6
View File
@@ -1,20 +1,30 @@
import axios from 'axios' import axios from 'axios'
import { useAuthStore } from '@/stores/authStore'
import { env } from '@/env'
export const api = axios.create({ export const api = axios.create({
baseURL: import.meta.env.VITE_API_URL ?? '/api/v1', baseURL: env('VITE_API_URL') ?? '/api/v1',
auth: {
username: 'neurosurgeon',
password: import.meta.env.VITE_APP_PASSWORD ?? 'changeme'
},
headers: { headers: {
'Content-Type': 'application/json' 'Content-Type': 'application/json'
} }
}) })
// Response interceptor for error normalisation api.interceptors.request.use((config) => {
const auth = useAuthStore()
if (auth.username && auth.password) {
config.auth = { username: auth.username, password: auth.password }
}
return config
})
api.interceptors.response.use( api.interceptors.response.use(
(response) => response, (response) => response,
(error) => { (error) => {
if (error.response?.status === 401) {
useAuthStore().clearCredentials()
window.location.href = '/login'
return Promise.reject(new Error('Session expired. Please sign in again.'))
}
const message = const message =
error.response?.data?.error ?? error.response?.data?.error ??
error.message ?? error.message ??
+28
View File
@@ -0,0 +1,28 @@
import { defineStore } from 'pinia'
import { ref, computed } from 'vue'
const SESSION_KEY = 'auth'
export const useAuthStore = defineStore('auth', () => {
const stored = sessionStorage.getItem(SESSION_KEY)
const parsed = stored ? (JSON.parse(stored) as { username: string; password: string }) : null
const username = ref<string | null>(parsed?.username ?? null)
const password = ref<string | null>(parsed?.password ?? null)
const isAuthenticated = computed(() => !!username.value && !!password.value)
function setCredentials(u: string, p: string) {
username.value = u
password.value = p
sessionStorage.setItem(SESSION_KEY, JSON.stringify({ username: u, password: p }))
}
function clearCredentials() {
username.value = null
password.value = null
sessionStorage.removeItem(SESSION_KEY)
}
return { username, password, isAuthenticated, setCredentials, clearCredentials }
})
+38 -1
View File
@@ -77,5 +77,42 @@ export const useBookStore = defineStore('books', () => {
} }
} }
return { books, loading, uploading, error, fetchBooks, uploadBook, refreshBook, deleteBook } async function startEnrichment(id: string): Promise<EnrichmentProgress | null> {
try {
const response = await api.post<EnrichmentProgress>(`/admin/books/${id}/enrich`)
return response.data
} catch (err: any) {
error.value = err.message
return null
}
}
async function fetchEnrichmentStatus(id: string): Promise<EnrichmentProgress | null> {
try {
const response = await api.get<EnrichmentProgress>(`/admin/books/${id}/enrich`)
return response.data
} catch {
return null
}
}
return {
books,
loading,
uploading,
error,
fetchBooks,
uploadBook,
refreshBook,
deleteBook,
startEnrichment,
fetchEnrichmentStatus
}
}) })
export interface EnrichmentProgress {
status: 'IDLE' | 'RUNNING' | 'COMPLETED'
chunksTotal: number
chunksEnriched: number
errorMessage: string | null
}
+17 -1
View File
@@ -2,11 +2,27 @@ import { defineStore } from 'pinia'
import { ref } from 'vue' import { ref } from 'vue'
import { api } from '@/services/api' import { api } from '@/services/api'
export interface ChatSource {
type: 'TEXT' | 'FIGURE'
bookId?: string
bookTitle: string
page: number | null
refLabel?: string
// TEXT-specific
chunkText?: string
// FIGURE-specific
figureId?: string
label?: string
caption?: string
figureType?: string
imageUrl?: string
}
export interface ChatMessage { export interface ChatMessage {
id: string id: string
role: 'USER' | 'ASSISTANT' role: 'USER' | 'ASSISTANT'
content: string content: string
sources: Array<{ bookTitle: string; page: number | null; chunkText?: string }> sources: ChatSource[]
createdAt: string createdAt: string
} }
+145 -3
View File
@@ -10,11 +10,22 @@ export interface Topic {
} }
export interface SourceReference { export interface SourceReference {
type?: 'TEXT' | 'FIGURE'
refLabel?: string
bookId: string | null
bookTitle: string bookTitle: string
page: number | null page: number | null
chunkText?: string
figureId?: string
label?: string
caption?: string
figureType?: string
imageUrl?: string
} }
export interface TopicSummary { export interface TopicSummary {
id: string
summaryNumber: number
topicId: string topicId: string
topicName: string topicName: string
summary: string summary: string
@@ -22,14 +33,50 @@ export interface TopicSummary {
generatedAt: string generatedAt: string
} }
export interface SavedSummaryItem {
id: string
summaryNumber: number
generatedAt: string
}
export interface FacetSection {
facetKey: string
title: string
markdown: string
refLabels: string[]
}
export interface ConceptReport {
id: string
reportNumber: number
topicId: string
topicName: string
facets: FacetSection[]
sources: SourceReference[]
generatedAt: string
}
export interface SavedConceptReportItem {
id: string
reportNumber: number
generatedAt: string
}
export const useTopicStore = defineStore('topics', () => { export const useTopicStore = defineStore('topics', () => {
const topics = ref<Topic[]>([]) const topics = ref<Topic[]>([])
const activeSummary = ref<TopicSummary | null>(null) const activeSummary = ref<TopicSummary | null>(null)
const activeSummaryTopicId = ref<string | null>(null) const activeSummaryTopicId = ref<string | null>(null)
const summaryList = ref<SavedSummaryItem[]>([])
const loading = ref(false) const loading = ref(false)
const summaryLoading = ref(false) const summaryLoading = ref(false)
const summaryListLoading = ref(false)
const error = ref<string | null>(null) const error = ref<string | null>(null)
const activeConceptReport = ref<ConceptReport | null>(null)
const conceptReportList = ref<SavedConceptReportItem[]>([])
const conceptReportLoading = ref(false)
const conceptReportListLoading = ref(false)
async function fetchTopics() { async function fetchTopics() {
loading.value = true loading.value = true
error.value = null error.value = null
@@ -43,13 +90,47 @@ export const useTopicStore = defineStore('topics', () => {
} }
} }
async function generateSummary(topicId: string): Promise<TopicSummary | null> { async function fetchSummaries(topicId: string) {
summaryListLoading.value = true
summaryList.value = []
error.value = null
try {
const response = await api.get<SavedSummaryItem[]>(`/topics/${topicId}/summaries`)
summaryList.value = response.data
} catch (err: any) {
error.value = err.message
} finally {
summaryListLoading.value = false
}
}
async function fetchSummaryDetail(topicId: string, summaryId: string): Promise<TopicSummary | null> {
summaryLoading.value = true
activeSummary.value = null
error.value = null
try {
const response = await api.get<TopicSummary>(`/topics/${topicId}/summaries/${summaryId}`)
activeSummary.value = response.data
return response.data
} catch (err: any) {
error.value = err.message
return null
} finally {
summaryLoading.value = false
}
}
async function generateSummary(topicId: string, language: 'en' | 'th' = 'en'): Promise<TopicSummary | null> {
summaryLoading.value = true summaryLoading.value = true
activeSummaryTopicId.value = topicId activeSummaryTopicId.value = topicId
activeSummary.value = null activeSummary.value = null
error.value = null error.value = null
try { try {
const response = await api.post<TopicSummary>(`/topics/${topicId}/summary`) const response = await api.post<TopicSummary>(
`/topics/${topicId}/summary`,
null,
{ params: { language } }
)
activeSummary.value = response.data activeSummary.value = response.data
return response.data return response.data
} catch (err: any) { } catch (err: any) {
@@ -61,14 +142,75 @@ export const useTopicStore = defineStore('topics', () => {
} }
} }
async function fetchConceptReports(topicId: string) {
conceptReportListLoading.value = true
conceptReportList.value = []
error.value = null
try {
const response = await api.get<SavedConceptReportItem[]>(`/topics/${topicId}/concept-reports`)
conceptReportList.value = response.data
} catch (err: any) {
error.value = err.message
} finally {
conceptReportListLoading.value = false
}
}
async function fetchConceptReportDetail(topicId: string, reportId: string): Promise<ConceptReport | null> {
conceptReportLoading.value = true
activeConceptReport.value = null
error.value = null
try {
const response = await api.get<ConceptReport>(`/topics/${topicId}/concept-reports/${reportId}`)
activeConceptReport.value = response.data
return response.data
} catch (err: any) {
error.value = err.message
return null
} finally {
conceptReportLoading.value = false
}
}
async function generateConceptReport(topicId: string, language: 'en' | 'th' = 'en'): Promise<ConceptReport | null> {
conceptReportLoading.value = true
activeConceptReport.value = null
error.value = null
try {
const response = await api.post<ConceptReport>(
`/topics/${topicId}/concept-reports`,
null,
{ params: { language } }
)
activeConceptReport.value = response.data
return response.data
} catch (err: any) {
error.value = err.message
return null
} finally {
conceptReportLoading.value = false
}
}
return { return {
topics, topics,
activeSummary, activeSummary,
activeSummaryTopicId, activeSummaryTopicId,
summaryList,
loading, loading,
summaryLoading, summaryLoading,
summaryListLoading,
error, error,
activeConceptReport,
conceptReportList,
conceptReportLoading,
conceptReportListLoading,
fetchTopics, fetchTopics,
generateSummary fetchSummaries,
fetchSummaryDetail,
generateSummary,
fetchConceptReports,
fetchConceptReportDetail,
generateConceptReport
} }
}) })
+335
View File
@@ -0,0 +1,335 @@
<template>
<div class="reader-view">
<!-- Header -->
<div class="reader-header">
<router-link to="/" class="back-link"> Library</router-link>
<div class="reader-title">
<h1 class="book-title">{{ book?.title ?? 'Loading…' }}</h1>
</div>
<div class="page-nav">
<button class="nav-btn" :disabled="currentPage <= 1" @click="goTo(currentPage - 1)">&#8592;</button>
<form class="page-jump" @submit.prevent="onJump">
<input
v-model.number="jumpInput"
type="number"
:min="1"
:max="book?.pageCount ?? 1"
class="page-input"
/>
<span class="page-sep">/ {{ book?.pageCount ?? '…' }}</span>
</form>
<button class="nav-btn" :disabled="!book || currentPage >= book.pageCount!" @click="goTo(currentPage + 1)">&#8594;</button>
</div>
</div>
<!-- Content -->
<div class="reader-body">
<div v-if="loading" class="reader-loading">
<div class="spinner spinner-dark" style="width:28px;height:28px;margin:0 auto 0.75rem;"></div>
<p>Loading page {{ currentPage }}</p>
</div>
<div v-else-if="error" class="reader-error card">
<strong>Could not load page {{ currentPage }}</strong><br />
{{ error }}
</div>
<div v-else class="reader-content card">
<div class="markdown-body" v-html="renderedHtml"></div>
</div>
</div>
</div>
</template>
<script setup lang="ts">
import { ref, watch, onMounted } from 'vue'
import { useRoute } from 'vue-router'
import { api } from '@/services/api'
import { useBookStore } from '@/stores/bookStore'
import type { Book } from '@/stores/bookStore'
const route = useRoute()
const bookStore = useBookStore()
const bookId = route.params.id as string
const book = ref<Book | null>(null)
const currentPage = ref(1)
const jumpInput = ref(1)
const loading = ref(false)
const error = ref<string | null>(null)
const renderedHtml = ref('')
// Blob URLs created this session — revoked on next page load
let activeBlobUrls: string[] = []
onMounted(async () => {
book.value = bookStore.books.find(b => b.id === bookId) ?? null
if (!book.value) {
try {
const res = await api.get<Book>(`/books/${bookId}`)
book.value = res.data
} catch {
error.value = 'Book not found.'
return
}
}
await loadPage(1)
})
watch(currentPage, (page) => {
jumpInput.value = page
loadPage(page)
})
async function goTo(page: number) {
if (!book.value) return
const clamped = Math.max(1, Math.min(page, book.value.pageCount ?? 1))
if (clamped !== currentPage.value) {
currentPage.value = clamped
}
}
function onJump() {
goTo(jumpInput.value)
}
async function loadPage(page: number) {
loading.value = true
error.value = null
renderedHtml.value = ''
// Revoke previous blob URLs to free memory
activeBlobUrls.forEach(u => URL.revokeObjectURL(u))
activeBlobUrls = []
try {
const res = await api.get<string>(`/books/${bookId}/pages/${page}/html`, {
headers: { Accept: 'text/html' },
responseType: 'text'
})
let html = await resolveImages(res.data)
renderedHtml.value = html
} catch (e: any) {
error.value = e.message ?? 'Failed to load page.'
} finally {
loading.value = false
}
}
/**
* Finds <img src="/api/v1/figures/..."> in the HTML, fetches each image
* through the authenticated axios instance, and replaces the src with a
* temporary blob URL so the browser can render it without re-authenticating.
*/
async function resolveImages(html: string): Promise<string> {
const srcPattern = /src="(\/api\/v1\/figures\/[^"]+)"/g
const matches = [...html.matchAll(srcPattern)]
if (matches.length === 0) return html
const unique = [...new Set(matches.map(m => m[1]))]
const blobMap: Record<string, string> = {}
await Promise.all(
unique.map(async (src) => {
try {
const res = await api.get(src.replace(/^\/api\/v1/, ''), { responseType: 'blob' })
const blobUrl = URL.createObjectURL(res.data)
activeBlobUrls.push(blobUrl)
blobMap[src] = blobUrl
} catch {
// leave original src — browser will attempt (and likely fail silently)
}
})
)
return html.replace(/src="(\/api\/v1\/figures\/[^"]+)"/g, (_, src) =>
blobMap[src] ? `src="${blobMap[src]}"` : `src="${src}"`
)
}
</script>
<style scoped>
.reader-view {
display: flex;
flex-direction: column;
gap: 1rem;
max-width: 860px;
margin: 0 auto;
flex: 1;
min-height: 0;
}
.reader-header {
display: flex;
align-items: center;
gap: 1rem;
flex-wrap: wrap;
}
.back-link {
color: #3182ce;
text-decoration: none;
font-size: 0.9rem;
white-space: nowrap;
}
.back-link:hover { text-decoration: underline; }
.reader-title {
flex: 1;
min-width: 0;
}
.book-title {
font-size: 1.1rem;
font-weight: 600;
color: #1a365d;
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
}
.page-nav {
display: flex;
align-items: center;
gap: 0.5rem;
}
.nav-btn {
width: 2rem;
height: 2rem;
border: 1px solid #cbd5e0;
border-radius: 6px;
background: #fff;
cursor: pointer;
font-size: 1rem;
display: flex;
align-items: center;
justify-content: center;
transition: background 0.15s;
}
.nav-btn:hover:not(:disabled) { background: #ebf8ff; border-color: #3182ce; }
.nav-btn:disabled { opacity: 0.4; cursor: not-allowed; }
.page-jump {
display: flex;
align-items: center;
gap: 0.35rem;
}
.page-input {
width: 3.5rem;
text-align: center;
border: 1px solid #cbd5e0;
border-radius: 6px;
padding: 0.25rem 0.4rem;
font-size: 0.9rem;
color: #2d3748;
}
.page-input:focus { outline: none; border-color: #3182ce; }
.page-sep {
font-size: 0.85rem;
color: #718096;
white-space: nowrap;
}
.reader-body {
flex: 1;
min-height: 0;
display: flex;
flex-direction: column;
}
.reader-loading {
text-align: center;
padding: 3rem;
color: #718096;
}
.reader-error {
padding: 1.25rem;
background: #fff5f5;
border: 1px solid #fed7d7;
color: #742a2a;
border-radius: 8px;
}
.reader-content {
flex: 1;
min-height: 0;
overflow-y: auto;
padding: 2rem;
}
/* Markdown rendering */
.markdown-body {
font-size: 0.95rem;
line-height: 1.75;
color: #2d3748;
}
.markdown-body :deep(h1),
.markdown-body :deep(h2),
.markdown-body :deep(h3) {
color: #1a365d;
font-weight: 600;
margin: 1.5rem 0 0.75rem;
}
.markdown-body :deep(h2) { font-size: 1.15rem; border-bottom: 1px solid #e2e8f0; padding-bottom: 0.4rem; }
.markdown-body :deep(h3) { font-size: 1rem; }
.markdown-body :deep(p) { margin: 0.75rem 0; }
.markdown-body :deep(img) {
max-width: 100%;
border-radius: 6px;
display: block;
margin: 1rem auto;
box-shadow: 0 1px 4px rgba(0,0,0,0.12);
}
.markdown-body :deep(ul),
.markdown-body :deep(ol) {
padding-left: 1.5rem;
margin: 0.75rem 0;
}
.markdown-body :deep(code) {
background: #f7fafc;
border: 1px solid #e2e8f0;
border-radius: 3px;
padding: 0.1em 0.35em;
font-size: 0.88em;
}
.markdown-body :deep(blockquote) {
border-left: 3px solid #3182ce;
padding-left: 1rem;
color: #4a5568;
margin: 0.75rem 0;
}
.markdown-body :deep(table) {
width: 100%;
border-collapse: collapse;
font-size: 0.9em;
margin: 1rem 0;
}
.markdown-body :deep(th),
.markdown-body :deep(td) {
border: 1px solid #e2e8f0;
padding: 0.4rem 0.75rem;
text-align: left;
}
.markdown-body :deep(th) { background: #f7fafc; font-weight: 600; }
@media (max-width: 768px) {
.reader-view {
max-width: 100%;
}
.reader-content {
padding: 1rem;
}
}
</style>
+125 -123
View File
@@ -3,27 +3,10 @@
<h1 class="page-title">Knowledge Chat</h1> <h1 class="page-title">Knowledge Chat</h1>
<p class="page-subtitle">Ask questions grounded in your uploaded medical textbooks.</p> <p class="page-subtitle">Ask questions grounded in your uploaded medical textbooks.</p>
<!-- Step 1: Topic Selection --> <!-- Session selection -->
<div v-if="!chatStore.session && !selectedTopic" class="topic-selection"> <div v-if="!chatStore.session" class="session-setup card">
<h2 class="section-title">Select a Topic</h2>
<div class="topic-grid">
<button
v-for="topic in topicStore.topics"
:key="topic.id"
:class="['topic-tile', { 'topic-tile-freeform': topic.id === 'free-form' }]"
@click="handleTopicSelect(topic)"
>
<span class="topic-tile-name">{{ topic.name }}</span>
<span v-if="topic.id === 'free-form'" class="topic-tile-hint">Any neurosurgery question</span>
</button>
</div>
</div>
<!-- Step 2: Topic selected previous sessions + new chat -->
<div v-else-if="!chatStore.session && selectedTopic" class="session-setup card">
<div class="setup-header"> <div class="setup-header">
<button class="btn-back" @click="handleBack"> Topics</button> <h2 class="section-title">Free-form Chat</h2>
<h2 class="section-title">{{ selectedTopic.name }}</h2>
</div> </div>
<div v-if="chatStore.error" class="error-banner">{{ chatStore.error }}</div> <div v-if="chatStore.error" class="error-banner">{{ chatStore.error }}</div>
@@ -71,56 +54,74 @@
</div> </div>
</div> </div>
<!-- Messages Area --> <!-- Chat + Reader split -->
<div class="messages-container" ref="messagesContainer"> <div class="chat-reader-split">
<div v-if="chatStore.loading && chatStore.messages.length === 0" class="empty-state"> <!-- Messages + Input -->
<div class="spinner spinner-dark" style="width:32px;height:32px;margin:0 auto 1rem;"></div> <div class="chat-column">
<p class="empty-state-text">Loading messages...</p> <!-- Messages Area -->
</div> <div class="messages-container" ref="messagesContainer">
<div v-if="chatStore.loading && chatStore.messages.length === 0" class="empty-state">
<div class="spinner spinner-dark" style="width:32px;height:32px;margin:0 auto 1rem;"></div>
<p class="empty-state-text">Loading messages...</p>
</div>
<div v-else-if="chatStore.messages.length === 0" class="empty-state"> <div v-else-if="chatStore.messages.length === 0" class="empty-state">
<div class="empty-state-icon">💬</div> <div class="empty-state-icon">💬</div>
<p class="empty-state-text">No messages yet</p> <p class="empty-state-text">No messages yet</p>
<p class="empty-state-hint">Ask a question about the uploaded books below.</p> <p class="empty-state-hint">Ask a question about the uploaded books below.</p>
</div> </div>
<div v-else class="messages-list"> <div v-else class="messages-list">
<ChatMessage <ChatMessage
v-for="message in chatStore.messages" v-for="message in chatStore.messages"
:key="message.id" :key="message.id"
:message="message" :message="message"
/> @open-source="handleOpenSource"
<div v-if="chatStore.sending" class="typing-indicator"> />
<div class="typing-bubble"> <div v-if="chatStore.sending" class="typing-indicator">
<span></span><span></span><span></span> <div class="typing-bubble">
<span></span><span></span><span></span>
</div>
</div>
</div> </div>
</div> </div>
</div>
</div>
<!-- Input Area --> <!-- Input Area -->
<div class="input-area card"> <div class="input-area card">
<div v-if="chatStore.error" class="error-banner">{{ chatStore.error }}</div> <div v-if="chatStore.error" class="error-banner">{{ chatStore.error }}</div>
<div class="input-row"> <div class="input-row">
<textarea <textarea
v-model="inputText" v-model="inputText"
class="message-input" class="message-input"
placeholder="Ask a question about your uploaded books..." placeholder="Ask a question about your uploaded books..."
rows="2" rows="2"
:disabled="chatStore.sending" :disabled="chatStore.sending"
@keydown.enter.exact.prevent="handleSend" @keydown.enter.exact.prevent="handleSend"
@keydown.enter.shift.exact="inputText += '\n'" @keydown.enter.shift.exact="inputText += '\n'"
></textarea> ></textarea>
<button <button
class="btn btn-primary send-btn" class="btn btn-primary send-btn"
:disabled="!inputText.trim() || chatStore.sending" :disabled="!inputText.trim() || chatStore.sending"
@click="handleSend" @click="handleSend"
> >
<span v-if="chatStore.sending" class="spinner"></span> <span v-if="chatStore.sending" class="spinner"></span>
<span v-else>Send</span> <span v-else>Send</span>
</button> </button>
</div>
<p class="input-hint">Press Enter to send, Shift+Enter for new line.</p>
</div>
</div> </div>
<p class="input-hint">Press Enter to send, Shift+Enter for new line.</p>
<!-- Inline book reader panel -->
<BookPagePanel
v-if="readerPanel"
:book-id="readerPanel.bookId"
:page="readerPanel.page"
:book-title="readerPanel.bookTitle"
class="reader-panel"
@close="readerPanel = null"
@navigate="(p) => readerPanel && (readerPanel.page = p)"
/>
</div> </div>
</div> </div>
</div> </div>
@@ -130,8 +131,10 @@
import { ref, nextTick, onMounted, watch, inject } from 'vue' import { ref, nextTick, onMounted, watch, inject } from 'vue'
import { useChatStore } from '@/stores/chatStore' import { useChatStore } from '@/stores/chatStore'
import { useTopicStore } from '@/stores/topicStore' import { useTopicStore } from '@/stores/topicStore'
import { useBookStore } from '@/stores/bookStore'
import type { ChatSession } from '@/stores/chatStore' import type { ChatSession } from '@/stores/chatStore'
import ChatMessage from '@/components/ChatMessage.vue' import ChatMessage from '@/components/ChatMessage.vue'
import BookPagePanel from '@/components/BookPagePanel.vue'
interface Topic { interface Topic {
id: string id: string
@@ -142,6 +145,7 @@ interface Topic {
const chatStore = useChatStore() const chatStore = useChatStore()
const topicStore = useTopicStore() const topicStore = useTopicStore()
const bookStore = useBookStore()
const showToast = inject<(msg: string, type?: 'error' | 'success') => void>('showToast') const showToast = inject<(msg: string, type?: 'error' | 'success') => void>('showToast')
const selectedTopic = ref<Topic | null>(null) const selectedTopic = ref<Topic | null>(null)
@@ -150,10 +154,22 @@ const loadingTopicSessions = ref(false)
const inputText = ref('') const inputText = ref('')
const messagesContainer = ref<HTMLElement | null>(null) const messagesContainer = ref<HTMLElement | null>(null)
interface ReaderPanel { bookId: string; page: number; bookTitle?: string }
const readerPanel = ref<ReaderPanel | null>(null)
function handleOpenSource(bookId: string, page: number) {
const book = bookStore.books.find(b => b.id === bookId)
readerPanel.value = { bookId, page, bookTitle: book?.title }
}
onMounted(async () => { onMounted(async () => {
if (topicStore.topics.length === 0) { if (topicStore.topics.length === 0) {
await topicStore.fetchTopics() await topicStore.fetchTopics()
} }
const freeForm = topicStore.topics.find((t) => t.id === 'free-form')
if (freeForm) {
await handleTopicSelect(freeForm)
}
}) })
watch( watch(
@@ -189,11 +205,6 @@ async function handleTopicSelect(topic: Topic) {
loadingTopicSessions.value = false loadingTopicSessions.value = false
} }
function handleBack() {
selectedTopic.value = null
topicSessions.value = []
}
async function handleNewChat() { async function handleNewChat() {
const ok = await chatStore.createSession(selectedTopic.value!.id) const ok = await chatStore.createSession(selectedTopic.value!.id)
if (!ok) { if (!ok) {
@@ -207,9 +218,7 @@ async function handleResumeSession(session: ChatSession) {
} }
function handleLeaveSession() { function handleLeaveSession() {
// Leave without deleting — session stays in DB and will appear in "Previous Chats"
chatStore.leaveSession() chatStore.leaveSession()
// Refresh the sessions list for the current topic
if (selectedTopic.value) { if (selectedTopic.value) {
loadingTopicSessions.value = true loadingTopicSessions.value = true
chatStore.fetchSessionsByTopic(selectedTopic.value.id).then((sessions) => { chatStore.fetchSessionsByTopic(selectedTopic.value.id).then((sessions) => {
@@ -231,12 +240,6 @@ async function handleSend() {
</script> </script>
<style scoped> <style scoped>
.topic-selection {
display: flex;
flex-direction: column;
gap: 1.25rem;
}
.section-title { .section-title {
font-size: 1.1rem; font-size: 1.1rem;
font-weight: 600; font-weight: 600;
@@ -244,52 +247,6 @@ async function handleSend() {
margin: 0; margin: 0;
} }
.topic-grid {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(220px, 1fr));
gap: 0.75rem;
}
.topic-tile {
display: flex;
flex-direction: column;
align-items: flex-start;
gap: 0.25rem;
padding: 1rem 1.1rem;
background: white;
border: 1px solid #e2e8f0;
border-radius: 8px;
cursor: pointer;
text-align: left;
transition: border-color 0.15s, box-shadow 0.15s;
}
.topic-tile:hover {
border-color: #3182ce;
box-shadow: 0 2px 8px rgba(49, 130, 206, 0.15);
}
.topic-tile-freeform {
border-style: dashed;
border-color: #a0aec0;
}
.topic-tile-freeform:hover {
border-color: #718096;
box-shadow: 0 2px 8px rgba(0, 0, 0, 0.08);
}
.topic-tile-name {
font-size: 0.9rem;
font-weight: 600;
color: #2d3748;
}
.topic-tile-hint {
font-size: 0.78rem;
color: #a0aec0;
}
.session-setup { .session-setup {
max-width: 540px; max-width: 540px;
} }
@@ -381,6 +338,29 @@ async function handleSend() {
min-height: 500px; min-height: 500px;
} }
.chat-reader-split {
display: flex;
flex: 1;
min-height: 0;
gap: 0;
}
.chat-column {
display: flex;
flex-direction: column;
flex: 1;
min-width: 0;
gap: 1rem;
}
.reader-panel {
width: 420px;
flex-shrink: 0;
border-radius: 10px;
margin-left: 1rem;
box-shadow: -2px 0 8px rgba(0, 0, 0, 0.07);
}
.session-bar { .session-bar {
display: flex; display: flex;
align-items: center; align-items: center;
@@ -505,4 +485,26 @@ async function handleSend() {
font-size: 0.875rem; font-size: 0.875rem;
margin-bottom: 0.75rem; margin-bottom: 0.75rem;
} }
@media (max-width: 768px) {
.chat-layout {
height: auto;
min-height: unset;
}
.chat-reader-split {
flex-direction: column;
}
.chat-column {
min-height: 60vh;
}
.reader-panel {
width: 100%;
margin-left: 0;
margin-top: 1rem;
box-shadow: none;
}
}
</style> </style>

Some files were not shown because too many files have changed in this diff Show More