Compare commits
21 Commits
fc5b22fba1
...
master
| Author | SHA1 | Date | |
|---|---|---|---|
| 0c226483c0 | |||
| ff97c24a55 | |||
| c7a77af2f4 | |||
| 5f03e1f41b | |||
| c98fe9ceaa | |||
| 767d1e2dbc | |||
| 820734c251 | |||
| 0711e40c66 | |||
| 0db31e91ab | |||
| d480d04145 | |||
| c2d034d1fe | |||
| 0908355704 | |||
| 8e227a9429 | |||
| d8bcdce879 | |||
| aee6a9dfba | |||
| 0cf318f0a7 | |||
| e5d53b4e80 | |||
| 5c641f4bcc | |||
| ea1276dc2e | |||
| b154e29f2d | |||
| 5acfdd33c1 |
@@ -0,0 +1,27 @@
|
||||
.git/
|
||||
.gitignore
|
||||
*.md
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# Java build artifacts
|
||||
target/
|
||||
*.class
|
||||
*.jar
|
||||
|
||||
# Node
|
||||
node_modules/
|
||||
dist/
|
||||
*.log
|
||||
|
||||
# Env files (never bake secrets into images)
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
|
||||
# Spec / docs
|
||||
specs/
|
||||
|
||||
# Editor
|
||||
.vscode/
|
||||
.idea/
|
||||
@@ -0,0 +1,13 @@
|
||||
# Copy this file to .env and fill in your values before running docker-compose.native.yml
|
||||
# .env is gitignored — never commit real credentials
|
||||
|
||||
# OpenAI
|
||||
OPENAI_API_KEY=sk-...
|
||||
|
||||
# AWS S3 (figure storage — leave blank if using local filesystem)
|
||||
AWS_ACCESS_KEY_ID=
|
||||
AWS_SECRET_ACCESS_KEY=
|
||||
AWS_REGION=eu-west-1
|
||||
|
||||
# S3 bucket name (if S3 storage enabled)
|
||||
APP_STORAGE_S3_BUCKET=ai-teacher-figures
|
||||
+12
@@ -1,3 +1,15 @@
|
||||
# Runtime uploads (extracted figures)
|
||||
uploads/
|
||||
|
||||
# Java build
|
||||
target/
|
||||
*.class
|
||||
*.jar
|
||||
|
||||
# Node
|
||||
node_modules/
|
||||
dist/
|
||||
|
||||
# OS
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
@@ -1,8 +1,23 @@
|
||||
# ai-teacher Development Guidelines
|
||||
|
||||
Auto-generated from all feature plans. Last updated: 2026-03-31
|
||||
Auto-generated from all feature plans. Last updated: 2026-04-10
|
||||
|
||||
## Active Technologies
|
||||
- Java 25 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API (embeddings + chat), PDFBox (via Spring AI PDF reader dependency) (002-image-aware-embedding)
|
||||
- PostgreSQL (JPA + Flyway), pgvector (Spring AI `VectorStore`), local file system (extracted images — `/uploads/figures/`) (002-image-aware-embedding)
|
||||
- Java 25 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API, PDFBox (rendering only), `com.google.cloud:google-cloud-documentai` (~2.40.x) (002-image-aware-embedding)
|
||||
- PostgreSQL (JPA + Flyway), pgvector (Spring AI VectorStore), S3 / local filesystem (figure images) (002-image-aware-embedding)
|
||||
- PostgreSQL (JPA + Flyway), pgvector (Spring AI `VectorStore`), S3-compatible (002-image-aware-embedding)
|
||||
- Java 21 (backend) / TypeScript + Node 20 (frontend) + Spring Boot 4.0.5, Spring Security (already included), Vue 3.4, Vue Router 4.3, Pinia 2.1, Axios 1.7 (003-basic-login)
|
||||
- No new storage — credentials held in browser `sessionStorage` (frontend only) (003-basic-login)
|
||||
- Java 21 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API (chat + embeddings), pgvector, Vue 3.4, Pinia 2.1 (004-rag-retrieval-quality)
|
||||
- PostgreSQL (sections, figures, messages — unchanged). No new tables needed. (004-rag-retrieval-quality)
|
||||
- Java 21 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API (chat + embeddings), Vue 3.4, Pinia 2.1, Axios 1.7 (004-rag-retrieval-quality)
|
||||
- PostgreSQL (JPA + Flyway), pgvector (`VectorStore`) (004-rag-retrieval-quality)
|
||||
- Java 25 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, `native-maven-plugin` 0.10.6, (005-native-image-deployment)
|
||||
- PostgreSQL 16 + pgvector (unchanged) (005-native-image-deployment)
|
||||
- TypeScript / Node 20 (frontend only) + Vue 3.4, Vue Router 4.3, Pinia 2.1 — no changes (006-mobile-responsive-ui)
|
||||
- N/A (frontend-only change) (006-mobile-responsive-ui)
|
||||
|
||||
- Java 21 (backend), TypeScript / Node 20 (frontend) (001-neuro-rag-learning)
|
||||
|
||||
@@ -22,8 +37,10 @@ npm test && npm run lint
|
||||
Java 21 (backend), TypeScript / Node 20 (frontend): Follow standard conventions
|
||||
|
||||
## Recent Changes
|
||||
- 006-mobile-responsive-ui: Added TypeScript / Node 20 (frontend only) + Vue 3.4, Vue Router 4.3, Pinia 2.1 — no changes
|
||||
- 005-native-image-deployment: Added Java 25 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, `native-maven-plugin` 0.10.6,
|
||||
- 004-rag-retrieval-quality: Added Java 21 (backend), TypeScript / Node 20 (frontend) + Spring Boot 4.0.5, Spring AI 2.0.0-M4, OpenAI API (chat + embeddings), Vue 3.4, Pinia 2.1, Axios 1.7
|
||||
|
||||
- 001-neuro-rag-learning: Added Java 21 (backend), TypeScript / Node 20 (frontend)
|
||||
|
||||
<!-- MANUAL ADDITIONS START -->
|
||||
<!-- MANUAL ADDITIONS END -->
|
||||
|
||||
@@ -0,0 +1,566 @@
|
||||
# Marker
|
||||
|
||||
Marker converts documents to markdown, JSON, chunks, and HTML quickly and accurately.
|
||||
|
||||
- Converts PDF, image, PPTX, DOCX, XLSX, HTML, EPUB files in all languages
|
||||
- Formats tables, forms, equations, inline math, links, references, and code blocks
|
||||
- Extracts and saves images
|
||||
- Removes headers/footers/other artifacts
|
||||
- Extensible with your own formatting and logic
|
||||
- Does structured extraction, given a JSON schema (beta)
|
||||
- Optionally boost accuracy with LLMs (and your own prompt)
|
||||
- Works on GPU, CPU, or MPS
|
||||
|
||||
For our managed API or on-prem document intelligence solution, check out [our platform here](https://datalab.to?utm_source=gh-marker).
|
||||
|
||||
## Performance
|
||||
|
||||
<img src="data/images/overall.png" width="800px"/>
|
||||
|
||||
Marker benchmarks favorably compared to cloud services like Llamaparse and Mathpix, as well as other open source tools.
|
||||
|
||||
The above results are running single PDF pages serially. Marker is significantly faster when running in batch mode, with a projected throughput of 25 pages/second on an H100.
|
||||
|
||||
See [below](#benchmarks) for detailed speed and accuracy benchmarks, and instructions on how to run your own benchmarks.
|
||||
|
||||
## Hybrid Mode
|
||||
|
||||
For the highest accuracy, pass the `--use_llm` flag to use an LLM alongside marker. This will do things like merge tables across pages, handle inline math, format tables properly, and extract values from forms. It can use any gemini or ollama model. By default, it uses `gemini-2.0-flash`. See [below](#llm-services) for details.
|
||||
|
||||
Here is a table benchmark comparing marker, gemini flash alone, and marker with use_llm:
|
||||
|
||||
<img src="data/images/table.png" width="400px"/>
|
||||
|
||||
As you can see, the use_llm mode offers higher accuracy than marker or gemini alone.
|
||||
|
||||
## Examples
|
||||
|
||||
| PDF | File type | Markdown | JSON |
|
||||
|-----|-----------|------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------|
|
||||
| [Think Python](https://greenteapress.com/thinkpython/thinkpython.pdf) | Textbook | [View](https://github.com/VikParuchuri/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md) | [View](https://github.com/VikParuchuri/marker/blob/master/data/examples/json/thinkpython.json) |
|
||||
| [Switch Transformers](https://arxiv.org/pdf/2101.03961.pdf) | arXiv paper | [View](https://github.com/VikParuchuri/marker/blob/master/data/examples/markdown/switch_transformers/switch_trans.md) | [View](https://github.com/VikParuchuri/marker/blob/master/data/examples/json/switch_trans.json) |
|
||||
| [Multi-column CNN](https://arxiv.org/pdf/1804.07821.pdf) | arXiv paper | [View](https://github.com/VikParuchuri/marker/blob/master/data/examples/markdown/multicolcnn/multicolcnn.md) | [View](https://github.com/VikParuchuri/marker/blob/master/data/examples/json/multicolcnn.json) |
|
||||
|
||||
# Commercial usage
|
||||
|
||||
Our model weights use a modified AI Pubs Open Rail-M license (free for research, personal use, and startups under $2M funding/revenue) and our code is GPL. For broader commercial licensing or to remove GPL requirements, visit our pricing page [here](https://www.datalab.to/pricing?utm_source=gh-marker).
|
||||
|
||||
# Hosted API & On-prem
|
||||
|
||||
There's a [hosted API](https://www.datalab.to?utm_source=gh-marker) and [painless on-prem solution](https://www.datalab.to/blog/self-serve-on-prem-licensing) for marker - it's free to sign up, and we'll throw in credits for you to test it out.
|
||||
|
||||
The API:
|
||||
- Supports PDF, image, PPT, PPTX, DOC, DOCX, XLS, XLSX, HTML, EPUB files
|
||||
- Is 1/4th the price of leading cloud-based competitors
|
||||
- Fast - ~15s for a 250 page PDF
|
||||
- Supports LLM mode
|
||||
- High uptime (99.99%)
|
||||
|
||||
# Community
|
||||
|
||||
[Discord](https://discord.gg//KuZwXNGnfH) is where we discuss future development.
|
||||
|
||||
# Installation
|
||||
|
||||
You'll need python 3.10+ and [PyTorch](https://pytorch.org/get-started/locally/).
|
||||
|
||||
Install with:
|
||||
|
||||
```shell
|
||||
pip install marker-pdf
|
||||
```
|
||||
|
||||
If you want to use marker on documents other than PDFs, you will need to install additional dependencies with:
|
||||
|
||||
```shell
|
||||
pip install marker-pdf[full]
|
||||
```
|
||||
|
||||
# Usage
|
||||
|
||||
First, some configuration:
|
||||
|
||||
- Your torch device will be automatically detected, but you can override this. For example, `TORCH_DEVICE=cuda`.
|
||||
- Some PDFs, even digital ones, have bad text in them. Set `--force_ocr` to force OCR on all lines, or the `strip_existing_ocr` to keep all digital text, and strip out any existing OCR text.
|
||||
- If you care about inline math, set `force_ocr` to convert inline math to LaTeX.
|
||||
|
||||
## Interactive App
|
||||
|
||||
I've included a streamlit app that lets you interactively try marker with some basic options. Run it with:
|
||||
|
||||
```shell
|
||||
pip install streamlit streamlit-ace
|
||||
marker_gui
|
||||
```
|
||||
|
||||
## Convert a single file
|
||||
|
||||
```shell
|
||||
marker_single /path/to/file.pdf
|
||||
```
|
||||
|
||||
You can pass in PDFs or images.
|
||||
|
||||
Options:
|
||||
- `--page_range TEXT`: Specify which pages to process. Accepts comma-separated page numbers and ranges. Example: `--page_range "0,5-10,20"` will process pages 0, 5 through 10, and page 20.
|
||||
- `--output_format [markdown|json|html|chunks]`: Specify the format for the output results.
|
||||
- `--output_dir PATH`: Directory where output files will be saved. Defaults to the value specified in settings.OUTPUT_DIR.
|
||||
- `--paginate_output`: Paginates the output, using `\n\n{PAGE_NUMBER}` followed by `-` * 48, then `\n\n`
|
||||
- `--use_llm`: Uses an LLM to improve accuracy. You will need to configure the LLM backend - see [below](#llm-services).
|
||||
- `--force_ocr`: Force OCR processing on the entire document, even for pages that might contain extractable text. This will also format inline math properly.
|
||||
- `--block_correction_prompt`: if LLM mode is active, an optional prompt that will be used to correct the output of marker. This is useful for custom formatting or logic that you want to apply to the output.
|
||||
- `--strip_existing_ocr`: Remove all existing OCR text in the document and re-OCR with surya.
|
||||
- `--redo_inline_math`: If you want the absolute highest quality inline math conversion, use this along with `--use_llm`.
|
||||
- `--disable_image_extraction`: Don't extract images from the PDF. If you also specify `--use_llm`, then images will be replaced with a description.
|
||||
- `--debug`: Enable debug mode for additional logging and diagnostic information.
|
||||
- `--processors TEXT`: Override the default processors by providing their full module paths, separated by commas. Example: `--processors "module1.processor1,module2.processor2"`
|
||||
- `--config_json PATH`: Path to a JSON configuration file containing additional settings.
|
||||
- `config --help`: List all available builders, processors, and converters, and their associated configuration. These values can be used to build a JSON configuration file for additional tweaking of marker defaults.
|
||||
- `--converter_cls`: One of `marker.converters.pdf.PdfConverter` (default) or `marker.converters.table.TableConverter`. The `PdfConverter` will convert the whole PDF, the `TableConverter` will only extract and convert tables.
|
||||
- `--llm_service`: Which llm service to use if `--use_llm` is passed. This defaults to `marker.services.gemini.GoogleGeminiService`.
|
||||
- `--help`: see all of the flags that can be passed into marker. (it supports many more options then are listed above)
|
||||
|
||||
The list of supported languages for surya OCR is [here](https://github.com/VikParuchuri/surya/blob/master/surya/recognition/languages.py). If you don't need OCR, marker can work with any language.
|
||||
|
||||
## Convert multiple files
|
||||
|
||||
```shell
|
||||
marker /path/to/input/folder
|
||||
```
|
||||
|
||||
- `marker` supports all the same options from `marker_single` above.
|
||||
- `--workers` is the number of conversion workers to run simultaneously. This is automatically set by default, but you can increase it to increase throughput, at the cost of more CPU/GPU usage. Marker will use 5GB of VRAM per worker at the peak, and 3.5GB average.
|
||||
|
||||
## Convert multiple files on multiple GPUs
|
||||
|
||||
```shell
|
||||
NUM_DEVICES=4 NUM_WORKERS=15 marker_chunk_convert ../pdf_in ../md_out
|
||||
```
|
||||
|
||||
- `NUM_DEVICES` is the number of GPUs to use. Should be `2` or greater.
|
||||
- `NUM_WORKERS` is the number of parallel processes to run on each GPU.
|
||||
|
||||
## Use from python
|
||||
|
||||
See the `PdfConverter` class at `marker/converters/pdf.py` function for additional arguments that can be passed.
|
||||
|
||||
```python
|
||||
from marker.converters.pdf import PdfConverter
|
||||
from marker.models import create_model_dict
|
||||
from marker.output import text_from_rendered
|
||||
|
||||
converter = PdfConverter(
|
||||
artifact_dict=create_model_dict(),
|
||||
)
|
||||
rendered = converter("FILEPATH")
|
||||
text, _, images = text_from_rendered(rendered)
|
||||
```
|
||||
|
||||
`rendered` will be a pydantic basemodel with different properties depending on the output type requested. With markdown output (default), you'll have the properties `markdown`, `metadata`, and `images`. For json output, you'll have `children`, `block_type`, and `metadata`.
|
||||
|
||||
### Custom configuration
|
||||
|
||||
You can pass configuration using the `ConfigParser`. To see all available options, do `marker_single --help`.
|
||||
|
||||
```python
|
||||
from marker.converters.pdf import PdfConverter
|
||||
from marker.models import create_model_dict
|
||||
from marker.config.parser import ConfigParser
|
||||
|
||||
config = {
|
||||
"output_format": "json",
|
||||
"ADDITIONAL_KEY": "VALUE"
|
||||
}
|
||||
config_parser = ConfigParser(config)
|
||||
|
||||
converter = PdfConverter(
|
||||
config=config_parser.generate_config_dict(),
|
||||
artifact_dict=create_model_dict(),
|
||||
processor_list=config_parser.get_processors(),
|
||||
renderer=config_parser.get_renderer(),
|
||||
llm_service=config_parser.get_llm_service()
|
||||
)
|
||||
rendered = converter("FILEPATH")
|
||||
```
|
||||
|
||||
### Extract blocks
|
||||
|
||||
Each document consists of one or more pages. Pages contain blocks, which can themselves contain other blocks. It's possible to programmatically manipulate these blocks.
|
||||
|
||||
Here's an example of extracting all forms from a document:
|
||||
|
||||
```python
|
||||
from marker.converters.pdf import PdfConverter
|
||||
from marker.models import create_model_dict
|
||||
from marker.schema import BlockTypes
|
||||
|
||||
converter = PdfConverter(
|
||||
artifact_dict=create_model_dict(),
|
||||
)
|
||||
document = converter.build_document("FILEPATH")
|
||||
forms = document.contained_blocks((BlockTypes.Form,))
|
||||
```
|
||||
|
||||
Look at the processors for more examples of extracting and manipulating blocks.
|
||||
|
||||
## Other converters
|
||||
|
||||
You can also use other converters that define different conversion pipelines:
|
||||
|
||||
### Extract tables
|
||||
|
||||
The `TableConverter` will only convert and extract tables:
|
||||
|
||||
```python
|
||||
from marker.converters.table import TableConverter
|
||||
from marker.models import create_model_dict
|
||||
from marker.output import text_from_rendered
|
||||
|
||||
converter = TableConverter(
|
||||
artifact_dict=create_model_dict(),
|
||||
)
|
||||
rendered = converter("FILEPATH")
|
||||
text, _, images = text_from_rendered(rendered)
|
||||
```
|
||||
|
||||
This takes all the same configuration as the PdfConverter. You can specify the configuration `force_layout_block=Table` to avoid layout detection and instead assume every page is a table. Set `output_format=json` to also get cell bounding boxes.
|
||||
|
||||
You can also run this via the CLI with
|
||||
```shell
|
||||
marker_single FILENAME --use_llm --force_layout_block Table --converter_cls marker.converters.table.TableConverter --output_format json
|
||||
```
|
||||
|
||||
### OCR Only
|
||||
|
||||
If you only want to run OCR, you can also do that through the `OCRConverter`. Set `--keep_chars` to keep individual characters and bounding boxes.
|
||||
|
||||
```python
|
||||
from marker.converters.ocr import OCRConverter
|
||||
from marker.models import create_model_dict
|
||||
|
||||
converter = OCRConverter(
|
||||
artifact_dict=create_model_dict(),
|
||||
)
|
||||
rendered = converter("FILEPATH")
|
||||
```
|
||||
|
||||
This takes all the same configuration as the PdfConverter.
|
||||
|
||||
You can also run this via the CLI with
|
||||
```shell
|
||||
marker_single FILENAME --converter_cls marker.converters.ocr.OCRConverter
|
||||
```
|
||||
|
||||
### Structured Extraction (beta)
|
||||
|
||||
You can run structured extraction via the `ExtractionConverter`. This requires an llm service to be setup first (see [here](#llm-services) for details). You'll get a JSON output with the extracted values.
|
||||
|
||||
```python
|
||||
from marker.converters.extraction import ExtractionConverter
|
||||
from marker.models import create_model_dict
|
||||
from marker.config.parser import ConfigParser
|
||||
from pydantic import BaseModel
|
||||
|
||||
class Links(BaseModel):
|
||||
links: list[str]
|
||||
|
||||
schema = Links.model_json_schema()
|
||||
config_parser = ConfigParser({
|
||||
"page_schema": schema
|
||||
})
|
||||
|
||||
converter = ExtractionConverter(
|
||||
artifact_dict=create_model_dict(),
|
||||
config=config_parser.generate_config_dict(),
|
||||
llm_service=config_parser.get_llm_service(),
|
||||
)
|
||||
rendered = converter("FILEPATH")
|
||||
```
|
||||
|
||||
Rendered will have an `original_markdown` field. If you pass this back in next time you run the converter, as the `existing_markdown` config key, you can skip re-parsing the document.
|
||||
|
||||
# Output Formats
|
||||
|
||||
## Markdown
|
||||
|
||||
Markdown output will include:
|
||||
|
||||
- image links (images will be saved in the same folder)
|
||||
- formatted tables
|
||||
- embedded LaTeX equations (fenced with `$$`)
|
||||
- Code is fenced with triple backticks
|
||||
- Superscripts for footnotes
|
||||
|
||||
## HTML
|
||||
|
||||
HTML output is similar to markdown output:
|
||||
|
||||
- Images are included via `img` tags
|
||||
- equations are fenced with `<math>` tags
|
||||
- code is in `pre` tags
|
||||
|
||||
## JSON
|
||||
|
||||
JSON output will be organized in a tree-like structure, with the leaf nodes being blocks. Examples of leaf nodes are a single list item, a paragraph of text, or an image.
|
||||
|
||||
The output will be a list, with each list item representing a page. Each page is considered a block in the internal marker schema. There are different types of blocks to represent different elements.
|
||||
|
||||
Pages have the keys:
|
||||
|
||||
- `id` - unique id for the block.
|
||||
- `block_type` - the type of block. The possible block types can be seen in `marker/schema/__init__.py`. As of this writing, they are ["Line", "Span", "FigureGroup", "TableGroup", "ListGroup", "PictureGroup", "Page", "Caption", "Code", "Figure", "Footnote", "Form", "Equation", "Handwriting", "TextInlineMath", "ListItem", "PageFooter", "PageHeader", "Picture", "SectionHeader", "Table", "Text", "TableOfContents", "Document"]
|
||||
- `html` - the HTML for the page. Note that this will have recursive references to children. The `content-ref` tags must be replaced with the child content if you want the full html. You can see an example of this at `marker/output.py:json_to_html`. That function will take in a single block from the json output, and turn it into HTML.
|
||||
- `polygon` - the 4-corner polygon of the page, in (x1,y1), (x2,y2), (x3, y3), (x4, y4) format. (x1,y1) is the top left, and coordinates go clockwise.
|
||||
- `children` - the child blocks.
|
||||
|
||||
The child blocks have two additional keys:
|
||||
|
||||
- `section_hierarchy` - indicates the sections that the block is part of. `1` indicates an h1 tag, `2` an h2, and so on.
|
||||
- `images` - base64 encoded images. The key will be the block id, and the data will be the encoded image.
|
||||
|
||||
Note that child blocks of pages can have their own children as well (a tree structure).
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "/page/10/Page/366",
|
||||
"block_type": "Page",
|
||||
"html": "<content-ref src='/page/10/SectionHeader/0'></content-ref><content-ref src='/page/10/SectionHeader/1'></content-ref><content-ref src='/page/10/Text/2'></content-ref><content-ref src='/page/10/Text/3'></content-ref><content-ref src='/page/10/Figure/4'></content-ref><content-ref src='/page/10/SectionHeader/5'></content-ref><content-ref src='/page/10/SectionHeader/6'></content-ref><content-ref src='/page/10/TextInlineMath/7'></content-ref><content-ref src='/page/10/TextInlineMath/8'></content-ref><content-ref src='/page/10/Table/9'></content-ref><content-ref src='/page/10/SectionHeader/10'></content-ref><content-ref src='/page/10/Text/11'></content-ref>",
|
||||
"polygon": [[0.0, 0.0], [612.0, 0.0], [612.0, 792.0], [0.0, 792.0]],
|
||||
"children": [
|
||||
{
|
||||
"id": "/page/10/SectionHeader/0",
|
||||
"block_type": "SectionHeader",
|
||||
"html": "<h1>Supplementary Material for <i>Subspace Adversarial Training</i> </h1>",
|
||||
"polygon": [
|
||||
[217.845703125, 80.630859375], [374.73046875, 80.630859375],
|
||||
[374.73046875, 107.0],
|
||||
[217.845703125, 107.0]
|
||||
],
|
||||
"children": null,
|
||||
"section_hierarchy": {
|
||||
"1": "/page/10/SectionHeader/1"
|
||||
},
|
||||
"images": {}
|
||||
},
|
||||
...
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
```
|
||||
|
||||
## Chunks
|
||||
|
||||
Chunks format is similar to JSON, but flattens everything into a single list instead of a tree. Only the top level blocks from each page show up. It also has the full HTML of each block inside, so you don't need to crawl the tree to reconstruct it. This enable flexible and easy chunking for RAG.
|
||||
|
||||
## Metadata
|
||||
|
||||
All output formats will return a metadata dictionary, with the following fields:
|
||||
|
||||
```json
|
||||
{
|
||||
"table_of_contents": [
|
||||
{
|
||||
"title": "Introduction",
|
||||
"heading_level": 1,
|
||||
"page_id": 0,
|
||||
"polygon": [...]
|
||||
}
|
||||
], // computed PDF table of contents
|
||||
"page_stats": [
|
||||
{
|
||||
"page_id": 0,
|
||||
"text_extraction_method": "pdftext",
|
||||
"block_counts": [("Span", 200), ...]
|
||||
},
|
||||
...
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
# LLM Services
|
||||
|
||||
When running with the `--use_llm` flag, you have a choice of services you can use:
|
||||
|
||||
- `Gemini` - this will use the Gemini developer API by default. You'll need to pass `--gemini_api_key` to configuration.
|
||||
- `Google Vertex` - this will use vertex, which can be more reliable. You'll need to pass `--vertex_project_id`. To use it, set `--llm_service=marker.services.vertex.GoogleVertexService`.
|
||||
- `Ollama` - this will use local models. You can configure `--ollama_base_url` and `--ollama_model`. To use it, set `--llm_service=marker.services.ollama.OllamaService`.
|
||||
- `Claude` - this will use the anthropic API. You can configure `--claude_api_key`, and `--claude_model_name`. To use it, set `--llm_service=marker.services.claude.ClaudeService`.
|
||||
- `OpenAI` - this supports any openai-like endpoint. You can configure `--openai_api_key`, `--openai_model`, and `--openai_base_url`. To use it, set `--llm_service=marker.services.openai.OpenAIService`.
|
||||
- `Azure OpenAI` - this uses the Azure OpenAI service. You can configure `--azure_endpoint`, `--azure_api_key`, and `--deployment_name`. To use it, set `--llm_service=marker.services.azure_openai.AzureOpenAIService`.
|
||||
|
||||
These services may have additional optional configuration as well - you can see it by viewing the classes.
|
||||
|
||||
# Internals
|
||||
|
||||
Marker is easy to extend. The core units of marker are:
|
||||
|
||||
- `Providers`, at `marker/providers`. These provide information from a source file, like a PDF.
|
||||
- `Builders`, at `marker/builders`. These generate the initial document blocks and fill in text, using info from the providers.
|
||||
- `Processors`, at `marker/processors`. These process specific blocks, for example the table formatter is a processor.
|
||||
- `Renderers`, at `marker/renderers`. These use the blocks to render output.
|
||||
- `Schema`, at `marker/schema`. The classes for all the block types.
|
||||
- `Converters`, at `marker/converters`. They run the whole end to end pipeline.
|
||||
|
||||
To customize processing behavior, override the `processors`. To add new output formats, write a new `renderer`. For additional input formats, write a new `provider.`
|
||||
|
||||
Processors and renderers can be directly passed into the base `PDFConverter`, so you can specify your own custom processing easily.
|
||||
|
||||
## API server
|
||||
|
||||
There is a very simple API server you can run like this:
|
||||
|
||||
```shell
|
||||
pip install -U uvicorn fastapi python-multipart
|
||||
marker_server --port 8001
|
||||
```
|
||||
|
||||
This will start a fastapi server that you can access at `localhost:8001`. You can go to `localhost:8001/docs` to see the endpoint options.
|
||||
|
||||
You can send requests like this:
|
||||
|
||||
```
|
||||
import requests
|
||||
import json
|
||||
|
||||
post_data = {
|
||||
'filepath': 'FILEPATH',
|
||||
# Add other params here
|
||||
}
|
||||
|
||||
requests.post("http://localhost:8001/marker", data=json.dumps(post_data)).json()
|
||||
```
|
||||
|
||||
Note that this is not a very robust API, and is only intended for small-scale use. If you want to use this server, but want a more robust conversion option, you can use the hosted [Datalab API](https://www.datalab.to/plans).
|
||||
|
||||
# Troubleshooting
|
||||
|
||||
There are some settings that you may find useful if things aren't working the way you expect:
|
||||
|
||||
- If you have issues with accuracy, try setting `--use_llm` to use an LLM to improve quality. You must set `GOOGLE_API_KEY` to a Gemini API key for this to work.
|
||||
- Make sure to set `force_ocr` if you see garbled text - this will re-OCR the document.
|
||||
- `TORCH_DEVICE` - set this to force marker to use a given torch device for inference.
|
||||
- If you're getting out of memory errors, decrease worker count. You can also try splitting up long PDFs into multiple files.
|
||||
|
||||
## Debugging
|
||||
|
||||
Pass the `debug` option to activate debug mode. This will save images of each page with detected layout and text, as well as output a json file with additional bounding box information.
|
||||
|
||||
# Benchmarks
|
||||
|
||||
## Overall PDF Conversion
|
||||
|
||||
We created a [benchmark set](https://huggingface.co/datasets/datalab-to/marker_benchmark) by extracting single PDF pages from common crawl. We scored based on a heuristic that aligns text with ground truth text segments, and an LLM as a judge scoring method.
|
||||
|
||||
| Method | Avg Time | Heuristic Score | LLM Score |
|
||||
|------------|----------|-----------------|-----------|
|
||||
| marker | 2.83837 | 95.6709 | 4.23916 |
|
||||
| llamaparse | 23.348 | 84.2442 | 3.97619 |
|
||||
| mathpix | 6.36223 | 86.4281 | 4.15626 |
|
||||
| docling | 3.69949 | 86.7073 | 3.70429 |
|
||||
|
||||
Benchmarks were run on an H100 for markjer and docling - llamaparse and mathpix used their cloud services. We can also look at it by document type:
|
||||
|
||||
<img src="data/images/per_doc.png" width="1000px"/>
|
||||
|
||||
| Document Type | Marker heuristic | Marker LLM | Llamaparse Heuristic | Llamaparse LLM | Mathpix Heuristic | Mathpix LLM | Docling Heuristic | Docling LLM |
|
||||
|----------------------|------------------|------------|----------------------|----------------|-------------------|-------------|-------------------|-------------|
|
||||
| Scientific paper | 96.6737 | 4.34899 | 87.1651 | 3.96421 | 91.2267 | 4.46861 | 92.135 | 3.72422 |
|
||||
| Book page | 97.1846 | 4.16168 | 90.9532 | 4.07186 | 93.8886 | 4.35329 | 90.0556 | 3.64671 |
|
||||
| Other | 95.1632 | 4.25076 | 81.1385 | 4.01835 | 79.6231 | 4.00306 | 83.8223 | 3.76147 |
|
||||
| Form | 88.0147 | 3.84663 | 66.3081 | 3.68712 | 64.7512 | 3.33129 | 68.3857 | 3.40491 |
|
||||
| Presentation | 95.1562 | 4.13669 | 81.2261 | 4 | 83.6737 | 3.95683 | 84.8405 | 3.86331 |
|
||||
| Financial document | 95.3697 | 4.39106 | 82.5812 | 4.16111 | 81.3115 | 4.05556 | 86.3882 | 3.8 |
|
||||
| Letter | 98.4021 | 4.5 | 93.4477 | 4.28125 | 96.0383 | 4.45312 | 92.0952 | 4.09375 |
|
||||
| Engineering document | 93.9244 | 4.04412 | 77.4854 | 3.72059 | 80.3319 | 3.88235 | 79.6807 | 3.42647 |
|
||||
| Legal document | 96.689 | 4.27759 | 86.9769 | 3.87584 | 91.601 | 4.20805 | 87.8383 | 3.65552 |
|
||||
| Newspaper page | 98.8733 | 4.25806 | 84.7492 | 3.90323 | 96.9963 | 4.45161 | 92.6496 | 3.51613 |
|
||||
| Magazine page | 98.2145 | 4.38776 | 87.2902 | 3.97959 | 93.5934 | 4.16327 | 93.0892 | 4.02041 |
|
||||
|
||||
## Throughput
|
||||
|
||||
We benchmarked throughput using a [single long PDF](https://www.greenteapress.com/thinkpython/thinkpython.pdf).
|
||||
|
||||
| Method | Time per page | Time per document | VRAM used |
|
||||
|---------|---------------|-------------------|---------- |
|
||||
| marker | 0.18 | 43.42 | 3.17GB |
|
||||
|
||||
The projected throughput is 122 pages per second on an H100 - we can run 22 individual processes given the VRAM used.
|
||||
|
||||
## Table Conversion
|
||||
|
||||
Marker can extract tables from PDFs using `marker.converters.table.TableConverter`. The table extraction performance is measured by comparing the extracted HTML representation of tables against the original HTML representations using the test split of [FinTabNet](https://developer.ibm.com/exchanges/data/all/fintabnet/). The HTML representations are compared using a tree edit distance based metric to judge both structure and content. Marker detects and identifies the structure of all tables in a PDF page and achieves these scores:
|
||||
|
||||
| Method | Avg score | Total tables |
|
||||
|------------------|-----------|--------------|
|
||||
| marker | 0.816 | 99 |
|
||||
| marker w/use_llm | 0.907 | 99 |
|
||||
| gemini | 0.829 | 99 |
|
||||
|
||||
The `--use_llm` flag can significantly improve table recognition performance, as you can see.
|
||||
|
||||
We filter out tables that we cannot align with the ground truth, since fintabnet and our layout model have slightly different detection methods (this results in some tables being split/merged).
|
||||
|
||||
## Running your own benchmarks
|
||||
|
||||
You can benchmark the performance of marker on your machine. Install marker manually with:
|
||||
|
||||
```shell
|
||||
git clone https://github.com/VikParuchuri/marker.git
|
||||
poetry install
|
||||
```
|
||||
|
||||
### Overall PDF Conversion
|
||||
|
||||
Download the benchmark data [here](https://drive.google.com/file/d/1ZSeWDo2g1y0BRLT7KnbmytV2bjWARWba/view?usp=sharing) and unzip. Then run the overall benchmark like this:
|
||||
|
||||
```shell
|
||||
python benchmarks/overall.py --methods marker --scores heuristic,llm
|
||||
```
|
||||
|
||||
Options:
|
||||
|
||||
- `--use_llm` use an llm to improve the marker results.
|
||||
- `--max_rows` how many rows to process for the benchmark.
|
||||
- `--methods` can be `llamaparse`, `mathpix`, `docling`, `marker`. Comma separated.
|
||||
- `--scores` which scoring functions to use, can be `llm`, `heuristic`. Comma separated.
|
||||
|
||||
### Table Conversion
|
||||
The processed FinTabNet dataset is hosted [here](https://huggingface.co/datasets/datalab-to/fintabnet-test) and is automatically downloaded. Run the benchmark with:
|
||||
|
||||
```shell
|
||||
python benchmarks/table/table.py --max_rows 100
|
||||
```
|
||||
|
||||
Options:
|
||||
|
||||
- `--use_llm` uses an llm with marker to improve accuracy.
|
||||
- `--use_gemini` also benchmarks gemini 2.0 flash.
|
||||
|
||||
# How it works
|
||||
|
||||
Marker is a pipeline of deep learning models:
|
||||
|
||||
- Extract text, OCR if necessary (heuristics, [surya](https://github.com/VikParuchuri/surya))
|
||||
- Detect page layout and find reading order ([surya](https://github.com/VikParuchuri/surya))
|
||||
- Clean and format each block (heuristics, [texify](https://github.com/VikParuchuri/texify), [surya](https://github.com/VikParuchuri/surya))
|
||||
- Optionally use an LLM to improve quality
|
||||
- Combine blocks and postprocess complete text
|
||||
|
||||
It only uses models where necessary, which improves speed and accuracy.
|
||||
|
||||
# Limitations
|
||||
|
||||
PDF is a tricky format, so marker will not always work perfectly. Here are some known limitations that are on the roadmap to address:
|
||||
|
||||
- Very complex layouts, with nested tables and forms, may not work
|
||||
- Forms may not be rendered well
|
||||
|
||||
Note: Passing the `--use_llm` and `--force_ocr` flags will mostly solve these issues.
|
||||
|
||||
# Usage and Deployment Examples
|
||||
|
||||
You can always run `marker` locally, but if you wanted to expose it as an API, we have a few options:
|
||||
- Our platform API which is powered by `marker` and `surya` and is easy to test out - it's free to sign up, and we'll include credits, [try it out here](https://datalab.to)
|
||||
- Our painless on-prem solution for commercial use, which you can [read about here](https://www.datalab.to/blog/self-serve-on-prem-licensing) and gives you privacy guarantees with high throughput inference optimizations.
|
||||
- [Deployment example with Modal](./examples/README_MODAL.md) that shows you how to deploy and access `marker` through a web endpoint using [`Modal`](https://modal.com). Modal is an AI compute platform that enables developers to deploy and scale models on GPUs in minutes.
|
||||
@@ -9,20 +9,181 @@ AI-generated cross-book summaries, and engage in grounded RAG chat.
|
||||
```mermaid
|
||||
graph TD
|
||||
User["Neurosurgeon (Browser)"]
|
||||
Login["Login Page\n(username + password form)"]
|
||||
FE["Frontend\nVue.js 3 / Vite\n:5173"]
|
||||
BE["Backend\nSpring Boot 4 / Spring AI\n:8080"]
|
||||
DB["PostgreSQL + pgvector\n(provided)"]
|
||||
LLM["LLM Provider\n(OpenAI / configurable)"]
|
||||
Auth["Spring Security\nHTTP Basic Auth"]
|
||||
DB["PostgreSQL + pgvector\n(source of truth)"]
|
||||
FS["File Store\nuploads/ (local disk)\nExtracted figure PNGs"]
|
||||
LLM["LLM Provider\n(OpenAI)\nEmbeddings + Chat + Vision"]
|
||||
|
||||
User -->|HTTP| FE
|
||||
FE -->|REST /api/v1/...| BE
|
||||
BE -->|JDBC / pgvector| DB
|
||||
BE -->|Embedding + Chat API| LLM
|
||||
User -->|"First visit / unauthenticated"| Login
|
||||
Login -->|"POST credentials\n(GET /api/v1/auth/check)"| Auth
|
||||
Auth -->|"401 → back to login\n200 → app access"| Login
|
||||
Login -->|"Authenticated"| FE
|
||||
FE -->|"REST /api/v1/...\n(HTTP Basic on every request)"| Auth
|
||||
Auth --> BE
|
||||
BE -->|"JDBC — books, chapters,\nsections, figures, refs"| DB
|
||||
BE -->|"pgvector — text chunks\n+ figure caption vectors"| DB
|
||||
BE -->|"PNG read/write\n(figure extraction)"| FS
|
||||
FE -->|"GET /api/v1/figures/**\n(static file serving)"| BE
|
||||
BE -->|"Embedding + Chat\n+ Vision (image description)"| LLM
|
||||
|
||||
subgraph "Embedding Pipeline (per PDF upload)"
|
||||
EP1["Parse pages → SectionEntity"]
|
||||
EP2["Extract images → FigureEntity"]
|
||||
EP3["Vision describe → embed caption"]
|
||||
EP4["Chunk text → embed chunks"]
|
||||
EP5["Link chunks ↔ figures"]
|
||||
EP6["LLM enrich chunk\n(entities, facet, summary)\n→ chunk_metadata"]
|
||||
EP1 --> EP2
|
||||
EP1 --> EP4
|
||||
EP2 --> EP3
|
||||
EP4 --> EP5
|
||||
EP3 --> EP5
|
||||
EP4 --> EP6
|
||||
end
|
||||
|
||||
subgraph "Retrieval Pipeline (per chat query)"
|
||||
RP0["Query expansion\n(QueryExpansionService)\nlay → clinical terms"]
|
||||
RP1["Text chunk search (topK=5)"]
|
||||
RP2["Figure caption search (topK=3)"]
|
||||
RP3["Expand chunks → ±1-page section text"]
|
||||
RP4["Fetch linked figures (chunk_figure_ref)"]
|
||||
RP5["Merge + deduplicate figures"]
|
||||
RP6["Build labelled prompt\n[S1],[F1]… tags"]
|
||||
RP7["LLM chat call"]
|
||||
RP8["Citation validation\n(CitationValidatorService)\nstrip hallucinated refs"]
|
||||
RP0 --> RP1
|
||||
RP0 --> RP2
|
||||
RP1 --> RP3
|
||||
RP1 --> RP4
|
||||
RP2 --> RP5
|
||||
RP4 --> RP5
|
||||
RP3 --> RP6
|
||||
RP5 --> RP6
|
||||
RP6 --> RP7
|
||||
RP7 --> RP8
|
||||
end
|
||||
```
|
||||
|
||||
### Concept Retrieval Pipeline (per concept report)
|
||||
|
||||
Concept retrieval is an alternative to the semantic-similarity flow above. It uses the
|
||||
LLM-tagged `chunk_metadata` rows written at indexing time to exhaustively gather every
|
||||
chunk that *concerns* a concept (e.g. "aneurysm"), bucketed by facet. One synthesis call
|
||||
per facet yields a structured, multi-section report.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant User
|
||||
participant FE as Frontend
|
||||
participant BE as Backend (ConceptReportService)
|
||||
participant Retr as ConceptRetriever
|
||||
participant DB as chunk_metadata (GIN)
|
||||
participant Vec as vector_store
|
||||
participant LLM
|
||||
|
||||
User->>FE: Click "Generate Concept Report" on topic
|
||||
FE->>BE: POST /api/v1/topics/{id}/concept-reports
|
||||
loop per READY book
|
||||
BE->>Retr: retrieveByConcept(topicName, bookId)
|
||||
Retr->>DB: WHERE entities @> [canonical]
|
||||
alt SQL hits found
|
||||
DB-->>Retr: chunks grouped by facet
|
||||
else no match (typo / synonym)
|
||||
Retr->>Vec: similaritySearch topK=30
|
||||
Vec-->>Retr: chunk ids
|
||||
Retr->>DB: findByChunkIdIn → group by facet
|
||||
end
|
||||
end
|
||||
BE->>BE: merge facets across books, assign global [S#]/[F#]
|
||||
loop per non-empty facet
|
||||
BE->>LLM: synthesize facet section (focused prompt)
|
||||
LLM-->>BE: facet markdown
|
||||
end
|
||||
BE->>BE: persist concept_report
|
||||
BE-->>FE: { facets[], sources[] }
|
||||
FE->>User: render facet-labelled report + inline figures
|
||||
```
|
||||
|
||||
Backfill path for already-embedded books:
|
||||
`POST /api/v1/admin/books/{id}/enrich` scans `vector_store` for TEXT chunks missing
|
||||
`chunk_metadata` rows and enriches them in place. Idempotent — re-running is a no-op.
|
||||
|
||||
## Marker API Response Structure
|
||||
|
||||
The PDF parsing pipeline calls a local [Marker](https://github.com/VikParuchuri/marker) server (`POST /marker/upload`).
|
||||
|
||||
### Top-level envelope
|
||||
|
||||
```json
|
||||
{
|
||||
"format": "json",
|
||||
"output": "<JSON-encoded string>"
|
||||
}
|
||||
```
|
||||
|
||||
`output` is a **JSON-encoded string** (not a nested object) and must be parsed a second time to get the document tree.
|
||||
|
||||
### Parsed `output` shape
|
||||
|
||||
```
|
||||
{
|
||||
"children": [ <Page block>, ... ]
|
||||
}
|
||||
```
|
||||
|
||||
### Block types
|
||||
|
||||
Every block shares these fields:
|
||||
|
||||
| Field | Type | Notes |
|
||||
|------------------|-------------------|--------------------------------------------|
|
||||
| `id` | string | e.g. `/page/0/Picture/2` |
|
||||
| `block_type` | string | see table below |
|
||||
| `html` | string | rendered HTML; may contain `<content-ref>` |
|
||||
| `bbox` | `[x0,y0,x1,y1]` | bounding box in page coordinates |
|
||||
| `children` | array or null | nested blocks |
|
||||
| `images` | object or null | base64 PNG map (leaf image blocks only) |
|
||||
| `section_hierarchy` | object | heading ancestry |
|
||||
|
||||
#### Known `block_type` values
|
||||
|
||||
| block_type | Category | Notes |
|
||||
|------------------|----------|-------------------------------------------------------|
|
||||
| `Page` | structure | Top-level; direct children are the page content |
|
||||
| `SectionHeader` | text | Section / chapter heading |
|
||||
| `Text` | text | |
|
||||
| `TextInlineMath` | text | |
|
||||
| `ListItem` | text | |
|
||||
| `Table` | text | |
|
||||
| `Code` | text | |
|
||||
| `Equation` | text | |
|
||||
| `Footnote` | text | |
|
||||
| `Caption` | text | Usually a child of a `*Group` block |
|
||||
| `PageHeader` | text | |
|
||||
| `PageFooter` | text | |
|
||||
| `Handwriting` | text | |
|
||||
| `Picture` | image | Leaf block; `images` map holds base64 PNG keyed by ID |
|
||||
| `Figure` | image | Leaf block; same as `Picture` |
|
||||
| `PictureGroup` | container | Wraps one `Picture` + one `Caption` child |
|
||||
| `FigureGroup` | container | Wraps one `Figure` + one `Caption` child |
|
||||
|
||||
### Image extraction
|
||||
|
||||
Images are only present on **leaf** image blocks (`Picture`, `Figure`).
|
||||
Group blocks (`PictureGroup`, `FigureGroup`) have `images: null` — the base64 PNG lives on the child leaf block.
|
||||
|
||||
```
|
||||
PictureGroup
|
||||
├── Picture ← images: { "/page/0/Picture/2": "<base64 PNG>" }
|
||||
└── Caption ← html: "<p>Figure 1 — ...</p>"
|
||||
```
|
||||
|
||||
## Stack
|
||||
|
||||
- **Backend**: Spring Boot 4.0.5 + Spring AI 2.0.0-M4, Java 21, Maven
|
||||
- **Backend**: Spring Boot 4.0.5 + Spring AI 2.0.0-M4, Java 25, Maven
|
||||
- **Frontend**: Vue.js 3 + Vite + TypeScript + Pinia + Axios
|
||||
- **Database**: PostgreSQL 16 + pgvector extension
|
||||
- **Auth**: HTTP Basic (single shared in-memory user)
|
||||
@@ -31,7 +192,7 @@ graph TD
|
||||
|
||||
See [specs/001-neuro-rag-learning/quickstart.md](specs/001-neuro-rag-learning/quickstart.md) for full instructions.
|
||||
|
||||
### Local Dev
|
||||
### Local Dev (JVM)
|
||||
|
||||
```bash
|
||||
# Start the database
|
||||
@@ -47,8 +208,99 @@ npm install
|
||||
npm run dev
|
||||
```
|
||||
|
||||
### Native Image Build
|
||||
|
||||
Produces a GraalVM native binary packaged into a minimal Docker image via Jib.
|
||||
|
||||
**Prerequisite**: GraalVM 25 must be installed and set as `JAVA_HOME`.
|
||||
|
||||
```bash
|
||||
# Install GraalVM 25 CE via sdkman (one-time)
|
||||
sdk install java 25-graalce
|
||||
sdk use java 25-graalce
|
||||
|
||||
# Build native executable + Docker image (requires Docker daemon)
|
||||
cd backend
|
||||
mvn -Pnative package jib:build -DskipTests
|
||||
mvn -Pnative jib:build -Djib.to.auth.username=admin -Djib.to.auth.password=""
|
||||
```
|
||||
|
||||
### Backend build (buildah)
|
||||
|
||||
**JVM image** (`Dockerfile` — Eclipse Temurin 21):
|
||||
|
||||
```bash
|
||||
buildah build \
|
||||
--platform linux/arm64 \
|
||||
--tag zot.immich-ad.ovh/ai-teacher-backend:latest \
|
||||
backend/
|
||||
|
||||
buildah login zot.immich-ad.ovh
|
||||
buildah push --tls-verify=false zot.immich-ad.ovh/ai-teacher-backend:latest
|
||||
```
|
||||
|
||||
**Native image** (`Dockerfile.native` — GraalVM 25, produces a minimal Debian-slim image):
|
||||
|
||||
```bash
|
||||
buildah build \
|
||||
--platform linux/arm64 \
|
||||
--file backend/Dockerfile.native \
|
||||
--tag zot.immich-ad.ovh/ai-teacher-backend-native:latest \
|
||||
backend/
|
||||
|
||||
buildah push --tls-verify=false zot.immich-ad.ovh/ai-teacher-backend-native:latest
|
||||
```
|
||||
|
||||
### Frontend build
|
||||
```
|
||||
buildah build \
|
||||
--platform linux/arm64 \
|
||||
--tag zot.immich-ad.ovh/ai-teacher-frontend:latest \
|
||||
frontend/
|
||||
buildah login zot.immich-ad.ovh
|
||||
```
|
||||
Push to the private repository:
|
||||
|
||||
```
|
||||
buildah push --tls-verify=false zot.immich-ad.ovh/ai-teacher-frontend:latest
|
||||
|
||||
```
|
||||
|
||||
### Run Native Stack (Docker Compose)
|
||||
|
||||
```bash
|
||||
# Copy and fill in secrets
|
||||
cp .env.example .env
|
||||
# edit .env — add OPENAI_API_KEY at minimum
|
||||
|
||||
# Start PostgreSQL + native backend
|
||||
docker compose -f docker-compose.native.yml up
|
||||
```
|
||||
|
||||
App available at `http://localhost:8080`.
|
||||
|
||||
### Build Pipeline (Native)
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
SRC["Source Code\n(Java 25)"]
|
||||
AOT["Spring Boot AOT\n(process-aot)"]
|
||||
NI["GraalVM native-image\n(native-maven-plugin)"]
|
||||
EXE["Native Executable\ntarget/ai-teacher-backend"]
|
||||
JIB["Jib\n(jib-native-image-extension)"]
|
||||
IMG["Docker Image\nai-teacher-backend:latest\n(distroless base)"]
|
||||
|
||||
SRC --> AOT
|
||||
AOT --> NI
|
||||
NI --> EXE
|
||||
EXE --> JIB
|
||||
JIB --> IMG
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
#### Backend
|
||||
|
||||
| Variable | Required | Description |
|
||||
|----------|----------|-------------|
|
||||
| `OPENAI_API_KEY` | Yes | OpenAI API key for embeddings and chat |
|
||||
@@ -56,3 +308,15 @@ npm run dev
|
||||
| `DB_URL` | Yes | JDBC URL, e.g. `jdbc:postgresql://localhost:5432/aiteacher` |
|
||||
| `DB_USERNAME` | Yes | Database username |
|
||||
| `DB_PASSWORD` | Yes | Database password |
|
||||
| `FIGURE_STORAGE_PATH` | No | Base path for uploaded PDFs and extracted figures (default: `./uploads`) |
|
||||
| `UPLOAD_ENABLED` | No | Set to `false` to disable the book upload endpoint (default: `true`) |
|
||||
| `DELETE_ENABLED` | No | Set to `false` to disable the book delete endpoint (default: `true`) |
|
||||
|
||||
#### Frontend
|
||||
|
||||
| Variable | Required | Description |
|
||||
|----------|----------|-------------|
|
||||
| `VITE_API_URL` | No | Backend API base URL (default: `/api/v1`) |
|
||||
| `VITE_APP_PASSWORD` | Yes | Shared password for HTTP Basic auth (must match `APP_PASSWORD`) |
|
||||
| `VITE_UPLOAD_ENABLED` | No | Set to `false` to hide the upload UI (default: `true`) |
|
||||
| `VITE_DELETE_ENABLED` | No | Set to `false` to hide the delete button (default: `true`) |
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
# Java build artifacts
|
||||
target/
|
||||
*.class
|
||||
*.jar
|
||||
|
||||
# Git
|
||||
.git/
|
||||
.gitignore
|
||||
|
||||
# Editor
|
||||
.vscode/
|
||||
.idea/
|
||||
*.iml
|
||||
|
||||
# OS
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# Logs
|
||||
*.log
|
||||
|
||||
# Environment
|
||||
.env
|
||||
.env.*
|
||||
@@ -0,0 +1,25 @@
|
||||
# ---- Pull Maven from its official image (avoids microdnf under QEMU) ----
|
||||
FROM docker.io/library/maven:3.9.9-eclipse-temurin-21 AS maven-dist
|
||||
|
||||
# ---- Build stage: GraalVM 25 + Maven ----
|
||||
FROM ghcr.io/graalvm/native-image-community:25 AS build
|
||||
|
||||
# Copy Maven from the official Maven image — no package installation needed
|
||||
COPY --from=maven-dist /usr/share/maven /opt/maven
|
||||
ENV PATH="/opt/maven/bin:$PATH"
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Cache dependency resolution separately from source compilation
|
||||
COPY pom.xml .
|
||||
RUN mvn -Pnative dependency:resolve dependency:resolve-plugins -q
|
||||
|
||||
# Build native executable
|
||||
COPY src ./src
|
||||
RUN mvn -Pnative package -DskipTests
|
||||
|
||||
# ---- Runtime stage: slim Debian with glibc + libz (required by GraalVM native binary) ----
|
||||
FROM docker.io/library/debian:12-slim
|
||||
COPY --from=build /app/target/ai-teacher-backend /app/ai-teacher-backend
|
||||
EXPOSE 8080
|
||||
ENTRYPOINT ["/app/ai-teacher-backend"]
|
||||
+126
-2
@@ -32,6 +32,13 @@
|
||||
<type>pom</type>
|
||||
<scope>import</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>software.amazon.awssdk</groupId>
|
||||
<artifactId>bom</artifactId>
|
||||
<version>2.30.14</version>
|
||||
<type>pom</type>
|
||||
<scope>import</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
</dependencyManagement>
|
||||
|
||||
@@ -95,12 +102,25 @@
|
||||
<artifactId>spring-ai-advisors-vector-store</artifactId>
|
||||
</dependency>
|
||||
|
||||
<!-- Spring AI — PDF document reader -->
|
||||
<!-- Spring AI — PDF document reader (includes PDFBox transitively) -->
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-pdf-document-reader</artifactId>
|
||||
</dependency>
|
||||
|
||||
<!-- PDFBox — page rendering and cropping for figure extraction -->
|
||||
<dependency>
|
||||
<groupId>org.apache.pdfbox</groupId>
|
||||
<artifactId>pdfbox</artifactId>
|
||||
<version>3.0.3</version>
|
||||
</dependency>
|
||||
|
||||
<!-- AWS SDK v2 — S3 figure storage -->
|
||||
<dependency>
|
||||
<groupId>software.amazon.awssdk</groupId>
|
||||
<artifactId>s3</artifactId>
|
||||
</dependency>
|
||||
|
||||
<!-- Jackson (JSON) -->
|
||||
<dependency>
|
||||
<groupId>com.fasterxml.jackson.core</groupId>
|
||||
@@ -120,15 +140,119 @@
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.graalvm.buildtools</groupId>
|
||||
<artifactId>native-maven-plugin</artifactId>
|
||||
</plugin>
|
||||
|
||||
<plugin>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-maven-plugin</artifactId>
|
||||
</plugin>
|
||||
|
||||
<!-- Jib — package native executable (or fat-jar) into Docker image -->
|
||||
<plugin>
|
||||
<groupId>com.google.cloud.tools</groupId>
|
||||
<artifactId>jib-maven-plugin</artifactId>
|
||||
<version>3.5.1</version>
|
||||
<configuration>
|
||||
<from>
|
||||
<!-- distroless glibc base — includes libz + libssl needed by GraalVM native binary -->
|
||||
<image>gcr.io/distroless/base-debian12</image>
|
||||
</from>
|
||||
<to>
|
||||
<image>zot.immich-ad.ovh/ai-teacher-backend</image>
|
||||
<tags>
|
||||
<tag>latest</tag>
|
||||
</tags>
|
||||
</to>
|
||||
<container>
|
||||
<format>OCI</format>
|
||||
<ports>
|
||||
<port>8080</port>
|
||||
</ports>
|
||||
<!-- invoke the native binary directly — no JVM -->
|
||||
<entrypoint>
|
||||
<arg>/app/ai-teacher-backend</arg>
|
||||
</entrypoint>
|
||||
</container>
|
||||
<!-- copy the GraalVM-compiled binary from target/ into /app/ -->
|
||||
<extraDirectories>
|
||||
<paths>
|
||||
<path>
|
||||
<from>${project.build.directory}</from>
|
||||
<into>/app</into>
|
||||
<includes>ai-teacher-backend</includes>
|
||||
</path>
|
||||
</paths>
|
||||
<permissions>
|
||||
<permission>
|
||||
<file>/app/ai-teacher-backend</file>
|
||||
<mode>755</mode>
|
||||
</permission>
|
||||
</permissions>
|
||||
</extraDirectories>
|
||||
</configuration>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
|
||||
<profiles>
|
||||
<profile>
|
||||
<id>native</id>
|
||||
<build>
|
||||
<plugins>
|
||||
|
||||
<!-- skip jib in native builds — use Dockerfile.native + buildah instead -->
|
||||
<plugin>
|
||||
<groupId>com.google.cloud.tools</groupId>
|
||||
<artifactId>jib-maven-plugin</artifactId>
|
||||
<configuration>
|
||||
<skip>true</skip>
|
||||
</configuration>
|
||||
</plugin>
|
||||
|
||||
<!-- GraalVM native-image compilation -->
|
||||
<plugin>
|
||||
<groupId>org.graalvm.buildtools</groupId>
|
||||
<artifactId>native-maven-plugin</artifactId>
|
||||
<version>1.0.0</version>
|
||||
<executions>
|
||||
<execution>
|
||||
<id>add-reachability-metadata</id>
|
||||
<goals>
|
||||
<goal>add-reachability-metadata</goal>
|
||||
</goals>
|
||||
</execution>
|
||||
<execution>
|
||||
<id>compile</id>
|
||||
<goals>
|
||||
<goal>compile-no-fork</goal>
|
||||
</goals>
|
||||
<phase>package</phase>
|
||||
</execution>
|
||||
</executions>
|
||||
<configuration>
|
||||
<imageName>ai-teacher-backend</imageName>
|
||||
<buildArgs>
|
||||
<buildArg>--initialize-at-build-time=org.slf4j,ch.qos.logback</buildArg>
|
||||
<buildArg>-H:+ReportExceptionStackTraces</buildArg>
|
||||
<buildArg>--gc=serial</buildArg>
|
||||
<buildArg>-Os</buildArg>
|
||||
<buildArg>-H:+RemoveUnusedSymbols</buildArg>
|
||||
<buildArg>-H:-EnableLoggingFeature</buildArg>
|
||||
<buildArg>-R:MaxHeapSize=128m</buildArg>
|
||||
<buildArg>-R:MinHeapSize=32m</buildArg>
|
||||
<!-- Limit native-image compiler RAM (build time, not runtime) -->
|
||||
<buildArg>-J-Xmx8g</buildArg>
|
||||
</buildArgs>
|
||||
</configuration>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
</profile>
|
||||
</profiles>
|
||||
|
||||
</project>
|
||||
|
||||
@@ -1,11 +1,15 @@
|
||||
package com.aiteacher;
|
||||
|
||||
import org.springframework.context.annotation.ImportRuntimeHints;
|
||||
import org.springframework.boot.SpringApplication;
|
||||
import org.springframework.boot.autoconfigure.SpringBootApplication;
|
||||
import org.springframework.scheduling.annotation.EnableAsync;
|
||||
|
||||
import com.aiteacher.config.NativeHintsConfig;
|
||||
|
||||
@SpringBootApplication
|
||||
@EnableAsync
|
||||
@ImportRuntimeHints(NativeHintsConfig.class)
|
||||
public class AiTeacherApplication {
|
||||
|
||||
public static void main(String[] args) {
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
package com.aiteacher.auth;
|
||||
|
||||
import org.springframework.http.ResponseEntity;
|
||||
import org.springframework.web.bind.annotation.GetMapping;
|
||||
import org.springframework.web.bind.annotation.RequestMapping;
|
||||
import org.springframework.web.bind.annotation.RestController;
|
||||
|
||||
import java.security.Principal;
|
||||
import java.util.Map;
|
||||
|
||||
@RestController
|
||||
@RequestMapping("/api/v1/auth")
|
||||
public class AuthController {
|
||||
|
||||
@GetMapping("/check")
|
||||
public ResponseEntity<Map<String, String>> check(Principal principal) {
|
||||
return ResponseEntity.ok(Map.of("username", principal.getName()));
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,11 @@
|
||||
package com.aiteacher.book;
|
||||
|
||||
import com.aiteacher.document.FigureEntity;
|
||||
import com.aiteacher.document.FigureRepository;
|
||||
import com.aiteacher.document.MarkdownStorageService;
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.http.HttpStatus;
|
||||
import org.springframework.http.MediaType;
|
||||
import org.springframework.http.ResponseEntity;
|
||||
import org.springframework.web.bind.annotation.*;
|
||||
import org.springframework.web.multipart.MultipartFile;
|
||||
@@ -15,13 +20,25 @@ import java.util.UUID;
|
||||
public class BookController {
|
||||
|
||||
private final BookService bookService;
|
||||
private final FigureRepository figureRepository;
|
||||
private final MarkdownStorageService markdownStorageService;
|
||||
|
||||
public BookController(BookService bookService) {
|
||||
@Value("${app.features.upload-enabled:true}")
|
||||
private boolean uploadEnabled;
|
||||
|
||||
@Value("${app.features.delete-enabled:true}")
|
||||
private boolean deleteEnabled;
|
||||
|
||||
public BookController(BookService bookService, FigureRepository figureRepository,
|
||||
MarkdownStorageService markdownStorageService) {
|
||||
this.bookService = bookService;
|
||||
this.figureRepository = figureRepository;
|
||||
this.markdownStorageService = markdownStorageService;
|
||||
}
|
||||
|
||||
@PostMapping(consumes = "multipart/form-data")
|
||||
public ResponseEntity<?> upload(@RequestParam("file") MultipartFile file) throws IOException {
|
||||
if (!uploadEnabled) return ResponseEntity.status(HttpStatus.METHOD_NOT_ALLOWED).build();
|
||||
Book book = bookService.upload(file);
|
||||
return ResponseEntity.status(HttpStatus.ACCEPTED).body(toSummaryResponse(book));
|
||||
}
|
||||
@@ -42,10 +59,52 @@ public class BookController {
|
||||
|
||||
@DeleteMapping("/{id}")
|
||||
public ResponseEntity<Void> delete(@PathVariable UUID id) {
|
||||
if (!deleteEnabled) return ResponseEntity.status(HttpStatus.METHOD_NOT_ALLOWED).build();
|
||||
bookService.delete(id);
|
||||
return ResponseEntity.noContent().build();
|
||||
}
|
||||
|
||||
@PostMapping("/{id}/reembed")
|
||||
public ResponseEntity<Map<String, Object>> reembed(@PathVariable UUID id) {
|
||||
Book book = bookService.reembed(id);
|
||||
return ResponseEntity.accepted().body(Map.of(
|
||||
"bookId", book.getId(),
|
||||
"status", BookStatus.PROCESSING.name()
|
||||
));
|
||||
}
|
||||
|
||||
@GetMapping(value = "/{id}/pages/{pageNumber}/html", produces = MediaType.TEXT_HTML_VALUE)
|
||||
public ResponseEntity<String> getPageHtml(@PathVariable UUID id,
|
||||
@PathVariable int pageNumber) {
|
||||
bookService.getById(id); // 404 if not found
|
||||
try {
|
||||
return ResponseEntity.ok(markdownStorageService.getText(id, pageNumber));
|
||||
} catch (Exception e) {
|
||||
return ResponseEntity.notFound().build();
|
||||
}
|
||||
}
|
||||
|
||||
@GetMapping("/{id}/figures")
|
||||
public ResponseEntity<List<FigureResponse>> figures(@PathVariable UUID id) {
|
||||
bookService.getById(id); // 404 if not found
|
||||
List<FigureResponse> responses = figureRepository.findAllByBookId(id)
|
||||
.stream()
|
||||
.map(f -> toFigureResponse(id, f))
|
||||
.toList();
|
||||
return ResponseEntity.ok(responses);
|
||||
}
|
||||
|
||||
private FigureResponse toFigureResponse(UUID bookId, FigureEntity f) {
|
||||
String filename = f.getImagePath().substring(f.getImagePath().lastIndexOf('/') + 1);
|
||||
String imageUrl = "/api/v1/figures/" + bookId + "/" + filename;
|
||||
return new FigureResponse(
|
||||
f.getId(), f.getLabel(), f.getCaption(),
|
||||
f.getFigureType().name(), f.getPage(), imageUrl,
|
||||
f.getSectionId(),
|
||||
null // section title not eagerly loaded here
|
||||
);
|
||||
}
|
||||
|
||||
private Map<String, Object> toSummaryResponse(Book book) {
|
||||
return Map.of(
|
||||
"id", book.getId(),
|
||||
|
||||
@@ -1,41 +1,90 @@
|
||||
package com.aiteacher.book;
|
||||
|
||||
import com.aiteacher.document.*;
|
||||
import com.aiteacher.enrichment.ChunkEnrichmentPipeline;
|
||||
import com.aiteacher.enrichment.ChunkMetadataRepository;
|
||||
import com.aiteacher.figure.FigureStorageService;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.reader.pdf.PagePdfDocumentReader;
|
||||
import org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig;
|
||||
import org.springframework.ai.vectorstore.VectorStore;
|
||||
import org.springframework.ai.vectorstore.filter.FilterExpressionBuilder;
|
||||
import org.springframework.core.io.FileSystemResource;
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.scheduling.annotation.Async;
|
||||
import org.springframework.stereotype.Service;
|
||||
import org.springframework.transaction.annotation.Transactional;
|
||||
|
||||
import java.nio.file.Path;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
import java.util.regex.Pattern;
|
||||
import java.time.Instant;
|
||||
import java.util.*;
|
||||
|
||||
@Service
|
||||
public class BookEmbeddingService {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(BookEmbeddingService.class);
|
||||
|
||||
// Pattern to detect diagram/figure captions
|
||||
private static final Pattern CAPTION_PATTERN =
|
||||
Pattern.compile("^(Figure|Fig\\.|Table|Diagram)\\s+[\\d.]+", Pattern.CASE_INSENSITIVE);
|
||||
|
||||
private final VectorStore vectorStore;
|
||||
private final BookRepository bookRepository;
|
||||
private final MarkerPageParser markerPageParser;
|
||||
private final FigureExtractionService figureExtractionService;
|
||||
private final VisionDescriptionService visionDescriptionService;
|
||||
private final TextChunkingService textChunkingService;
|
||||
private final ChunkFigureRefService chunkFigureRefService;
|
||||
private final SectionRepository sectionRepository;
|
||||
private final ChapterRepository chapterRepository;
|
||||
private final FigureRepository figureRepository;
|
||||
private final ChunkFigureRefRepository chunkFigureRefRepository;
|
||||
private final FigureStorageService figureStorageService;
|
||||
private final MarkdownStorageService markdownStorageService;
|
||||
private final ChunkEnrichmentPipeline chunkEnrichmentPipeline;
|
||||
private final ChunkMetadataRepository chunkMetadataRepository;
|
||||
|
||||
public BookEmbeddingService(VectorStore vectorStore, BookRepository bookRepository) {
|
||||
@Value("${app.embedding.batch-size:50}")
|
||||
private int embeddingBatchSize;
|
||||
|
||||
@Value("${app.embedding.batch-delay-ms:1000}")
|
||||
private long embeddingBatchDelayMs;
|
||||
|
||||
@Value("${app.embedding.skip-embedding:false}")
|
||||
private boolean skipEmbedding;
|
||||
|
||||
public BookEmbeddingService(
|
||||
VectorStore vectorStore,
|
||||
BookRepository bookRepository,
|
||||
MarkerPageParser markerPageParser,
|
||||
FigureExtractionService figureExtractionService,
|
||||
VisionDescriptionService visionDescriptionService,
|
||||
TextChunkingService textChunkingService,
|
||||
ChunkFigureRefService chunkFigureRefService,
|
||||
SectionRepository sectionRepository,
|
||||
ChapterRepository chapterRepository,
|
||||
FigureRepository figureRepository,
|
||||
ChunkFigureRefRepository chunkFigureRefRepository,
|
||||
FigureStorageService figureStorageService,
|
||||
MarkdownStorageService markdownStorageService,
|
||||
ChunkEnrichmentPipeline chunkEnrichmentPipeline,
|
||||
ChunkMetadataRepository chunkMetadataRepository) {
|
||||
this.vectorStore = vectorStore;
|
||||
this.bookRepository = bookRepository;
|
||||
this.markerPageParser = markerPageParser;
|
||||
this.figureExtractionService = figureExtractionService;
|
||||
this.visionDescriptionService = visionDescriptionService;
|
||||
this.textChunkingService = textChunkingService;
|
||||
this.chunkFigureRefService = chunkFigureRefService;
|
||||
this.sectionRepository = sectionRepository;
|
||||
this.chapterRepository = chapterRepository;
|
||||
this.figureRepository = figureRepository;
|
||||
this.chunkFigureRefRepository = chunkFigureRefRepository;
|
||||
this.figureStorageService = figureStorageService;
|
||||
this.markdownStorageService = markdownStorageService;
|
||||
this.chunkEnrichmentPipeline = chunkEnrichmentPipeline;
|
||||
this.chunkMetadataRepository = chunkMetadataRepository;
|
||||
}
|
||||
|
||||
@Async
|
||||
public void embedBook(UUID bookId, String bookTitle, Path pdfPath) {
|
||||
log.info("Starting embedding for book {} ({})", bookId, bookTitle);
|
||||
log.info("Starting Marker-powered embedding for book {} ({})", bookId, bookTitle);
|
||||
|
||||
Book book = bookRepository.findById(bookId).orElse(null);
|
||||
if (book == null) {
|
||||
@@ -47,29 +96,102 @@ public class BookEmbeddingService {
|
||||
book.setStatus(BookStatus.PROCESSING);
|
||||
bookRepository.save(book);
|
||||
|
||||
PagePdfDocumentReader reader = new PagePdfDocumentReader(
|
||||
new FileSystemResource(pdfPath.toFile()),
|
||||
PdfDocumentReaderConfig.builder()
|
||||
.withPagesPerDocument(1)
|
||||
.build()
|
||||
);
|
||||
String chapterId = bookId + "-ch1";
|
||||
ChapterEntity chapter = new ChapterEntity(chapterId, bookId, 1, bookTitle, 1);
|
||||
chapterRepository.save(chapter);
|
||||
|
||||
List<Document> pages = reader.get();
|
||||
int pageCount = pages.size();
|
||||
// Step 1: Parse with Marker — split into 100-page chunks, then merge results
|
||||
ParsedBook parsed = markerPageParser.parse(pdfPath);
|
||||
|
||||
// Enrich metadata and tag diagram captions
|
||||
List<Document> enriched = pages.stream()
|
||||
.map(doc -> enrichDocument(doc, bookId.toString(), bookTitle))
|
||||
.toList();
|
||||
List<PageResult> pageResults = parsed.pages();
|
||||
|
||||
vectorStore.add(enriched);
|
||||
// Step 2: Build SectionEntity per page and persist
|
||||
List<SectionEntity> sections = buildAndSaveSections(bookId, bookTitle, chapterId, pageResults);
|
||||
|
||||
// Step 3: Chunk and embed text
|
||||
List<Document> allChunks = new ArrayList<>();
|
||||
for (SectionEntity section : sections) {
|
||||
allChunks.addAll(textChunkingService.chunk(section, bookTitle));
|
||||
}
|
||||
if (skipEmbedding) {
|
||||
log.info("skip-embedding=true — skipping text embedding for book {}", bookId);
|
||||
} else {
|
||||
embedInBatches(allChunks, bookId);
|
||||
log.info("Embedded {} text chunks for book {}", allChunks.size(), bookId);
|
||||
Map<String, SectionEntity> sectionsById = new HashMap<>();
|
||||
for (SectionEntity s : sections) sectionsById.put(s.getId(), s);
|
||||
try {
|
||||
chunkEnrichmentPipeline.enrichAndPersist(allChunks, sectionsById, bookTitle);
|
||||
} catch (Exception ex) {
|
||||
log.warn("Chunk enrichment failed for book {} — backfill endpoint can recover: {}",
|
||||
bookId, ex.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
// Step 4: Decode pre-cropped figures from Marker output
|
||||
FigureExtractionService.ExtractionResult extraction =
|
||||
figureExtractionService.extract(bookId, chapterId, pageResults);
|
||||
List<FigureEntity> figures = extraction.figures();
|
||||
|
||||
// Step 4b: Save per-page HTML to S3, replacing Marker image src with API URLs
|
||||
parsed.htmlByPage().forEach((pageNumber, html) -> {
|
||||
String resolved = resolveImageSrcs(html, bookId, extraction.blockIdToFigureId());
|
||||
markdownStorageService.save(bookId, pageNumber, resolved);
|
||||
});
|
||||
log.info("Saved {} HTML pages to S3 for book {}", parsed.htmlByPage().size(), bookId);
|
||||
|
||||
// Step 5: Vision analysis (description + visible text) → embed figure chunks
|
||||
Map<String, SectionEntity> sectionById = new HashMap<>();
|
||||
for (SectionEntity s : sections) sectionById.put(s.getId(), s);
|
||||
|
||||
for (FigureEntity figure : figures) {
|
||||
// Prefer caption extracted from the linked section's full text
|
||||
if (figure.getCaption() == null || figure.getCaption().isBlank()) {
|
||||
String sectionCaption = extractCaptionFromSection(sectionById.get(figure.getSectionId()));
|
||||
if (sectionCaption != null) {
|
||||
figure.setCaption(sectionCaption);
|
||||
figureRepository.save(figure);
|
||||
} else {
|
||||
byte[] imageBytes = figureStorageService.getBytes(figure.getImagePath());
|
||||
VisionDescriptionService.ImageAnalysis analysis =
|
||||
visionDescriptionService.analyze(imageBytes, figure.getCaption());
|
||||
figure.setCaption(analysis.description());
|
||||
figureRepository.save(figure);
|
||||
}
|
||||
}
|
||||
|
||||
// Embedding content: description
|
||||
String embeddingContent = (figure.getCaption() != null ? "\n" + figure.getCaption() : "");
|
||||
|
||||
String embeddingId = UUID.randomUUID().toString();
|
||||
if (!skipEmbedding) {
|
||||
Document figureDoc = new Document(embeddingId, embeddingContent,
|
||||
buildFigureMetadata(figure, bookTitle, embeddingId, ""));
|
||||
vectorStore.add(List.of(figureDoc));
|
||||
figure.setCaptionEmbeddingId(UUID.fromString(embeddingId));
|
||||
}
|
||||
figureRepository.save(figure);
|
||||
}
|
||||
log.info("Embedded {} figure chunks for book {}", figures.size(), bookId);
|
||||
|
||||
// Step 6: Link text chunks to figures via in-text references
|
||||
for (SectionEntity section : sections) {
|
||||
List<Document> sectionChunks = allChunks.stream()
|
||||
.filter(d -> section.getId().equals(d.getMetadata().get("section_id")))
|
||||
.toList();
|
||||
List<FigureEntity> sectionFigures = figures.stream()
|
||||
.filter(f -> section.getId().equals(f.getSectionId()))
|
||||
.toList();
|
||||
chunkFigureRefService.linkChunksToFigures(sectionChunks, sectionFigures, section.getPageStart());
|
||||
}
|
||||
|
||||
book.setStatus(BookStatus.READY);
|
||||
book.setPageCount(pageCount);
|
||||
book.setProcessedAt(java.time.Instant.now());
|
||||
book.setPageCount(parsed.htmlByPage().size());
|
||||
book.setProcessedAt(Instant.now());
|
||||
bookRepository.save(book);
|
||||
|
||||
log.info("Finished embedding book {} — {} pages", bookId, pageCount);
|
||||
log.info("Finished embedding book {} — {} pages, {} figures",
|
||||
bookId, sections.size(), figures.size());
|
||||
|
||||
} catch (Exception ex) {
|
||||
log.error("Failed to embed book {}", bookId, ex);
|
||||
@@ -79,40 +201,114 @@ public class BookEmbeddingService {
|
||||
}
|
||||
}
|
||||
|
||||
private Document enrichDocument(Document doc, String bookId, String bookTitle) {
|
||||
String content = doc.getText();
|
||||
String chunkType = detectChunkType(content);
|
||||
|
||||
doc.getMetadata().put("book_id", bookId);
|
||||
doc.getMetadata().put("book_title", bookTitle);
|
||||
doc.getMetadata().put("chunk_type", chunkType);
|
||||
|
||||
return doc;
|
||||
}
|
||||
|
||||
private String detectChunkType(String content) {
|
||||
if (content != null) {
|
||||
for (String line : content.split("\\r?\\n")) {
|
||||
if (CAPTION_PATTERN.matcher(line.trim()).find()) {
|
||||
return "diagram";
|
||||
}
|
||||
}
|
||||
}
|
||||
return "text";
|
||||
}
|
||||
|
||||
@Transactional
|
||||
public void deleteBookChunks(UUID bookId) {
|
||||
log.info("Deleting vector chunks for book {}", bookId);
|
||||
log.info("Deleting all data for book {}", bookId);
|
||||
try {
|
||||
List<String> figureIds = figureRepository.findAllByBookId(bookId)
|
||||
.stream().map(FigureEntity::getId).toList();
|
||||
if (!figureIds.isEmpty()) {
|
||||
chunkFigureRefRepository.deleteByFigureIdIn(figureIds);
|
||||
}
|
||||
figureRepository.deleteAllByBookId(bookId);
|
||||
figureStorageService.deleteAll(bookId);
|
||||
markdownStorageService.deleteAll(bookId);
|
||||
sectionRepository.deleteAllByBookId(bookId);
|
||||
chapterRepository.deleteAllByBookId(bookId);
|
||||
|
||||
chunkMetadataRepository.deleteByBookId(bookId);
|
||||
|
||||
FilterExpressionBuilder b = new FilterExpressionBuilder();
|
||||
vectorStore.delete(b.eq("book_id", bookId.toString()).build());
|
||||
} catch (Exception ex) {
|
||||
log.warn("Could not delete vector chunks for book {}: {}", bookId, ex.getMessage());
|
||||
log.warn("Error during cleanup for book {}: {}", bookId, ex.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
private String truncate(String message, int maxLength) {
|
||||
if (message == null) return null;
|
||||
return message.length() <= maxLength ? message : message.substring(0, maxLength);
|
||||
// --- Private helpers ---
|
||||
|
||||
private List<SectionEntity> buildAndSaveSections(UUID bookId, String bookTitle,
|
||||
String chapterId,
|
||||
List<PageResult> pageResults) {
|
||||
List<SectionEntity> sections = new ArrayList<>();
|
||||
for (PageResult page : pageResults) {
|
||||
if (page.orderedText().isBlank()) continue;
|
||||
|
||||
String sectionId = bookId + "-p" + page.pageNumber();
|
||||
String title = truncate(page.headingTitle() != null ? page.headingTitle() : "Page " + page.pageNumber(), 500);
|
||||
|
||||
SectionEntity section = new SectionEntity(
|
||||
sectionId, chapterId, bookId,
|
||||
String.valueOf(page.pageNumber()),
|
||||
title,
|
||||
page.pageNumber(), page.pageNumber(),
|
||||
page.orderedText());
|
||||
sections.add(sectionRepository.save(section));
|
||||
}
|
||||
return sections;
|
||||
}
|
||||
|
||||
private void embedInBatches(List<Document> docs, UUID bookId) {
|
||||
int total = docs.size();
|
||||
for (int i = 0; i < total; i += embeddingBatchSize) {
|
||||
List<Document> batch = docs.subList(i, Math.min(i + embeddingBatchSize, total));
|
||||
vectorStore.add(batch);
|
||||
log.debug("Embedded batch {}/{} for book {}",
|
||||
i / embeddingBatchSize + 1, (total - 1) / embeddingBatchSize + 1, bookId);
|
||||
if (i + embeddingBatchSize < total) {
|
||||
try { Thread.sleep(embeddingBatchDelayMs); }
|
||||
catch (InterruptedException e) { Thread.currentThread().interrupt(); }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private Map<String, Object> buildFigureMetadata(FigureEntity figure, String bookTitle,
|
||||
String embeddingId, String imageText) {
|
||||
Map<String, Object> m = new HashMap<>();
|
||||
m.put("type", "FIGURE");
|
||||
m.put("book_id", figure.getBookId().toString());
|
||||
m.put("book_title", bookTitle);
|
||||
m.put("chapter_id", figure.getChapterId() != null ? figure.getChapterId() : "");
|
||||
m.put("section_id", figure.getSectionId() != null ? figure.getSectionId() : "");
|
||||
m.put("figure_id", figure.getId());
|
||||
m.put("figure_type", figure.getFigureType().name());
|
||||
m.put("image_path", figure.getImagePath());
|
||||
m.put("label", figure.getLabel() != null ? figure.getLabel() : "");
|
||||
m.put("page", figure.getPage());
|
||||
m.put("embedding_id", embeddingId);
|
||||
m.put("image_text", imageText); // verbatim text visible inside the image
|
||||
return m;
|
||||
}
|
||||
|
||||
/**
|
||||
* Replaces Marker's {@code src='{blockId}'} image attributes with resolved API URLs.
|
||||
* Block IDs look like {@code /page/0/Figure/2}.
|
||||
*/
|
||||
private String resolveImageSrcs(String html, UUID bookId, Map<String, String> blockIdToFigureId) {
|
||||
for (Map.Entry<String, String> entry : blockIdToFigureId.entrySet()) {
|
||||
String blockId = entry.getKey();
|
||||
String figureId = entry.getValue();
|
||||
String apiUrl = "/api/v1/figures/" + bookId + "/" + figureId + ".png";
|
||||
// Marker emits both single and double-quoted src attributes
|
||||
html = html.replace("src='" + blockId + "'", "src='" + apiUrl + "'");
|
||||
html = html.replace("src=\"" + blockId + "\"", "src=\"" + apiUrl + "\"");
|
||||
}
|
||||
return html;
|
||||
}
|
||||
|
||||
private String extractCaptionFromSection(SectionEntity section) {
|
||||
if (section == null) return null;
|
||||
for (String line : section.getFullText().split("\n")) {
|
||||
String trimmed = line.strip();
|
||||
if (trimmed.startsWith("Fig.") || trimmed.startsWith("Figure") || trimmed.startsWith("Algorithm")) {
|
||||
return trimmed;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
private String truncate(String msg, int max) {
|
||||
if (msg == null) return null;
|
||||
return msg.length() <= max ? msg : msg.substring(0, max);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,11 +1,13 @@
|
||||
package com.aiteacher.book;
|
||||
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.stereotype.Service;
|
||||
import org.springframework.web.multipart.MultipartFile;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.nio.file.Paths;
|
||||
import java.util.List;
|
||||
import java.util.NoSuchElementException;
|
||||
import java.util.UUID;
|
||||
@@ -15,10 +17,15 @@ public class BookService {
|
||||
|
||||
private final BookRepository bookRepository;
|
||||
private final BookEmbeddingService bookEmbeddingService;
|
||||
private final Path bookStoragePath;
|
||||
|
||||
public BookService(BookRepository bookRepository, BookEmbeddingService bookEmbeddingService) {
|
||||
public BookService(
|
||||
BookRepository bookRepository,
|
||||
BookEmbeddingService bookEmbeddingService,
|
||||
@Value("${app.figure-storage.base-path:./uploads}") String basePath) {
|
||||
this.bookRepository = bookRepository;
|
||||
this.bookEmbeddingService = bookEmbeddingService;
|
||||
this.bookStoragePath = Paths.get(basePath).toAbsolutePath().normalize().resolve("books");
|
||||
}
|
||||
|
||||
public Book upload(MultipartFile file) throws IOException {
|
||||
@@ -28,20 +35,35 @@ public class BookService {
|
||||
}
|
||||
|
||||
String title = deriveTitle(originalFilename);
|
||||
|
||||
Book book = new Book(title, originalFilename, file.getSize());
|
||||
book = bookRepository.save(book);
|
||||
|
||||
// Write to a temp file so the async task can read it
|
||||
Path tempFile = Files.createTempFile("aiteacher-", "-" + book.getId() + ".pdf");
|
||||
file.transferTo(tempFile.toFile());
|
||||
// Persist PDF in a stable location for potential re-embedding
|
||||
Files.createDirectories(bookStoragePath);
|
||||
Path pdfPath = bookStoragePath.resolve(book.getId() + ".pdf");
|
||||
file.transferTo(pdfPath.toFile());
|
||||
|
||||
UUID bookId = book.getId();
|
||||
Path pdfPath = tempFile;
|
||||
String bookTitle = title;
|
||||
bookEmbeddingService.embedBook(bookId, title, pdfPath);
|
||||
return book;
|
||||
}
|
||||
|
||||
bookEmbeddingService.embedBook(bookId, bookTitle, pdfPath);
|
||||
public Book reembed(UUID id) {
|
||||
Book book = bookRepository.findById(id)
|
||||
.orElseThrow(() -> new NoSuchElementException("Book not found."));
|
||||
|
||||
if (book.getStatus() == BookStatus.PROCESSING) {
|
||||
throw new IllegalStateException("Book is already being processed.");
|
||||
}
|
||||
|
||||
Path pdfPath = bookStoragePath.resolve(id + ".pdf");
|
||||
if (!Files.exists(pdfPath)) {
|
||||
throw new IllegalStateException(
|
||||
"Original PDF not found. Please re-upload the book before re-embedding.");
|
||||
}
|
||||
|
||||
bookEmbeddingService.deleteBookChunks(id);
|
||||
bookEmbeddingService.embedBook(id, book.getTitle(), pdfPath);
|
||||
return book;
|
||||
}
|
||||
|
||||
@@ -63,14 +85,21 @@ public class BookService {
|
||||
}
|
||||
|
||||
bookEmbeddingService.deleteBookChunks(id);
|
||||
|
||||
// Delete the stored PDF
|
||||
Path pdfPath = bookStoragePath.resolve(id + ".pdf");
|
||||
try {
|
||||
Files.deleteIfExists(pdfPath);
|
||||
} catch (IOException ex) {
|
||||
// Non-fatal — log only
|
||||
}
|
||||
|
||||
bookRepository.deleteById(id);
|
||||
}
|
||||
|
||||
private String deriveTitle(String filename) {
|
||||
// Strip .pdf extension and replace separators with spaces
|
||||
String name = filename.replaceAll("(?i)\\.pdf$", "");
|
||||
name = name.replaceAll("[-_]", " ");
|
||||
// Capitalise first letter
|
||||
if (!name.isEmpty()) {
|
||||
name = Character.toUpperCase(name.charAt(0)) + name.substring(1);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
package com.aiteacher.book;
|
||||
|
||||
public record FigureResponse(
|
||||
String figureId,
|
||||
String label,
|
||||
String caption,
|
||||
String figureType,
|
||||
int page,
|
||||
String imageUrl,
|
||||
String sectionId,
|
||||
String sectionTitle
|
||||
) {}
|
||||
@@ -3,28 +3,21 @@ package com.aiteacher.chat;
|
||||
import com.aiteacher.book.BookRepository;
|
||||
import com.aiteacher.book.BookStatus;
|
||||
import com.aiteacher.book.NoKnowledgeSourceException;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import com.aiteacher.document.FigureEntity;
|
||||
import com.aiteacher.document.SectionEntity;
|
||||
import com.aiteacher.retrieval.CitationValidatorService;
|
||||
import com.aiteacher.retrieval.LabelledContext;
|
||||
import com.aiteacher.retrieval.NeurosurgeryRetriever;
|
||||
import com.aiteacher.retrieval.QueryExpansionService;
|
||||
import com.aiteacher.retrieval.RetrievalResult;
|
||||
import org.springframework.ai.chat.client.ChatClient;
|
||||
import org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor;
|
||||
import org.springframework.ai.chat.model.ChatResponse;
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.vectorstore.SearchRequest;
|
||||
import org.springframework.ai.vectorstore.VectorStore;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.NoSuchElementException;
|
||||
import java.util.UUID;
|
||||
import java.util.*;
|
||||
|
||||
@Service
|
||||
public class ChatService {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(ChatService.class);
|
||||
|
||||
private static final String SYSTEM_PROMPT = """
|
||||
You are an expert neurosurgery educator assistant. Answer questions using the
|
||||
medical textbook content provided to you as context.
|
||||
@@ -35,26 +28,34 @@ public class ChatService {
|
||||
- Build answers from what is present: procedures, conditions, techniques, and descriptions all contribute; combine them into a rich, structured response
|
||||
- Use clear structure: headings, bullet points, or numbered steps where appropriate to maximize clarity
|
||||
- Only say you cannot answer if the context is entirely unrelated to the question
|
||||
- Cite sources for each major point (book title and page number from the context metadata)
|
||||
- Cite sources for each major claim using the reference labels from the context (e.g. [S1], [F2]). Prefer these labels over inventing page numbers, but you may also describe the source naturally if needed.
|
||||
- Figures (labeled [F1], [F2], etc.) are actual images and drawings from the textbook — they will be rendered as inline illustrations in your response. Use them actively to support your explanations: reference a figure when it visually demonstrates anatomy, a surgical step, or a clinical concept you are describing.
|
||||
- Maintain continuity with the conversation history
|
||||
- Never fabricate clinical information not present in the context
|
||||
""";
|
||||
|
||||
private final ChatClient chatClient;
|
||||
private final VectorStore vectorStore;
|
||||
private final BookRepository bookRepository;
|
||||
private final ChatSessionRepository sessionRepository;
|
||||
private final MessageRepository messageRepository;
|
||||
private final NeurosurgeryRetriever retriever;
|
||||
private final QueryExpansionService queryExpansionService;
|
||||
private final CitationValidatorService citationValidatorService;
|
||||
|
||||
public ChatService(ChatClient chatClient, VectorStore vectorStore,
|
||||
public ChatService(ChatClient chatClient,
|
||||
BookRepository bookRepository,
|
||||
ChatSessionRepository sessionRepository,
|
||||
MessageRepository messageRepository) {
|
||||
MessageRepository messageRepository,
|
||||
NeurosurgeryRetriever retriever,
|
||||
QueryExpansionService queryExpansionService,
|
||||
CitationValidatorService citationValidatorService) {
|
||||
this.chatClient = chatClient;
|
||||
this.vectorStore = vectorStore;
|
||||
this.bookRepository = bookRepository;
|
||||
this.sessionRepository = sessionRepository;
|
||||
this.messageRepository = messageRepository;
|
||||
this.retriever = retriever;
|
||||
this.queryExpansionService = queryExpansionService;
|
||||
this.citationValidatorService = citationValidatorService;
|
||||
}
|
||||
|
||||
public ChatSession createSession(String topicId) {
|
||||
@@ -73,7 +74,11 @@ public class ChatService {
|
||||
ChatSession session = sessionRepository.findById(sessionId)
|
||||
.orElseThrow(() -> new NoSuchElementException("Session not found."));
|
||||
|
||||
if (!bookRepository.existsByStatus(BookStatus.READY)) {
|
||||
List<com.aiteacher.book.Book> readyBooks = bookRepository.findAll().stream()
|
||||
.filter(b -> b.getStatus() == BookStatus.READY)
|
||||
.toList();
|
||||
|
||||
if (readyBooks.isEmpty()) {
|
||||
throw new NoKnowledgeSourceException("No books are available as knowledge sources.");
|
||||
}
|
||||
|
||||
@@ -81,27 +86,40 @@ public class ChatService {
|
||||
Message userMessage = new Message(sessionId, MessageRole.USER, userContent);
|
||||
messageRepository.save(userMessage);
|
||||
|
||||
// Build conversation history for context
|
||||
// Build full question with conversation history
|
||||
List<Message> history = messageRepository.findBySessionIdOrderByCreatedAtAsc(sessionId);
|
||||
|
||||
// Build the prompt with full conversation history as context
|
||||
String fullQuestion = buildQuestionWithHistory(history, userContent, session.getTopicId());
|
||||
|
||||
var qaAdvisor = QuestionAnswerAdvisor.builder(vectorStore)
|
||||
.searchRequest(SearchRequest.builder().similarityThreshold(0.5d).topK(6).build())
|
||||
.build();
|
||||
|
||||
ChatResponse response = chatClient.prompt()
|
||||
.advisors(qaAdvisor)
|
||||
// Expand only the current user question to clinical terminology for retrieval (US1).
|
||||
// fullQuestion (which includes conversation history) is used for the LLM context prompt,
|
||||
// but retrieval should be driven by a concise clinical rewrite of the actual question.
|
||||
String retrievalQuery = queryExpansionService.expand(userContent).rewritten();
|
||||
|
||||
// Retrieve context from all ready books using the expanded query
|
||||
List<SectionEntity> allSections = new ArrayList<>();
|
||||
List<FigureEntity> allFigures = new ArrayList<>();
|
||||
for (com.aiteacher.book.Book book : readyBooks) {
|
||||
RetrievalResult result = retriever.retrieve(retrievalQuery, book.getId());
|
||||
allSections.addAll(result.parentSections());
|
||||
allFigures.addAll(result.figures());
|
||||
}
|
||||
|
||||
// Build labelled context prompt (US2): assigns [S1]/[F1] labels to each source
|
||||
LabelledContext ctx = buildContextPrompt(fullQuestion, allSections, allFigures);
|
||||
|
||||
// Generate answer
|
||||
String rawContent = chatClient.prompt()
|
||||
.system(SYSTEM_PROMPT)
|
||||
.user(fullQuestion)
|
||||
.user(ctx.promptText())
|
||||
.call()
|
||||
.chatResponse();
|
||||
.content();
|
||||
|
||||
String assistantContent = response.getResult().getOutput().getText();
|
||||
List<Map<String, Object>> sources = extractSources(response);
|
||||
// Strip any citation labels not present in the retrieved context (US2)
|
||||
String assistantContent = citationValidatorService.validate(rawContent, ctx.allLabels());
|
||||
|
||||
// Attach sources with their ref-labels for frontend traceability
|
||||
List<Map<String, Object>> sources = buildSources(allSections, allFigures);
|
||||
|
||||
// Persist assistant message
|
||||
Message assistantMessage = new Message(sessionId, MessageRole.ASSISTANT, assistantContent);
|
||||
assistantMessage.setSources(sources);
|
||||
return messageRepository.save(assistantMessage);
|
||||
@@ -118,24 +136,114 @@ public class ChatService {
|
||||
sessionRepository.deleteById(sessionId);
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
// Private helpers
|
||||
// -------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Builds the LLM context prompt, tagging each section as [S1], [S2]… and
|
||||
* each figure as [F1], [F2]… so the model can cite only known sources.
|
||||
*/
|
||||
private LabelledContext buildContextPrompt(String question,
|
||||
List<SectionEntity> sections,
|
||||
List<FigureEntity> figures) {
|
||||
Map<String, SectionEntity> sectionLabels = new LinkedHashMap<>();
|
||||
Map<String, FigureEntity> figureLabels = new LinkedHashMap<>();
|
||||
StringBuilder sb = new StringBuilder();
|
||||
|
||||
if (!sections.isEmpty()) {
|
||||
sb.append("CONTEXT:\n\n");
|
||||
for (int i = 0; i < sections.size(); i++) {
|
||||
SectionEntity section = sections.get(i);
|
||||
String label = "S" + (i + 1);
|
||||
sectionLabels.put(label, section);
|
||||
sb.append("[").append(label).append("] ")
|
||||
.append(section.getTitle())
|
||||
.append(", p.").append(section.getPageStart()).append("\n");
|
||||
sb.append(section.getFullText()).append("\n\n");
|
||||
}
|
||||
}
|
||||
|
||||
if (!figures.isEmpty()) {
|
||||
sb.append("AVAILABLE FIGURES:\n");
|
||||
for (int i = 0; i < figures.size(); i++) {
|
||||
FigureEntity figure = figures.get(i);
|
||||
String label = "F" + (i + 1);
|
||||
figureLabels.put(label, figure);
|
||||
sb.append("[").append(label).append("] ")
|
||||
.append(figure.getLabel() != null ? figure.getLabel() : "Figure")
|
||||
.append(" (p.").append(figure.getPage()).append("): ")
|
||||
.append(figure.getCaption() != null ? figure.getCaption() : "")
|
||||
.append("\n");
|
||||
}
|
||||
sb.append("\nWhen referencing diagrams, use their label from the context (e.g. [F1]).\n\n");
|
||||
}
|
||||
|
||||
sb.append("QUESTION:\n").append(question);
|
||||
return new LabelledContext(sectionLabels, figureLabels, sb.toString());
|
||||
}
|
||||
|
||||
private List<Map<String, Object>> buildSources(List<SectionEntity> sections,
|
||||
List<FigureEntity> figures) {
|
||||
List<Map<String, Object>> sources = new ArrayList<>();
|
||||
|
||||
for (int i = 0; i < sections.size(); i++) {
|
||||
SectionEntity section = sections.get(i);
|
||||
Map<String, Object> source = new LinkedHashMap<>();
|
||||
source.put("type", "TEXT");
|
||||
source.put("refLabel", "S" + (i + 1));
|
||||
source.put("bookId", section.getBookId());
|
||||
source.put("bookTitle", deriveTitleFromSection(section));
|
||||
source.put("page", section.getPageStart());
|
||||
source.put("chunkText", truncate(section.getFullText(), 500));
|
||||
sources.add(source);
|
||||
}
|
||||
|
||||
for (int i = 0; i < figures.size(); i++) {
|
||||
FigureEntity figure = figures.get(i);
|
||||
Map<String, Object> source = new LinkedHashMap<>();
|
||||
source.put("type", "FIGURE");
|
||||
source.put("refLabel", "F" + (i + 1));
|
||||
source.put("bookId", figure.getBookId());
|
||||
source.put("bookTitle", bookRepository.findById(figure.getBookId())
|
||||
.map(com.aiteacher.book.Book::getTitle).orElse("Book"));
|
||||
source.put("page", figure.getPage());
|
||||
source.put("figureId", figure.getId());
|
||||
source.put("label", figure.getLabel() != null ? figure.getLabel() : "");
|
||||
source.put("caption", figure.getCaption() != null ? figure.getCaption() : "");
|
||||
source.put("figureType", figure.getFigureType().name());
|
||||
String filename = figure.getImagePath().substring(
|
||||
figure.getImagePath().lastIndexOf('/') + 1);
|
||||
source.put("imageUrl", "/api/v1/figures/" + figure.getBookId() + "/" + filename);
|
||||
sources.add(source);
|
||||
}
|
||||
|
||||
return sources;
|
||||
}
|
||||
|
||||
private String deriveTitleFromSection(SectionEntity section) {
|
||||
if (section == null) return "Book";
|
||||
return bookRepository.findById(section.getBookId())
|
||||
.map(com.aiteacher.book.Book::getTitle)
|
||||
.orElse("Book");
|
||||
}
|
||||
|
||||
private String buildQuestionWithHistory(List<Message> history, String currentQuestion,
|
||||
String topicId) {
|
||||
boolean hasTopic = topicId != null && !topicId.equals("free-form");
|
||||
|
||||
if (history.size() <= 1) {
|
||||
return hasTopic
|
||||
? String.format("[Context: This is a question about the neurosurgery topic '%s']\n%s",
|
||||
? String.format("[Context: question about neurosurgery topic '%s']\n%s",
|
||||
topicId, currentQuestion)
|
||||
: currentQuestion;
|
||||
}
|
||||
|
||||
StringBuilder sb = new StringBuilder();
|
||||
if (hasTopic) {
|
||||
sb.append(String.format("[Context: This conversation is about the neurosurgery topic '%s']\n\n",
|
||||
topicId));
|
||||
sb.append(String.format("[Context: conversation about '%s']\n\n", topicId));
|
||||
}
|
||||
sb.append("Previous conversation:\n");
|
||||
// Include all messages except the last (which is the current user message just saved)
|
||||
for (int i = 0; i < history.size() - 1; i++) {
|
||||
Message msg = history.get(i);
|
||||
sb.append(msg.getRole().name()).append(": ").append(msg.getContent()).append("\n");
|
||||
@@ -144,30 +252,8 @@ public class ChatService {
|
||||
return sb.toString();
|
||||
}
|
||||
|
||||
private List<Map<String, Object>> extractSources(ChatResponse response) {
|
||||
List<Map<String, Object>> sources = new ArrayList<>();
|
||||
|
||||
if (response.getMetadata() != null) {
|
||||
Object retrieved = response.getMetadata().get(QuestionAnswerAdvisor.RETRIEVED_DOCUMENTS);
|
||||
if (retrieved instanceof List<?> docs) {
|
||||
for (Object docObj : docs) {
|
||||
if (docObj instanceof Document doc) {
|
||||
Map<String, Object> metadata = doc.getMetadata();
|
||||
String bookTitle = (String) metadata.get("book_title");
|
||||
Object pageObj = metadata.get("page_number");
|
||||
Integer page = pageObj instanceof Number n ? n.intValue() : null;
|
||||
if (bookTitle != null) {
|
||||
Map<String, Object> source = new HashMap<>();
|
||||
source.put("bookTitle", bookTitle);
|
||||
source.put("page", page);
|
||||
source.put("chunkText", doc.getText());
|
||||
sources.add(source);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return sources;
|
||||
private String truncate(String text, int maxChars) {
|
||||
if (text == null) return "";
|
||||
return text.length() <= maxChars ? text : text.substring(0, maxChars) + "…";
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
package com.aiteacher.concept;
|
||||
|
||||
import com.aiteacher.topic.Topic;
|
||||
import com.aiteacher.topic.TopicRepository;
|
||||
import org.springframework.http.ResponseEntity;
|
||||
import org.springframework.web.bind.annotation.*;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.NoSuchElementException;
|
||||
import java.util.UUID;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
@RestController
|
||||
@RequestMapping("/api/v1/topics/{id}/concept-reports")
|
||||
public class ConceptReportController {
|
||||
|
||||
private final TopicRepository topicRepository;
|
||||
private final ConceptReportService conceptReportService;
|
||||
|
||||
public ConceptReportController(TopicRepository topicRepository,
|
||||
ConceptReportService conceptReportService) {
|
||||
this.topicRepository = topicRepository;
|
||||
this.conceptReportService = conceptReportService;
|
||||
}
|
||||
|
||||
@PostMapping
|
||||
public ResponseEntity<ConceptReportResponse> generate(
|
||||
@PathVariable String id,
|
||||
@RequestParam(defaultValue = "en") String language) {
|
||||
Topic topic = topicRepository.findById(id)
|
||||
.orElseThrow(() -> new NoSuchElementException("Topic not found."));
|
||||
return ResponseEntity.ok(conceptReportService.generateReport(topic, language));
|
||||
}
|
||||
|
||||
@GetMapping
|
||||
public ResponseEntity<List<SavedConceptReportItem>> list(@PathVariable String id) {
|
||||
topicRepository.findById(id)
|
||||
.orElseThrow(() -> new NoSuchElementException("Topic not found."));
|
||||
return ResponseEntity.ok(conceptReportService.listReports(id));
|
||||
}
|
||||
|
||||
@GetMapping("/{reportId}")
|
||||
public ResponseEntity<ConceptReportResponse> get(@PathVariable String id,
|
||||
@PathVariable UUID reportId) {
|
||||
topicRepository.findById(id)
|
||||
.orElseThrow(() -> new NoSuchElementException("Topic not found."));
|
||||
Map<String, String> topicNames = topicRepository.findAll().stream()
|
||||
.collect(Collectors.toMap(Topic::getId, Topic::getName, (a, b) -> a));
|
||||
return ResponseEntity.ok(conceptReportService.getReport(reportId, topicNames));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
package com.aiteacher.concept;
|
||||
|
||||
import jakarta.persistence.*;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.UUID;
|
||||
|
||||
@Entity
|
||||
@Table(name = "concept_report")
|
||||
public class ConceptReportEntity {
|
||||
|
||||
@Id
|
||||
@GeneratedValue(strategy = GenerationType.UUID)
|
||||
private UUID id;
|
||||
|
||||
@Column(name = "topic_id", nullable = false, length = 100)
|
||||
private String topicId;
|
||||
|
||||
@Column(name = "report_number", nullable = false)
|
||||
private int reportNumber;
|
||||
|
||||
@Column(name = "facets_json", nullable = false, columnDefinition = "TEXT")
|
||||
private String facetsJson;
|
||||
|
||||
@Column(name = "sources_json", nullable = false, columnDefinition = "TEXT")
|
||||
private String sourcesJson;
|
||||
|
||||
@Column(name = "generated_at", nullable = false)
|
||||
private Instant generatedAt;
|
||||
|
||||
protected ConceptReportEntity() {}
|
||||
|
||||
public ConceptReportEntity(String topicId, int reportNumber, String facetsJson,
|
||||
String sourcesJson, Instant generatedAt) {
|
||||
this.topicId = topicId;
|
||||
this.reportNumber = reportNumber;
|
||||
this.facetsJson = facetsJson;
|
||||
this.sourcesJson = sourcesJson;
|
||||
this.generatedAt = generatedAt;
|
||||
}
|
||||
|
||||
public UUID getId() { return id; }
|
||||
public String getTopicId() { return topicId; }
|
||||
public int getReportNumber() { return reportNumber; }
|
||||
public String getFacetsJson() { return facetsJson; }
|
||||
public String getSourcesJson() { return sourcesJson; }
|
||||
public Instant getGeneratedAt() { return generatedAt; }
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
package com.aiteacher.concept;
|
||||
|
||||
import org.springframework.data.jpa.repository.JpaRepository;
|
||||
import org.springframework.stereotype.Repository;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
|
||||
@Repository
|
||||
public interface ConceptReportRepository extends JpaRepository<ConceptReportEntity, UUID> {
|
||||
long countByTopicId(String topicId);
|
||||
List<ConceptReportEntity> findByTopicIdOrderByReportNumberAsc(String topicId);
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
package com.aiteacher.concept;
|
||||
|
||||
import com.aiteacher.topic.TopicSummaryResponse.SourceReference;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
|
||||
public record ConceptReportResponse(
|
||||
UUID id,
|
||||
int reportNumber,
|
||||
String topicId,
|
||||
String topicName,
|
||||
List<FacetSection> facets,
|
||||
List<SourceReference> sources,
|
||||
Instant generatedAt
|
||||
) {
|
||||
public record FacetSection(
|
||||
String facetKey,
|
||||
String title,
|
||||
String markdown,
|
||||
List<String> refLabels
|
||||
) {}
|
||||
}
|
||||
@@ -0,0 +1,299 @@
|
||||
package com.aiteacher.concept;
|
||||
|
||||
import com.aiteacher.book.Book;
|
||||
import com.aiteacher.book.BookRepository;
|
||||
import com.aiteacher.book.BookStatus;
|
||||
import com.aiteacher.book.NoKnowledgeSourceException;
|
||||
import com.aiteacher.document.FigureEntity;
|
||||
import com.aiteacher.document.SectionEntity;
|
||||
import com.aiteacher.enrichment.ConceptFacet;
|
||||
import com.aiteacher.topic.Topic;
|
||||
import com.aiteacher.topic.TopicSummaryResponse.SourceReference;
|
||||
import com.fasterxml.jackson.core.JsonProcessingException;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.ai.chat.client.ChatClient;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.*;
|
||||
|
||||
@Service
|
||||
public class ConceptReportService {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(ConceptReportService.class);
|
||||
|
||||
private static final String SYSTEM_PROMPT = """
|
||||
You are an expert neurosurgery educator. You write focused, facet-specific sections of
|
||||
a structured concept report for highly experienced neurosurgeons. The audience wants
|
||||
concise, clinically relevant teaching.
|
||||
|
||||
When writing a facet section:
|
||||
- Stick strictly to the facet you are asked about (e.g. definition, complications).
|
||||
- Cite claims using ONLY the reference labels provided in the context.
|
||||
Do not invent page numbers, section titles, or labels not present in CONTEXT.
|
||||
- Citation format: each citation must be a SINGLE label per bracket — write `[S1], [S2]` or
|
||||
`[S3] [F2]`. NEVER combine labels inside one bracket (no `[S1 S2]`, `[S1, S2]`, `[S1 2]`).
|
||||
- Figures ([F#]) are actual images that will be rendered inline — reference them when they
|
||||
visually support your explanation.
|
||||
- If CONTEXT is insufficient for the requested facet, write exactly:
|
||||
"The uploaded books do not contain sufficient information on this aspect."
|
||||
- Never hallucinate clinical information outside the provided context.
|
||||
""";
|
||||
|
||||
private final ChatClient chatClient;
|
||||
private final BookRepository bookRepository;
|
||||
private final ConceptRetriever conceptRetriever;
|
||||
private final ConceptReportRepository reportRepository;
|
||||
private final ObjectMapper objectMapper;
|
||||
|
||||
public ConceptReportService(ChatClient chatClient,
|
||||
BookRepository bookRepository,
|
||||
ConceptRetriever conceptRetriever,
|
||||
ConceptReportRepository reportRepository,
|
||||
ObjectMapper objectMapper) {
|
||||
this.chatClient = chatClient;
|
||||
this.bookRepository = bookRepository;
|
||||
this.conceptRetriever = conceptRetriever;
|
||||
this.reportRepository = reportRepository;
|
||||
this.objectMapper = objectMapper;
|
||||
}
|
||||
|
||||
public ConceptReportResponse generateReport(Topic topic, String language) {
|
||||
List<Book> readyBooks = bookRepository.findAll().stream()
|
||||
.filter(b -> b.getStatus() == BookStatus.READY)
|
||||
.toList();
|
||||
|
||||
if (readyBooks.isEmpty()) {
|
||||
throw new NoKnowledgeSourceException(
|
||||
"No books are available as knowledge sources. Please upload and process at least one book.");
|
||||
}
|
||||
|
||||
Map<ConceptFacet, MergedFacet> merged = new EnumMap<>(ConceptFacet.class);
|
||||
for (Book book : readyBooks) {
|
||||
ConceptRetrievalResult result = conceptRetriever.retrieveByConcept(topic.getName(), book.getId());
|
||||
result.byFacet().forEach((facet, bundle) -> merged
|
||||
.computeIfAbsent(facet, k -> new MergedFacet())
|
||||
.add(bundle));
|
||||
}
|
||||
|
||||
// Global, deduplicated sources across all facets
|
||||
List<SectionEntity> globalSections = new ArrayList<>();
|
||||
Set<String> seenSections = new LinkedHashSet<>();
|
||||
List<FigureEntity> globalFigures = new ArrayList<>();
|
||||
Set<String> seenFigures = new LinkedHashSet<>();
|
||||
|
||||
for (MergedFacet mf : merged.values()) {
|
||||
for (SectionEntity s : mf.sections) if (seenSections.add(s.getId())) globalSections.add(s);
|
||||
for (FigureEntity f : mf.figures) if (seenFigures.add(f.getId())) globalFigures.add(f);
|
||||
}
|
||||
|
||||
// Global label maps: section id -> "S#", figure id -> "F#"
|
||||
Map<String, String> sectionLabel = new HashMap<>();
|
||||
for (int i = 0; i < globalSections.size(); i++) {
|
||||
sectionLabel.put(globalSections.get(i).getId(), "S" + (i + 1));
|
||||
}
|
||||
Map<String, String> figureLabel = new HashMap<>();
|
||||
for (int i = 0; i < globalFigures.size(); i++) {
|
||||
figureLabel.put(globalFigures.get(i).getId(), "F" + (i + 1));
|
||||
}
|
||||
|
||||
List<ConceptReportResponse.FacetSection> facetSections = new ArrayList<>();
|
||||
// Preserve enum declaration order for consistent UI rendering
|
||||
for (ConceptFacet facet : ConceptFacet.values()) {
|
||||
MergedFacet mf = merged.get(facet);
|
||||
if (mf == null || mf.isEmpty()) continue;
|
||||
if (facet == ConceptFacet.OTHER) continue; // skip OTHER bucket in the rendered report
|
||||
|
||||
String prompt = buildFacetPrompt(topic, facet, mf, sectionLabel, figureLabel, language);
|
||||
String markdown = chatClient.prompt()
|
||||
.system(SYSTEM_PROMPT)
|
||||
.user(prompt)
|
||||
.call()
|
||||
.content();
|
||||
|
||||
List<String> refs = collectRefs(mf, sectionLabel, figureLabel);
|
||||
facetSections.add(new ConceptReportResponse.FacetSection(
|
||||
facet.name(), facet.displayTitle(), markdown != null ? markdown : "", refs));
|
||||
}
|
||||
|
||||
List<SourceReference> sources = buildSources(globalSections, globalFigures, readyBooks);
|
||||
Instant generatedAt = Instant.now();
|
||||
|
||||
int reportNumber = (int) reportRepository.countByTopicId(topic.getId()) + 1;
|
||||
ConceptReportEntity entity = new ConceptReportEntity(
|
||||
topic.getId(), reportNumber,
|
||||
serialize(facetSections), serialize(sources), generatedAt);
|
||||
entity = reportRepository.save(entity);
|
||||
|
||||
return new ConceptReportResponse(
|
||||
entity.getId(), reportNumber, topic.getId(), topic.getName(),
|
||||
facetSections, sources, generatedAt);
|
||||
}
|
||||
|
||||
public List<SavedConceptReportItem> listReports(String topicId) {
|
||||
return reportRepository.findByTopicIdOrderByReportNumberAsc(topicId).stream()
|
||||
.map(e -> new SavedConceptReportItem(e.getId(), e.getReportNumber(), e.getGeneratedAt()))
|
||||
.toList();
|
||||
}
|
||||
|
||||
public ConceptReportResponse getReport(UUID reportId, Map<String, String> topicNamesById) {
|
||||
ConceptReportEntity entity = reportRepository.findById(reportId)
|
||||
.orElseThrow(() -> new NoSuchElementException("Concept report not found."));
|
||||
List<ConceptReportResponse.FacetSection> facets = deserializeFacets(entity.getFacetsJson());
|
||||
List<SourceReference> sources = deserializeSources(entity.getSourcesJson());
|
||||
String topicName = topicNamesById.getOrDefault(entity.getTopicId(), entity.getTopicId());
|
||||
return new ConceptReportResponse(
|
||||
entity.getId(), entity.getReportNumber(), entity.getTopicId(), topicName,
|
||||
facets, sources, entity.getGeneratedAt());
|
||||
}
|
||||
|
||||
private String buildFacetPrompt(Topic topic, ConceptFacet facet, MergedFacet mf,
|
||||
Map<String, String> sectionLabel,
|
||||
Map<String, String> figureLabel,
|
||||
String language) {
|
||||
StringBuilder sb = new StringBuilder();
|
||||
sb.append("CONCEPT: ").append(topic.getName()).append("\n");
|
||||
sb.append("FACET: ").append(facet.displayTitle()).append("\n\n");
|
||||
|
||||
sb.append("CONTEXT:\n\n");
|
||||
for (SectionEntity s : mf.sections) {
|
||||
String label = sectionLabel.get(s.getId());
|
||||
sb.append("[").append(label).append("] ")
|
||||
.append(s.getTitle() != null ? s.getTitle() : "")
|
||||
.append(", p.").append(s.getPageStart()).append("\n");
|
||||
sb.append(s.getFullText()).append("\n\n");
|
||||
}
|
||||
|
||||
if (!mf.figures.isEmpty()) {
|
||||
sb.append("AVAILABLE FIGURES:\n");
|
||||
for (FigureEntity f : mf.figures) {
|
||||
String label = figureLabel.get(f.getId());
|
||||
sb.append("[").append(label).append("] ")
|
||||
.append(f.getLabel() != null ? f.getLabel() : "Figure")
|
||||
.append(" (p.").append(f.getPage()).append("): ")
|
||||
.append(f.getCaption() != null ? f.getCaption() : "")
|
||||
.append("\n");
|
||||
}
|
||||
sb.append("\n");
|
||||
}
|
||||
|
||||
sb.append("Write the ").append(facet.displayTitle()).append(" section of a concept report on \"")
|
||||
.append(topic.getName())
|
||||
.append("\". Stay strictly within this facet. Use the [S#]/[F#] labels above for citations.");
|
||||
|
||||
if ("th".equalsIgnoreCase(language)) {
|
||||
sb.append("\n\nIMPORTANT: Write the narrative in Thai. ")
|
||||
.append("Keep all medical, anatomical, surgical, pharmacological, and clinical ")
|
||||
.append("terminology in English (e.g., cerebellopontine angle, glioblastoma, craniotomy, ")
|
||||
.append("dexamethasone). Do NOT translate disease names, anatomical structures, drug names, ")
|
||||
.append("procedures, eponyms, or imaging modalities. Translate only connective prose, ")
|
||||
.append("explanations, and general descriptions. Citation labels [S#]/[F#] stay unchanged. ")
|
||||
.append("The sentinel string for insufficient context must remain exactly: ")
|
||||
.append("\"The uploaded books do not contain sufficient information on this aspect.\"");
|
||||
}
|
||||
return sb.toString();
|
||||
}
|
||||
|
||||
private List<String> collectRefs(MergedFacet mf,
|
||||
Map<String, String> sectionLabel,
|
||||
Map<String, String> figureLabel) {
|
||||
List<String> refs = new ArrayList<>();
|
||||
for (SectionEntity s : mf.sections) {
|
||||
String l = sectionLabel.get(s.getId());
|
||||
if (l != null) refs.add(l);
|
||||
}
|
||||
for (FigureEntity f : mf.figures) {
|
||||
String l = figureLabel.get(f.getId());
|
||||
if (l != null) refs.add(l);
|
||||
}
|
||||
return refs;
|
||||
}
|
||||
|
||||
private List<SourceReference> buildSources(List<SectionEntity> sections,
|
||||
List<FigureEntity> figures,
|
||||
List<Book> readyBooks) {
|
||||
List<SourceReference> sources = new ArrayList<>();
|
||||
for (int i = 0; i < sections.size(); i++) {
|
||||
SectionEntity s = sections.get(i);
|
||||
Book book = findBook(readyBooks, s.getBookId());
|
||||
String title = book != null ? book.getTitle() : "Book";
|
||||
String bookId = book != null ? book.getId().toString() : null;
|
||||
sources.add(new SourceReference(
|
||||
"TEXT", "S" + (i + 1), bookId, title, s.getPageStart(),
|
||||
truncate(s.getFullText(), 500), null, null, null, null, null));
|
||||
}
|
||||
for (int i = 0; i < figures.size(); i++) {
|
||||
FigureEntity f = figures.get(i);
|
||||
Book book = findBook(readyBooks, f.getBookId());
|
||||
String title = book != null ? book.getTitle() : "Book";
|
||||
String bookId = book != null ? book.getId().toString() : null;
|
||||
String filename = f.getImagePath().substring(f.getImagePath().lastIndexOf('/') + 1);
|
||||
String imageUrl = "/api/v1/figures/" + f.getBookId() + "/" + filename;
|
||||
sources.add(new SourceReference(
|
||||
"FIGURE", "F" + (i + 1), bookId, title, f.getPage(),
|
||||
null, f.getId(), f.getLabel(), f.getCaption(),
|
||||
f.getFigureType().name(), imageUrl));
|
||||
}
|
||||
return sources;
|
||||
}
|
||||
|
||||
private Book findBook(List<Book> books, UUID bookId) {
|
||||
return books.stream().filter(b -> b.getId().equals(bookId)).findFirst().orElse(null);
|
||||
}
|
||||
|
||||
private String serialize(Object value) {
|
||||
try {
|
||||
return objectMapper.writeValueAsString(value);
|
||||
} catch (JsonProcessingException e) {
|
||||
log.warn("Failed to serialize concept report field", e);
|
||||
return "[]";
|
||||
}
|
||||
}
|
||||
|
||||
private List<ConceptReportResponse.FacetSection> deserializeFacets(String json) {
|
||||
try {
|
||||
return objectMapper.readValue(json,
|
||||
objectMapper.getTypeFactory().constructCollectionType(
|
||||
List.class, ConceptReportResponse.FacetSection.class));
|
||||
} catch (JsonProcessingException e) {
|
||||
log.warn("Failed to deserialize facets", e);
|
||||
return List.of();
|
||||
}
|
||||
}
|
||||
|
||||
private List<SourceReference> deserializeSources(String json) {
|
||||
try {
|
||||
return objectMapper.readValue(json,
|
||||
objectMapper.getTypeFactory().constructCollectionType(
|
||||
List.class, SourceReference.class));
|
||||
} catch (JsonProcessingException e) {
|
||||
log.warn("Failed to deserialize sources", e);
|
||||
return List.of();
|
||||
}
|
||||
}
|
||||
|
||||
private String truncate(String text, int maxChars) {
|
||||
if (text == null) return "";
|
||||
return text.length() <= maxChars ? text : text.substring(0, maxChars) + "…";
|
||||
}
|
||||
|
||||
private static class MergedFacet {
|
||||
final List<SectionEntity> sections = new ArrayList<>();
|
||||
final List<FigureEntity> figures = new ArrayList<>();
|
||||
final Set<String> sectionIds = new HashSet<>();
|
||||
final Set<String> figureIds = new HashSet<>();
|
||||
|
||||
void add(FacetBundle bundle) {
|
||||
for (SectionEntity s : bundle.sections()) {
|
||||
if (sectionIds.add(s.getId())) sections.add(s);
|
||||
}
|
||||
for (FigureEntity f : bundle.figures()) {
|
||||
if (figureIds.add(f.getId())) figures.add(f);
|
||||
}
|
||||
}
|
||||
|
||||
boolean isEmpty() { return sections.isEmpty() && figures.isEmpty(); }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
package com.aiteacher.concept;
|
||||
|
||||
import com.aiteacher.enrichment.ConceptFacet;
|
||||
|
||||
import java.util.Map;
|
||||
|
||||
public record ConceptRetrievalResult(
|
||||
Map<ConceptFacet, FacetBundle> byFacet,
|
||||
boolean usedFallback
|
||||
) {}
|
||||
@@ -0,0 +1,163 @@
|
||||
package com.aiteacher.concept;
|
||||
|
||||
import com.aiteacher.document.*;
|
||||
import com.aiteacher.enrichment.ChunkMetadataEntity;
|
||||
import com.aiteacher.enrichment.ChunkMetadataRepository;
|
||||
import com.aiteacher.enrichment.ConceptFacet;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.vectorstore.SearchRequest;
|
||||
import org.springframework.ai.vectorstore.VectorStore;
|
||||
import org.springframework.ai.vectorstore.filter.FilterExpressionBuilder;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.util.*;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
@Service
|
||||
public class ConceptRetriever {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(ConceptRetriever.class);
|
||||
|
||||
private static final int FALLBACK_TOP_K = 30;
|
||||
private static final int FIGURE_TOP_K = 6;
|
||||
|
||||
private final ChunkMetadataRepository metadataRepository;
|
||||
private final VectorStore vectorStore;
|
||||
private final SectionRepository sectionRepository;
|
||||
private final FigureRepository figureRepository;
|
||||
private final ChunkFigureRefRepository chunkFigureRefRepository;
|
||||
|
||||
public ConceptRetriever(ChunkMetadataRepository metadataRepository,
|
||||
VectorStore vectorStore,
|
||||
SectionRepository sectionRepository,
|
||||
FigureRepository figureRepository,
|
||||
ChunkFigureRefRepository chunkFigureRefRepository) {
|
||||
this.metadataRepository = metadataRepository;
|
||||
this.vectorStore = vectorStore;
|
||||
this.sectionRepository = sectionRepository;
|
||||
this.figureRepository = figureRepository;
|
||||
this.chunkFigureRefRepository = chunkFigureRefRepository;
|
||||
}
|
||||
|
||||
public ConceptRetrievalResult retrieveByConcept(String conceptKeyword, UUID bookId) {
|
||||
String canonical = canonicalise(conceptKeyword);
|
||||
|
||||
List<ChunkMetadataEntity> hits = metadataRepository
|
||||
.findByBookIdAndEntityContains(bookId, canonical);
|
||||
boolean fallback = false;
|
||||
|
||||
if (hits.isEmpty()) {
|
||||
log.debug("Entity match miss for '{}' in book {} — falling back to vector search", canonical, bookId);
|
||||
fallback = true;
|
||||
hits = vectorFallback(conceptKeyword, bookId);
|
||||
}
|
||||
|
||||
if (hits.isEmpty()) {
|
||||
return new ConceptRetrievalResult(Map.of(), fallback);
|
||||
}
|
||||
|
||||
List<FigureEntity> semanticFigures = semanticFigureSearch(conceptKeyword, bookId);
|
||||
|
||||
Map<ConceptFacet, List<ChunkMetadataEntity>> grouped = hits.stream()
|
||||
.collect(Collectors.groupingBy(
|
||||
ChunkMetadataEntity::getFacet,
|
||||
LinkedHashMap::new,
|
||||
Collectors.toList()));
|
||||
|
||||
Map<ConceptFacet, FacetBundle> result = new LinkedHashMap<>();
|
||||
for (Map.Entry<ConceptFacet, List<ChunkMetadataEntity>> entry : grouped.entrySet()) {
|
||||
result.put(entry.getKey(), hydrate(entry.getValue(), semanticFigures));
|
||||
}
|
||||
return new ConceptRetrievalResult(result, fallback);
|
||||
}
|
||||
|
||||
private List<ChunkMetadataEntity> vectorFallback(String query, UUID bookId) {
|
||||
FilterExpressionBuilder b = new FilterExpressionBuilder();
|
||||
List<Document> textHits = vectorStore.similaritySearch(
|
||||
SearchRequest.builder()
|
||||
.query(query)
|
||||
.topK(FALLBACK_TOP_K)
|
||||
.filterExpression(b.and(
|
||||
b.eq("type", "TEXT"),
|
||||
b.eq("book_id", bookId.toString())
|
||||
).build())
|
||||
.build()
|
||||
);
|
||||
List<UUID> chunkIds = textHits.stream()
|
||||
.map(d -> {
|
||||
try { return UUID.fromString(d.getId()); }
|
||||
catch (Exception e) { return null; }
|
||||
})
|
||||
.filter(Objects::nonNull)
|
||||
.toList();
|
||||
if (chunkIds.isEmpty()) return List.of();
|
||||
return metadataRepository.findByChunkIdIn(chunkIds);
|
||||
}
|
||||
|
||||
private FacetBundle hydrate(List<ChunkMetadataEntity> chunks, List<FigureEntity> semanticFigures) {
|
||||
List<String> sectionIds = chunks.stream()
|
||||
.map(ChunkMetadataEntity::getSectionId)
|
||||
.distinct()
|
||||
.toList();
|
||||
List<SectionEntity> sections = sectionIds.isEmpty()
|
||||
? List.of()
|
||||
: sectionRepository.findAllById(sectionIds);
|
||||
|
||||
List<UUID> chunkIds = chunks.stream().map(ChunkMetadataEntity::getChunkId).toList();
|
||||
List<String> linkedFigureIds = chunkFigureRefRepository.findByChunkIdIn(chunkIds)
|
||||
.stream()
|
||||
.map(ChunkFigureRefEntity::getFigureId)
|
||||
.distinct()
|
||||
.toList();
|
||||
List<FigureEntity> linkedFigures = linkedFigureIds.isEmpty()
|
||||
? List.of()
|
||||
: figureRepository.findAllById(linkedFigureIds);
|
||||
|
||||
// Merge caption-semantic-search figures with chunk-linked figures (dedupe by id, linked first)
|
||||
Map<String, FigureEntity> merged = new LinkedHashMap<>();
|
||||
linkedFigures.forEach(f -> merged.put(f.getId(), f));
|
||||
semanticFigures.forEach(f -> merged.putIfAbsent(f.getId(), f));
|
||||
|
||||
List<String> summaries = chunks.stream()
|
||||
.map(ChunkMetadataEntity::getSummary)
|
||||
.filter(s -> s != null && !s.isBlank())
|
||||
.distinct()
|
||||
.toList();
|
||||
|
||||
return new FacetBundle(sections, new ArrayList<>(merged.values()), summaries);
|
||||
}
|
||||
|
||||
private List<FigureEntity> semanticFigureSearch(String query, UUID bookId) {
|
||||
FilterExpressionBuilder b = new FilterExpressionBuilder();
|
||||
List<Document> figureHits = vectorStore.similaritySearch(
|
||||
SearchRequest.builder()
|
||||
.query(query)
|
||||
.topK(FIGURE_TOP_K)
|
||||
.filterExpression(b.and(
|
||||
b.eq("type", "FIGURE"),
|
||||
b.eq("book_id", bookId.toString())
|
||||
).build())
|
||||
.build()
|
||||
);
|
||||
List<String> figureIds = figureHits.stream()
|
||||
.map(d -> (String) d.getMetadata().get("figure_id"))
|
||||
.filter(Objects::nonNull)
|
||||
.toList();
|
||||
return figureIds.isEmpty() ? List.of() : figureRepository.findAllById(figureIds);
|
||||
}
|
||||
|
||||
static String canonicalise(String raw) {
|
||||
if (raw == null) return "";
|
||||
String s = raw.trim().toLowerCase(Locale.ROOT);
|
||||
if (s.endsWith("ies") && s.length() > 3) {
|
||||
s = s.substring(0, s.length() - 3) + "y";
|
||||
} else if (s.endsWith("es") && s.length() > 2) {
|
||||
s = s.substring(0, s.length() - 2);
|
||||
} else if (s.endsWith("s") && s.length() > 1 && !s.endsWith("ss")) {
|
||||
s = s.substring(0, s.length() - 1);
|
||||
}
|
||||
return s;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
package com.aiteacher.concept;
|
||||
|
||||
import com.aiteacher.document.FigureEntity;
|
||||
import com.aiteacher.document.SectionEntity;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
public record FacetBundle(
|
||||
List<SectionEntity> sections,
|
||||
List<FigureEntity> figures,
|
||||
List<String> chunkSummaries
|
||||
) {}
|
||||
@@ -0,0 +1,10 @@
|
||||
package com.aiteacher.concept;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.UUID;
|
||||
|
||||
public record SavedConceptReportItem(
|
||||
UUID id,
|
||||
int reportNumber,
|
||||
Instant generatedAt
|
||||
) {}
|
||||
@@ -0,0 +1,37 @@
|
||||
package com.aiteacher.config;
|
||||
|
||||
import com.aiteacher.figure.FigureStorageService;
|
||||
import org.springframework.http.HttpStatus;
|
||||
import org.springframework.web.bind.annotation.*;
|
||||
import org.springframework.web.server.ResponseStatusException;
|
||||
|
||||
import jakarta.servlet.http.HttpServletResponse;
|
||||
import java.io.IOException;
|
||||
|
||||
/**
|
||||
* Serves figure images by redirecting to a presigned S3 URL.
|
||||
* The key stored in DB is the full S3 object key, e.g. "figures/{bookId}/{figureId}.png".
|
||||
*/
|
||||
@RestController
|
||||
@RequestMapping("/api/v1/figures")
|
||||
public class FigureStorageConfig {
|
||||
|
||||
private final FigureStorageService figureStorageService;
|
||||
|
||||
public FigureStorageConfig(FigureStorageService figureStorageService) {
|
||||
this.figureStorageService = figureStorageService;
|
||||
}
|
||||
|
||||
@GetMapping("/{bookId}/{filename}")
|
||||
public void serve(@PathVariable String bookId,
|
||||
@PathVariable String filename,
|
||||
HttpServletResponse response) throws IOException {
|
||||
String key = "figures/" + bookId + "/" + filename;
|
||||
try {
|
||||
String url = figureStorageService.presignedUrl(key);
|
||||
response.sendRedirect(url);
|
||||
} catch (Exception ex) {
|
||||
throw new ResponseStatusException(HttpStatus.NOT_FOUND, "Figure not found: " + key);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
package com.aiteacher.config;
|
||||
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
import org.springframework.http.client.JdkClientHttpRequestFactory;
|
||||
import org.springframework.web.client.RestClient;
|
||||
|
||||
import java.net.http.HttpClient;
|
||||
|
||||
@Configuration
|
||||
public class MarkerConfig {
|
||||
|
||||
@Value("${app.marker.base-url:http://localhost:8000}")
|
||||
private String markerBaseUrl;
|
||||
|
||||
@Bean
|
||||
RestClient markerRestClient() {
|
||||
// Use the JDK HTTP client with no timeout — Marker conversions can take several minutes.
|
||||
HttpClient httpClient = HttpClient.newBuilder()
|
||||
.build();
|
||||
JdkClientHttpRequestFactory factory = new JdkClientHttpRequestFactory(httpClient);
|
||||
// No read timeout set: JDK HTTP client defaults to no deadline.
|
||||
|
||||
return RestClient.builder()
|
||||
.baseUrl(markerBaseUrl)
|
||||
.requestFactory(factory)
|
||||
.build();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
package com.aiteacher.config;
|
||||
|
||||
import org.springframework.aot.hint.MemberCategory;
|
||||
import org.springframework.aot.hint.RuntimeHints;
|
||||
import org.springframework.aot.hint.RuntimeHintsRegistrar;
|
||||
import org.springframework.aot.hint.TypeReference;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* GraalVM native-image runtime hints for third-party libraries that use reflection
|
||||
* or classpath resource scanning not covered by Spring Boot's AOT processor.
|
||||
*
|
||||
* Registered via @ImportRuntimeHints on AiTeacherApplication.
|
||||
*/
|
||||
public class NativeHintsConfig implements RuntimeHintsRegistrar {
|
||||
|
||||
@Override
|
||||
public void registerHints(RuntimeHints hints, ClassLoader classLoader) {
|
||||
// PDFBox — font and encoding resources loaded via classpath scanning at runtime
|
||||
hints.resources().registerPattern("org/apache/pdfbox/resources/*");
|
||||
hints.resources().registerPattern("org/apache/pdfbox/resources/afm/*");
|
||||
hints.resources().registerPattern("org/apache/pdfbox/resources/cmap/*");
|
||||
hints.resources().registerPattern("org/apache/pdfbox/resources/glyphlist/*");
|
||||
hints.resources().registerPattern("org/apache/pdfbox/resources/icc/*");
|
||||
hints.resources().registerPattern("org/apache/pdfbox/resources/ttf/*");
|
||||
hints.resources().registerPattern("org/apache/pdfbox/resources/version.properties");
|
||||
|
||||
// PDFBox — font encoding classes instantiated via reflection
|
||||
hints.reflection().registerType(
|
||||
org.apache.pdfbox.pdmodel.font.encoding.GlyphList.class,
|
||||
MemberCategory.INVOKE_PUBLIC_CONSTRUCTORS,
|
||||
MemberCategory.INVOKE_PUBLIC_METHODS
|
||||
);
|
||||
hints.reflection().registerType(
|
||||
org.apache.pdfbox.pdmodel.font.encoding.WinAnsiEncoding.class,
|
||||
MemberCategory.INVOKE_PUBLIC_CONSTRUCTORS
|
||||
);
|
||||
hints.reflection().registerType(
|
||||
org.apache.pdfbox.pdmodel.font.encoding.MacRomanEncoding.class,
|
||||
MemberCategory.INVOKE_PUBLIC_CONSTRUCTORS
|
||||
);
|
||||
hints.reflection().registerType(
|
||||
org.apache.pdfbox.pdmodel.font.encoding.MacExpertEncoding.class,
|
||||
MemberCategory.INVOKE_PUBLIC_CONSTRUCTORS
|
||||
);
|
||||
hints.reflection().registerType(
|
||||
org.apache.pdfbox.pdmodel.font.encoding.StandardEncoding.class,
|
||||
MemberCategory.INVOKE_PUBLIC_CONSTRUCTORS
|
||||
);
|
||||
|
||||
// JPA / Hibernate — array types used in entity mappings
|
||||
hints.reflection().registerType(java.util.UUID[].class, MemberCategory.INVOKE_PUBLIC_CONSTRUCTORS);
|
||||
|
||||
// JBoss Logging — message logger implementations generated by annotation processor.
|
||||
// JBoss Logging uses reflection to look up the generated *_$logger class by name.
|
||||
registerJBossLogger(hints, "org.hibernate.jpa.internal.JpaLogger_$logger");
|
||||
registerJBossLogger(hints, "org.hibernate.internal.CoreMessageLogger_$logger");
|
||||
registerJBossLogger(hints, "org.hibernate.internal.EntityManagerMessageLogger_$logger");
|
||||
|
||||
// AWS SDK v2 — HTTP client and SdkPojo serialization
|
||||
hints.resources().registerPattern("software/amazon/awssdk/global/handlers/execution.interceptors");
|
||||
hints.resources().registerPattern("software/amazon/awssdk/services/s3/execution.interceptors");
|
||||
hints.resources().registerPattern("codegen-resources/s3/*");
|
||||
hints.reflection().registerType(
|
||||
software.amazon.awssdk.services.s3.S3Client.class,
|
||||
MemberCategory.INVOKE_PUBLIC_METHODS
|
||||
);
|
||||
|
||||
// Jackson deserialization of records persisted as JSON in DB columns.
|
||||
// These are reached only via ObjectMapper.readValue in services, so Spring's
|
||||
// BindingReflectionHintsRegistrar does not auto-discover all accessors.
|
||||
for (Class<?> type : List.of(
|
||||
com.aiteacher.topic.TopicSummaryResponse.class,
|
||||
com.aiteacher.topic.TopicSummaryResponse.SourceReference.class,
|
||||
com.aiteacher.concept.ConceptReportResponse.class,
|
||||
com.aiteacher.concept.ConceptReportResponse.FacetSection.class
|
||||
)) {
|
||||
hints.reflection().registerType(type,
|
||||
MemberCategory.INVOKE_DECLARED_CONSTRUCTORS,
|
||||
MemberCategory.INVOKE_DECLARED_METHODS);
|
||||
}
|
||||
}
|
||||
|
||||
private void registerJBossLogger(RuntimeHints hints, String className) {
|
||||
hints.reflection().registerType(
|
||||
TypeReference.of(className),
|
||||
MemberCategory.INVOKE_PUBLIC_CONSTRUCTORS,
|
||||
MemberCategory.INVOKE_PUBLIC_METHODS
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -20,7 +20,9 @@ public class SecurityConfig {
|
||||
@Bean
|
||||
public SecurityFilterChain filterChain(HttpSecurity http) throws Exception {
|
||||
http
|
||||
.authorizeHttpRequests(auth -> auth.anyRequest().authenticated())
|
||||
.authorizeHttpRequests(auth -> auth
|
||||
.requestMatchers("/api/v1/figures/**").permitAll()
|
||||
.anyRequest().authenticated())
|
||||
.httpBasic(Customizer.withDefaults())
|
||||
.csrf(AbstractHttpConfigurer::disable);
|
||||
return http.build();
|
||||
@@ -28,9 +30,10 @@ public class SecurityConfig {
|
||||
|
||||
@Bean
|
||||
public UserDetailsService userDetailsService(
|
||||
@Value("${app.auth.username}") String username,
|
||||
@Value("${app.auth.password}") String password) {
|
||||
UserDetails user = User.builder()
|
||||
.username("neurosurgeon")
|
||||
.username(username)
|
||||
.password("{noop}" + password)
|
||||
.roles("USER")
|
||||
.build();
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
package com.aiteacher.document;
|
||||
|
||||
import jakarta.persistence.*;
|
||||
import java.time.Instant;
|
||||
import java.util.UUID;
|
||||
|
||||
@Entity
|
||||
@Table(name = "chapter")
|
||||
public class ChapterEntity {
|
||||
|
||||
@Id
|
||||
@Column(name = "id", length = 200)
|
||||
private String id;
|
||||
|
||||
@Column(name = "book_id", nullable = false)
|
||||
private UUID bookId;
|
||||
|
||||
@Column(name = "number", nullable = false)
|
||||
private int number;
|
||||
|
||||
@Column(name = "title", length = 500)
|
||||
private String title;
|
||||
|
||||
@Column(name = "page_start")
|
||||
private Integer pageStart;
|
||||
|
||||
@Column(name = "created_at", nullable = false)
|
||||
private Instant createdAt;
|
||||
|
||||
public ChapterEntity() {}
|
||||
|
||||
public ChapterEntity(String id, UUID bookId, int number, String title, Integer pageStart) {
|
||||
this.id = id;
|
||||
this.bookId = bookId;
|
||||
this.number = number;
|
||||
this.title = title;
|
||||
this.pageStart = pageStart;
|
||||
this.createdAt = Instant.now();
|
||||
}
|
||||
|
||||
public String getId() { return id; }
|
||||
public UUID getBookId() { return bookId; }
|
||||
public int getNumber() { return number; }
|
||||
public String getTitle() { return title; }
|
||||
public Integer getPageStart() { return pageStart; }
|
||||
public Instant getCreatedAt() { return createdAt; }
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
package com.aiteacher.document;
|
||||
|
||||
import org.springframework.data.jpa.repository.JpaRepository;
|
||||
|
||||
import java.util.UUID;
|
||||
|
||||
public interface ChapterRepository extends JpaRepository<ChapterEntity, String> {
|
||||
void deleteAllByBookId(UUID bookId);
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
package com.aiteacher.document;
|
||||
|
||||
import jakarta.persistence.*;
|
||||
import java.io.Serializable;
|
||||
import java.util.Objects;
|
||||
import java.util.UUID;
|
||||
|
||||
@Entity
|
||||
@Table(name = "chunk_figure_ref")
|
||||
@IdClass(ChunkFigureRefEntity.PK.class)
|
||||
public class ChunkFigureRefEntity {
|
||||
|
||||
@Id
|
||||
@Column(name = "chunk_id", nullable = false)
|
||||
private UUID chunkId;
|
||||
|
||||
@Id
|
||||
@Column(name = "figure_id", nullable = false, length = 200)
|
||||
private String figureId;
|
||||
|
||||
@Column(name = "mention_page")
|
||||
private Integer mentionPage;
|
||||
|
||||
public ChunkFigureRefEntity() {}
|
||||
|
||||
public ChunkFigureRefEntity(UUID chunkId, String figureId, Integer mentionPage) {
|
||||
this.chunkId = chunkId;
|
||||
this.figureId = figureId;
|
||||
this.mentionPage = mentionPage;
|
||||
}
|
||||
|
||||
public UUID getChunkId() { return chunkId; }
|
||||
public String getFigureId() { return figureId; }
|
||||
public Integer getMentionPage() { return mentionPage; }
|
||||
|
||||
public static class PK implements Serializable {
|
||||
private UUID chunkId;
|
||||
private String figureId;
|
||||
|
||||
public PK() {}
|
||||
public PK(UUID chunkId, String figureId) {
|
||||
this.chunkId = chunkId;
|
||||
this.figureId = figureId;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean equals(Object o) {
|
||||
if (this == o) return true;
|
||||
if (!(o instanceof PK pk)) return false;
|
||||
return Objects.equals(chunkId, pk.chunkId) && Objects.equals(figureId, pk.figureId);
|
||||
}
|
||||
|
||||
@Override
|
||||
public int hashCode() {
|
||||
return Objects.hash(chunkId, figureId);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
package com.aiteacher.document;
|
||||
|
||||
import org.springframework.data.jpa.repository.JpaRepository;
|
||||
import org.springframework.data.jpa.repository.Query;
|
||||
import org.springframework.data.repository.query.Param;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
|
||||
public interface ChunkFigureRefRepository extends JpaRepository<ChunkFigureRefEntity, ChunkFigureRefEntity.PK> {
|
||||
|
||||
@Query("SELECT r FROM ChunkFigureRefEntity r WHERE r.chunkId IN :chunkIds")
|
||||
List<ChunkFigureRefEntity> findByChunkIdIn(@Param("chunkIds") List<UUID> chunkIds);
|
||||
|
||||
@Query("DELETE FROM ChunkFigureRefEntity r WHERE r.figureId IN :figureIds")
|
||||
@org.springframework.data.jpa.repository.Modifying
|
||||
void deleteByFigureIdIn(@Param("figureIds") List<String> figureIds);
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
package com.aiteacher.document;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
import java.util.regex.Matcher;
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
/**
|
||||
* Scans chunk text for "Fig. X" and "Figure X" references and persists
|
||||
* ChunkFigureRefEntity rows linking that chunk to its referenced figures.
|
||||
*/
|
||||
@Service
|
||||
public class ChunkFigureRefService {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(ChunkFigureRefService.class);
|
||||
|
||||
// Matches: "Fig. 12-4", "Fig. 12.4", "Fig 12", "Figure 12-4", etc.
|
||||
private static final Pattern REF_PATTERN =
|
||||
Pattern.compile("(?i)\\b(Fig\\.?|Figure)\\s+(\\d+[\\-.\\d]*)");
|
||||
|
||||
private final ChunkFigureRefRepository refRepository;
|
||||
|
||||
public ChunkFigureRefService(ChunkFigureRefRepository refRepository) {
|
||||
this.refRepository = refRepository;
|
||||
}
|
||||
|
||||
/**
|
||||
* For each text chunk, finds figure references and persists ChunkFigureRefEntity rows.
|
||||
*/
|
||||
public void linkChunksToFigures(List<Document> chunks, List<FigureEntity> bookFigures,
|
||||
int pageNum) {
|
||||
if (bookFigures.isEmpty()) return;
|
||||
|
||||
for (Document chunk : chunks) {
|
||||
String chunkIdStr = chunk.getId();
|
||||
UUID chunkId;
|
||||
try {
|
||||
chunkId = UUID.fromString(chunkIdStr);
|
||||
} catch (IllegalArgumentException ex) {
|
||||
log.warn("Chunk has non-UUID id: {}", chunkIdStr);
|
||||
continue;
|
||||
}
|
||||
|
||||
Matcher m = REF_PATTERN.matcher(chunk.getText());
|
||||
while (m.find()) {
|
||||
String refNum = m.group(2).trim();
|
||||
// Find matching figure by label suffix
|
||||
for (FigureEntity figure : bookFigures) {
|
||||
if (figure.getLabel() != null && figure.getLabel().endsWith(refNum)) {
|
||||
refRepository.save(new ChunkFigureRefEntity(chunkId, figure.getId(), pageNum));
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
package com.aiteacher.document;
|
||||
|
||||
import jakarta.persistence.*;
|
||||
import java.time.Instant;
|
||||
import java.util.UUID;
|
||||
|
||||
@Entity
|
||||
@Table(name = "figure")
|
||||
public class FigureEntity {
|
||||
|
||||
@Id
|
||||
@Column(name = "id", length = 200)
|
||||
private String id;
|
||||
|
||||
@Column(name = "book_id", nullable = false)
|
||||
private UUID bookId;
|
||||
|
||||
@Column(name = "section_id", length = 200)
|
||||
private String sectionId;
|
||||
|
||||
@Column(name = "chapter_id", length = 200)
|
||||
private String chapterId;
|
||||
|
||||
@Column(name = "label", length = 100)
|
||||
private String label;
|
||||
|
||||
@Column(name = "caption", columnDefinition = "TEXT")
|
||||
private String caption;
|
||||
|
||||
@Enumerated(EnumType.STRING)
|
||||
@Column(name = "figure_type", nullable = false, length = 50)
|
||||
private FigureType figureType;
|
||||
|
||||
@Column(name = "page", nullable = false)
|
||||
private int page;
|
||||
|
||||
@Column(name = "image_path", nullable = false, length = 1000)
|
||||
private String imagePath;
|
||||
|
||||
@Column(name = "caption_embedding_id")
|
||||
private UUID captionEmbeddingId;
|
||||
|
||||
@Column(name = "created_at", nullable = false)
|
||||
private Instant createdAt;
|
||||
|
||||
public FigureEntity() {}
|
||||
|
||||
public FigureEntity(String id, UUID bookId, String sectionId, String chapterId,
|
||||
String label, String caption, FigureType figureType,
|
||||
int page, String imagePath) {
|
||||
this.id = id;
|
||||
this.bookId = bookId;
|
||||
this.sectionId = sectionId;
|
||||
this.chapterId = chapterId;
|
||||
this.label = label;
|
||||
this.caption = caption;
|
||||
this.figureType = figureType;
|
||||
this.page = page;
|
||||
this.imagePath = imagePath;
|
||||
this.createdAt = Instant.now();
|
||||
}
|
||||
|
||||
public String getId() { return id; }
|
||||
public UUID getBookId() { return bookId; }
|
||||
public String getSectionId() { return sectionId; }
|
||||
public String getChapterId() { return chapterId; }
|
||||
public String getLabel() { return label; }
|
||||
public String getCaption() { return caption; }
|
||||
public FigureType getFigureType() { return figureType; }
|
||||
public int getPage() { return page; }
|
||||
public String getImagePath() { return imagePath; }
|
||||
public UUID getCaptionEmbeddingId() { return captionEmbeddingId; }
|
||||
public Instant getCreatedAt() { return createdAt; }
|
||||
|
||||
public void setCaptionEmbeddingId(UUID captionEmbeddingId) {
|
||||
this.captionEmbeddingId = captionEmbeddingId;
|
||||
}
|
||||
|
||||
public void setCaption(String caption) {
|
||||
this.caption = caption;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,151 @@
|
||||
package com.aiteacher.document;
|
||||
|
||||
import com.aiteacher.figure.FigureStorageService;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import javax.imageio.ImageIO;
|
||||
import java.awt.image.BufferedImage;
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.IOException;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.UUID;
|
||||
import java.util.regex.Matcher;
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
/**
|
||||
* Extracts figure images from {@link PageResult.FigureData} entries produced by
|
||||
* {@link MarkerPageParser}.
|
||||
*
|
||||
* <p>Marker returns pre-cropped PNG bytes for each detected figure, so no PDFBox
|
||||
* page rendering or bounding-box cropping is needed. This service:
|
||||
* <ol>
|
||||
* <li>Decodes the PNG bytes to check dimensions (skip images below min size)</li>
|
||||
* <li>Classifies the figure type from caption and surrounding text keywords</li>
|
||||
* <li>Persists the image via {@link FigureStorageService}</li>
|
||||
* <li>Persists a {@link FigureEntity} to the database</li>
|
||||
* </ol>
|
||||
*/
|
||||
@Service
|
||||
public class FigureExtractionService {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(FigureExtractionService.class);
|
||||
|
||||
private static final Pattern LABEL_PATTERN =
|
||||
Pattern.compile("(?i)Fig\\.?\\s*(\\d+[\\-.\\d]*)");
|
||||
|
||||
private final FigureStorageService storageService;
|
||||
private final FigureRepository figureRepository;
|
||||
private final int minImageSizePx;
|
||||
|
||||
public FigureExtractionService(
|
||||
FigureStorageService storageService,
|
||||
FigureRepository figureRepository,
|
||||
@Value("${app.figure-storage.min-image-size-px:100}") int minImageSizePx) {
|
||||
this.storageService = storageService;
|
||||
this.figureRepository = figureRepository;
|
||||
this.minImageSizePx = minImageSizePx;
|
||||
}
|
||||
|
||||
/** Holds the extraction output: persisted figures and a Marker blockId → DB figureId map. */
|
||||
public record ExtractionResult(List<FigureEntity> figures, Map<String, String> blockIdToFigureId) {}
|
||||
|
||||
/**
|
||||
* Extracts and persists figures for all pages described by {@code pageResults}.
|
||||
*
|
||||
* @param bookId owning book
|
||||
* @param chapterId chapter bucket for these sections
|
||||
* @param pageResults Marker parse output — each entry's {@code figures} list
|
||||
* carries pre-cropped PNG bytes for that page
|
||||
* @return {@link ExtractionResult} with persisted figures and blockId→figureId map
|
||||
* (used to resolve markdown image placeholders)
|
||||
*/
|
||||
public ExtractionResult extract(UUID bookId, String chapterId,
|
||||
List<PageResult> pageResults) {
|
||||
List<FigureEntity> figures = new ArrayList<>();
|
||||
Map<String, String> blockIdToFigureId = new HashMap<>();
|
||||
int figureCounter = 0;
|
||||
|
||||
for (PageResult page : pageResults) {
|
||||
if (page.figures().isEmpty()) continue;
|
||||
|
||||
for (PageResult.FigureData figureData : page.figures()) {
|
||||
try {
|
||||
BufferedImage image = decodeImage(figureData.imageBytes());
|
||||
if (image == null) {
|
||||
log.debug("Could not decode image on page {} of book {} (block {})",
|
||||
page.pageNumber(), bookId, figureData.blockId());
|
||||
continue;
|
||||
}
|
||||
if (image.getWidth() < minImageSizePx || image.getHeight() < minImageSizePx) {
|
||||
log.debug("Skipping small figure on page {} ({}×{})",
|
||||
page.pageNumber(), image.getWidth(), image.getHeight());
|
||||
continue;
|
||||
}
|
||||
|
||||
figureCounter++;
|
||||
String figureId = bookId + "-fig-" + page.pageNumber() + "-" + figureCounter;
|
||||
String caption = figureData.nearestCaption();
|
||||
String label = detectLabel(caption, figureCounter);
|
||||
FigureType type = classifyType(caption, page.orderedText());
|
||||
|
||||
String sectionId = bookId + "-p" + page.pageNumber();
|
||||
String imagePath = storageService.save(bookId, figureId, image);
|
||||
|
||||
FigureEntity figure = new FigureEntity(
|
||||
figureId, bookId, sectionId, chapterId,
|
||||
label, caption, type, page.pageNumber(), imagePath);
|
||||
figures.add(figureRepository.save(figure));
|
||||
blockIdToFigureId.put(figureData.blockId(), figureId);
|
||||
|
||||
} catch (Exception ex) {
|
||||
log.warn("Failed to extract figure on page {} of book {}: {}",
|
||||
page.pageNumber(), bookId, ex.getMessage());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
log.info("Extracted {} figures for book {}", figures.size(), bookId);
|
||||
return new ExtractionResult(figures, blockIdToFigureId);
|
||||
}
|
||||
|
||||
// --- Private helpers ---
|
||||
|
||||
private BufferedImage decodeImage(byte[] imageBytes) {
|
||||
if (imageBytes == null || imageBytes.length == 0) return null;
|
||||
try {
|
||||
return ImageIO.read(new ByteArrayInputStream(imageBytes));
|
||||
} catch (IOException ex) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private String detectLabel(String caption, int counter) {
|
||||
if (caption != null) {
|
||||
Matcher m = LABEL_PATTERN.matcher(caption);
|
||||
if (m.find()) return "Fig. " + m.group(1).trim();
|
||||
}
|
||||
return "Fig. " + counter;
|
||||
}
|
||||
|
||||
private FigureType classifyType(String caption, String pageText) {
|
||||
String combined = ((caption != null ? caption : "") + " " +
|
||||
(pageText != null ? pageText : "")).toLowerCase();
|
||||
if (combined.contains("mri") || combined.contains("ct ") || combined.contains("magnetic")
|
||||
|| combined.contains("tomography")) return FigureType.MRI_CT_SCAN;
|
||||
if (combined.contains("intraoperative") || combined.contains("intra-op"))
|
||||
return FigureType.INTRAOPERATIVE_IMAGE;
|
||||
if (caption != null && caption.toLowerCase().startsWith("table"))
|
||||
return FigureType.TABLE;
|
||||
if (combined.contains("chart") || combined.contains("histogram") || combined.contains("graph"))
|
||||
return FigureType.CHART;
|
||||
if (combined.contains("photograph") || combined.contains("photo"))
|
||||
return FigureType.SURGICAL_PHOTOGRAPH;
|
||||
return FigureType.ANATOMICAL_DIAGRAM;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
package com.aiteacher.document;
|
||||
|
||||
import org.springframework.data.jpa.repository.JpaRepository;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
|
||||
public interface FigureRepository extends JpaRepository<FigureEntity, String> {
|
||||
List<FigureEntity> findAllByBookId(UUID bookId);
|
||||
void deleteAllByBookId(UUID bookId);
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
package com.aiteacher.document;
|
||||
|
||||
public enum FigureType {
|
||||
ANATOMICAL_DIAGRAM,
|
||||
SURGICAL_PHOTOGRAPH,
|
||||
MRI_CT_SCAN,
|
||||
TABLE,
|
||||
CHART,
|
||||
INTRAOPERATIVE_IMAGE
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
package com.aiteacher.document;
|
||||
|
||||
import java.util.UUID;
|
||||
|
||||
public interface MarkdownStorageService {
|
||||
/** Uploads the markdown content and returns the S3 key. */
|
||||
String save(UUID bookId, int pageNumber, String markdown);
|
||||
|
||||
/** Downloads and returns the markdown content for the given book and page. */
|
||||
String getText(UUID bookId, int pageNumber);
|
||||
|
||||
/** Deletes all markdown files for the given book. */
|
||||
void deleteAll(UUID bookId);
|
||||
}
|
||||
@@ -0,0 +1,335 @@
|
||||
package com.aiteacher.document;
|
||||
|
||||
import tools.jackson.databind.JsonNode;
|
||||
import tools.jackson.databind.ObjectMapper;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.factory.annotation.Qualifier;
|
||||
import org.springframework.core.io.FileSystemResource;
|
||||
import org.springframework.http.MediaType;
|
||||
import org.springframework.stereotype.Service;
|
||||
import org.springframework.util.LinkedMultiValueMap;
|
||||
import org.springframework.util.MultiValueMap;
|
||||
import org.springframework.web.client.RestClient;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.*;
|
||||
|
||||
|
||||
/**
|
||||
* Parses a PDF with a single call to the Marker server using {@code output_format=json}.
|
||||
*
|
||||
* <p>The JSON response contains an {@code output} field that is itself a JSON string with a
|
||||
* tree structure: the root has a {@code children} array where each item is a {@code Page} block.
|
||||
* Each block carries an {@code html} field with {@code <content-ref src='blockId'>} placeholders
|
||||
* that reference its {@code children} by ID.
|
||||
*
|
||||
* <p>{@link #jsonToHtml} mirrors the Marker Python {@code json_to_html} utility: it walks the
|
||||
* tree recursively and resolves every {@code content-ref} with the rendered HTML of the
|
||||
* referenced child block.
|
||||
*
|
||||
* <p>Returns a {@link ParsedBook} with:
|
||||
* <ul>
|
||||
* <li>{@code pages} — one {@link PageResult} per non-empty page (drives embeddings)</li>
|
||||
* <li>{@code htmlByPage} — full resolved HTML per page (saved to S3 for the reader)</li>
|
||||
* </ul>
|
||||
*/
|
||||
@Service
|
||||
public class MarkerPageParser {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(MarkerPageParser.class);
|
||||
|
||||
private static final Set<String> TEXT_BLOCK_TYPES = Set.of(
|
||||
"Text", "TextInlineMath", "ListItem", "Table", "TableOfContents", "Code", "Equation",
|
||||
"Footnote", "Caption", "PageHeader", "PageFooter", "Handwriting"
|
||||
);
|
||||
private static final Set<String> FIGURE_BLOCK_TYPES = Set.of("Figure", "Picture", "FigureGroup", "PictureGroup");
|
||||
|
||||
private static final int CHUNK_SIZE = 100;
|
||||
|
||||
private static final ObjectMapper MAPPER = new ObjectMapper();
|
||||
|
||||
private final RestClient restClient;
|
||||
private final PdfSplitterService pdfSplitterService;
|
||||
|
||||
public MarkerPageParser(@Qualifier("markerRestClient") RestClient restClient,
|
||||
PdfSplitterService pdfSplitterService) {
|
||||
this.restClient = restClient;
|
||||
this.pdfSplitterService = pdfSplitterService;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parses a PDF by splitting it into {@value #CHUNK_SIZE}-page chunks, submitting each
|
||||
* chunk to Marker individually, and merging the results into a single {@link ParsedBook}.
|
||||
* Page numbers in the merged result are absolute (1-based across the whole document).
|
||||
*/
|
||||
public ParsedBook parse(Path pdfPath) throws IOException {
|
||||
List<PdfSplitterService.PdfChunk> chunks = pdfSplitterService.split(pdfPath, CHUNK_SIZE);
|
||||
log.info("Processing {} chunk(s) for {}", chunks.size(), pdfPath.getFileName());
|
||||
|
||||
List<PageResult> allPages = new ArrayList<>();
|
||||
Map<Integer, String> allHtml = new LinkedHashMap<>();
|
||||
|
||||
try {
|
||||
for (int c = 0; c < chunks.size(); c++) {
|
||||
PdfSplitterService.PdfChunk chunk = chunks.get(c);
|
||||
log.info("Submitting chunk {}/{} to Marker (page offset {})", c + 1, chunks.size(), chunk.pageOffset());
|
||||
|
||||
ParsedBook chunkResult = submitChunk(chunk.tempFile());
|
||||
|
||||
// Rebase page numbers from chunk-relative to document-absolute
|
||||
for (PageResult page : chunkResult.pages()) {
|
||||
int absolutePage = chunk.pageOffset() + page.pageNumber();
|
||||
allPages.add(new PageResult(absolutePage, page.orderedText(), page.headingTitle(), page.figures()));
|
||||
}
|
||||
chunkResult.htmlByPage().forEach((chunkPage, html) ->
|
||||
allHtml.put(chunk.pageOffset() + chunkPage, html));
|
||||
}
|
||||
} finally {
|
||||
// Delete temporary chunk files (skip if the chunk is the original PDF)
|
||||
for (PdfSplitterService.PdfChunk chunk : chunks) {
|
||||
if (!chunk.tempFile().equals(pdfPath)) {
|
||||
try { Files.deleteIfExists(chunk.tempFile()); }
|
||||
catch (IOException e) { log.warn("Could not delete temp chunk {}", chunk.tempFile()); }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
log.info("Marker produced {} non-empty pages from {} chunk(s) of {}",
|
||||
allPages.size(), chunks.size(), pdfPath.getFileName());
|
||||
return new ParsedBook(allPages, allHtml);
|
||||
}
|
||||
|
||||
/** Submits a single PDF file to Marker and returns the parsed result with chunk-relative page numbers. */
|
||||
private ParsedBook submitChunk(Path chunkPath) {
|
||||
MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
|
||||
body.add("file", new FileSystemResource(chunkPath));
|
||||
body.add("output_format", "json");
|
||||
|
||||
JsonNode response = restClient.post()
|
||||
.uri("/marker/upload")
|
||||
.contentType(MediaType.MULTIPART_FORM_DATA)
|
||||
.body(body)
|
||||
.retrieve()
|
||||
.body(JsonNode.class);
|
||||
|
||||
try {
|
||||
Files.writeString(Path.of("/tmp/marker-response-json.json"), response.toPrettyString());
|
||||
} catch (IOException e) {
|
||||
log.warn("Could not save Marker response to /tmp/marker-response-json.json", e);
|
||||
}
|
||||
|
||||
List<JsonNode> pageNodes = extractPages(response);
|
||||
if (pageNodes.isEmpty()) {
|
||||
log.warn("Marker returned no pages for chunk {}", chunkPath.getFileName());
|
||||
return new ParsedBook(List.of(), Map.of());
|
||||
}
|
||||
|
||||
List<PageResult> pages = new ArrayList<>();
|
||||
Map<Integer, String> htmlByPage = new LinkedHashMap<>();
|
||||
|
||||
for (int i = 0; i < pageNodes.size(); i++) {
|
||||
JsonNode pageNode = pageNodes.get(i);
|
||||
int pageNumber = i + 1; // 1-based, chunk-relative
|
||||
|
||||
PageResult result = buildPageResult(pageNode, pageNumber);
|
||||
String html = jsonToHtml(pageNode);
|
||||
|
||||
// Always save HTML so the reader can navigate to every page
|
||||
htmlByPage.put(pageNumber, html);
|
||||
|
||||
// Only queue for embedding if the page has extractable content
|
||||
if (!result.orderedText().isBlank() || !result.figures().isEmpty()) {
|
||||
pages.add(result);
|
||||
}
|
||||
}
|
||||
|
||||
return new ParsedBook(pages, htmlByPage);
|
||||
}
|
||||
|
||||
// ── Page extraction ───────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Parses the {@code output} JSON string and returns the list of page nodes
|
||||
* (the top-level {@code children} of the document root).
|
||||
*/
|
||||
private List<JsonNode> extractPages(JsonNode response) {
|
||||
if (response == null) return List.of();
|
||||
JsonNode outputNode = response.path("output");
|
||||
if (outputNode.isMissingNode()) {
|
||||
log.warn("Marker response has no 'output' field");
|
||||
return List.of();
|
||||
}
|
||||
try {
|
||||
JsonNode root = MAPPER.readTree(outputNode.stringValue());
|
||||
JsonNode children = root.path("children");
|
||||
if (children.isMissingNode() || !children.isArray()) {
|
||||
log.warn("Marker output root has no 'children' array");
|
||||
return List.of();
|
||||
}
|
||||
List<JsonNode> result = new ArrayList<>();
|
||||
children.forEach(result::add);
|
||||
return result;
|
||||
} catch (Exception e) {
|
||||
log.warn("Could not parse Marker 'output' string as JSON: {}", e.getMessage());
|
||||
return List.of();
|
||||
}
|
||||
}
|
||||
|
||||
// ── HTML rendering ────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Java equivalent of the Marker Python {@code json_to_html} utility.
|
||||
*
|
||||
* <p>Algorithm:
|
||||
* <ol>
|
||||
* <li>If the block has no children, return its {@code html} as-is (leaf node).</li>
|
||||
* <li>Otherwise recursively render each child, then replace every
|
||||
* {@code <content-ref src='childId'>} placeholder in the block's own {@code html}
|
||||
* with the rendered child HTML.</li>
|
||||
* </ol>
|
||||
*/
|
||||
String jsonToHtml(JsonNode block) {
|
||||
String html = str(block.path("html"));
|
||||
|
||||
// If the block carries image data, inject <img> data-URI tags.
|
||||
// Marker stores base64 image bytes in block.images keyed by block ID.
|
||||
// Picture/Figure leaf blocks have empty html, so this is the only way to
|
||||
// get the image into the rendered output.
|
||||
JsonNode images = block.path("images");
|
||||
if (!images.isMissingNode() && !images.isNull() && !images.isEmpty()) {
|
||||
StringBuilder imgTags = new StringBuilder();
|
||||
images.properties().forEach(entry -> {
|
||||
String base64 = str(entry.getValue());
|
||||
if (!base64.isEmpty()) {
|
||||
String mime = detectImageMime(base64);
|
||||
imgTags.append("<img src=\"data:").append(mime)
|
||||
.append(";base64,").append(base64).append("\">");
|
||||
}
|
||||
});
|
||||
if (!imgTags.isEmpty()) {
|
||||
html = html + imgTags;
|
||||
}
|
||||
}
|
||||
|
||||
JsonNode children = block.path("children");
|
||||
if (children.isMissingNode() || children.isNull() || !children.isArray() || children.isEmpty()) {
|
||||
return html; // leaf node
|
||||
}
|
||||
|
||||
// Build id → rendered-html map for all direct children
|
||||
Map<String, String> childHtml = new LinkedHashMap<>();
|
||||
for (JsonNode child : children) {
|
||||
String id = str(child.path("id"));
|
||||
childHtml.put(id, jsonToHtml(child));
|
||||
}
|
||||
|
||||
// Replace every <content-ref src='id'></content-ref> with the child's HTML
|
||||
for (Map.Entry<String, String> entry : childHtml.entrySet()) {
|
||||
String ref = "<content-ref src='" + entry.getKey() + "'></content-ref>";
|
||||
html = html.replace(ref, entry.getValue());
|
||||
}
|
||||
|
||||
return html;
|
||||
}
|
||||
|
||||
// ── PageResult (text + figures for embeddings) ────────────────────────────
|
||||
|
||||
private PageResult buildPageResult(JsonNode pageBlock, int pageNumber) {
|
||||
StringBuilder text = new StringBuilder();
|
||||
String[] headingTitle = {null};
|
||||
List<PageResult.FigureData> figures = new ArrayList<>();
|
||||
|
||||
walkBlock(pageBlock, text, headingTitle, figures);
|
||||
return new PageResult(pageNumber, text.toString().strip(), headingTitle[0], figures);
|
||||
}
|
||||
|
||||
/** Recursively walks the block tree, collecting text and figures in reading order. */
|
||||
private void walkBlock(JsonNode block, StringBuilder text, String[] headingTitle,
|
||||
List<PageResult.FigureData> figures) {
|
||||
String type = str(block.path("block_type"));
|
||||
|
||||
if ("SectionHeader".equals(type)) {
|
||||
String heading = stripHtml(str(block.path("html"))).strip();
|
||||
if (!heading.isEmpty() && headingTitle[0] == null) headingTitle[0] = heading;
|
||||
appendText(text, heading);
|
||||
|
||||
} else if (TEXT_BLOCK_TYPES.contains(type)) {
|
||||
appendText(text, stripHtml(str(block.path("html"))));
|
||||
|
||||
} else if (FIGURE_BLOCK_TYPES.contains(type)) {
|
||||
String caption = findCaption(block);
|
||||
extractFigures(block, caption, figures);
|
||||
}
|
||||
|
||||
// Recurse into children (content-ref ordering is implicit via tree order)
|
||||
JsonNode children = block.path("children");
|
||||
if (!children.isMissingNode() && !children.isNull() && children.isArray()) {
|
||||
for (JsonNode child : children) {
|
||||
walkBlock(child, text, headingTitle, figures);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Finds the first Caption child inside a figure block, if any. */
|
||||
private String findCaption(JsonNode figureBlock) {
|
||||
JsonNode children = figureBlock.path("children");
|
||||
if (children.isMissingNode() || !children.isArray()) return null;
|
||||
for (JsonNode child : children) {
|
||||
if ("Caption".equals(str(child.path("block_type")))) {
|
||||
String caption = stripHtml(str(child.path("html"))).strip();
|
||||
return caption.isEmpty() ? null : caption;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
private void extractFigures(JsonNode block, String caption, List<PageResult.FigureData> out) {
|
||||
JsonNode images = block.path("images");
|
||||
if (images.isMissingNode() || images.isEmpty()) return;
|
||||
|
||||
images.properties().forEach(entry -> {
|
||||
String blockId = entry.getKey();
|
||||
String base64 = str(entry.getValue());
|
||||
if (base64.isEmpty()) return;
|
||||
try {
|
||||
byte[] bytes = Base64.getDecoder().decode(base64);
|
||||
out.add(new PageResult.FigureData(bytes, caption, blockId));
|
||||
} catch (IllegalArgumentException ex) {
|
||||
log.warn("Could not decode base64 image for block {}: {}", blockId, ex.getMessage());
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// ── Utilities ─────────────────────────────────────────────────────────────
|
||||
|
||||
private void appendText(StringBuilder sb, String text) {
|
||||
if (text == null) return;
|
||||
String stripped = text.strip();
|
||||
if (stripped.isEmpty()) return;
|
||||
if (sb.length() > 0) sb.append("\n\n");
|
||||
sb.append(stripped);
|
||||
}
|
||||
|
||||
private String stripHtml(String html) {
|
||||
if (html == null || html.isEmpty()) return "";
|
||||
return html.replaceAll("<[^>]*>", "").replaceAll("\\s{2,}", " ").strip();
|
||||
}
|
||||
|
||||
/** Detects MIME type from the first characters of a base64-encoded image. */
|
||||
private static String detectImageMime(String base64) {
|
||||
if (base64.startsWith("/9j/")) return "image/jpeg";
|
||||
if (base64.startsWith("iVBOR")) return "image/png";
|
||||
if (base64.startsWith("R0lGO")) return "image/gif";
|
||||
if (base64.startsWith("UklGR")) return "image/webp";
|
||||
return "image/png"; // safe fallback
|
||||
}
|
||||
|
||||
/** Null-safe string extraction from a JsonNode (Jackson 3: stringValue() returns null for non-strings). */
|
||||
private static String str(JsonNode node) {
|
||||
String v = node.stringValue();
|
||||
return v != null ? v : "";
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
package com.aiteacher.document;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* Internal DTO produced by MarkerPageParser for one PDF page.
|
||||
* Decouples the Marker HTTP API from downstream services.
|
||||
*/
|
||||
public record PageResult(
|
||||
int pageNumber, // 1-based, derived from Marker page block index
|
||||
String orderedText, // full page text in correct reading order (blocks joined by \n\n)
|
||||
String headingTitle, // first SectionHeader block on page, or null
|
||||
List<FigureData> figures // extracted figure images (may be empty)
|
||||
) {
|
||||
|
||||
/**
|
||||
* A figure extracted from the page.
|
||||
* Image bytes are PNG data decoded from the Marker JSON {@code images} map.
|
||||
*/
|
||||
public record FigureData(
|
||||
byte[] imageBytes, // PNG image data (base64-decoded from Marker response)
|
||||
String nearestCaption, // text of the adjacent Caption block, or null
|
||||
String blockId // Marker block ID (e.g. "/page/0/Figure/2") for traceability
|
||||
) {}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
package com.aiteacher.document;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* Result of a full Marker parse: structured page data (from JSON) plus
|
||||
* native per-page markdown (from the separate Markdown API call).
|
||||
*
|
||||
* @param pages one entry per non-empty page, derived from the chunks response
|
||||
* @param htmlByPage concatenated block HTML keyed by 1-based page number
|
||||
*/
|
||||
public record ParsedBook(
|
||||
List<PageResult> pages,
|
||||
Map<Integer, String> htmlByPage
|
||||
) {}
|
||||
@@ -0,0 +1,72 @@
|
||||
package com.aiteacher.document;
|
||||
|
||||
import org.apache.pdfbox.io.RandomAccessReadBufferedFile;
|
||||
import org.apache.pdfbox.multipdf.Splitter;
|
||||
import org.apache.pdfbox.pdfparser.PDFParser;
|
||||
import org.apache.pdfbox.pdmodel.PDDocument;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* Splits a PDF file into fixed-size chunks using PDFBox.
|
||||
* Each chunk is saved as a temporary file so it can be submitted independently to Marker.
|
||||
*/
|
||||
@Service
|
||||
public class PdfSplitterService {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(PdfSplitterService.class);
|
||||
|
||||
/**
|
||||
* A chunk of a split PDF.
|
||||
*
|
||||
* @param tempFile path to the temporary PDF file (caller must delete when done)
|
||||
* @param pageOffset 0-based index of the first page in this chunk within the original document
|
||||
*/
|
||||
public record PdfChunk(Path tempFile, int pageOffset) {}
|
||||
|
||||
/**
|
||||
* Splits {@code pdfPath} into chunks of at most {@code maxPagesPerChunk} pages.
|
||||
* Returns a single-element list when the document fits in one chunk.
|
||||
*
|
||||
* @param pdfPath source PDF
|
||||
* @param maxPagesPerChunk maximum pages per chunk
|
||||
* @return ordered list of chunks; caller is responsible for deleting {@code tempFile}s
|
||||
*/
|
||||
public List<PdfChunk> split(Path pdfPath, int maxPagesPerChunk) throws IOException {
|
||||
try (PDDocument doc = new PDFParser(new RandomAccessReadBufferedFile(pdfPath.toFile())).parse()) {
|
||||
int totalPages = doc.getNumberOfPages();
|
||||
log.info("PDF {} has {} pages, splitting into chunks of {}", pdfPath.getFileName(), totalPages, maxPagesPerChunk);
|
||||
|
||||
if (totalPages <= maxPagesPerChunk) {
|
||||
// No split needed — return the original file as a single virtual chunk
|
||||
return List.of(new PdfChunk(pdfPath, 0));
|
||||
}
|
||||
|
||||
Splitter splitter = new Splitter();
|
||||
splitter.setSplitAtPage(maxPagesPerChunk);
|
||||
List<PDDocument> parts = splitter.split(doc);
|
||||
|
||||
List<PdfChunk> chunks = new ArrayList<>(parts.size());
|
||||
int offset = 0;
|
||||
for (PDDocument part : parts) {
|
||||
try {
|
||||
Path tmp = Files.createTempFile("marker-chunk-", ".pdf");
|
||||
part.save(tmp.toFile());
|
||||
chunks.add(new PdfChunk(tmp, offset));
|
||||
log.debug("Created chunk at {} (page offset {})", tmp, offset);
|
||||
offset += part.getNumberOfPages();
|
||||
} finally {
|
||||
part.close();
|
||||
}
|
||||
}
|
||||
return chunks;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,114 @@
|
||||
package com.aiteacher.document;
|
||||
|
||||
import org.apache.pdfbox.Loader;
|
||||
import org.apache.pdfbox.pdmodel.PDDocument;
|
||||
import org.apache.pdfbox.pdmodel.PDPage;
|
||||
import org.apache.pdfbox.pdmodel.common.PDRectangle;
|
||||
import org.apache.pdfbox.text.PDFTextStripperByArea;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.stereotype.Service;
|
||||
import org.springframework.transaction.annotation.Transactional;
|
||||
|
||||
import java.awt.Rectangle;
|
||||
import java.io.IOException;
|
||||
import java.nio.file.Path;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
|
||||
/**
|
||||
* Parses a PDF into page-level SectionEntity records stored in Postgres.
|
||||
* Uses column-aware extraction via PDFTextStripperByArea: for two-column pages,
|
||||
* left column is extracted first then right, preserving correct reading order.
|
||||
* Text is also normalized (collapsed whitespace) before storage.
|
||||
*/
|
||||
@Service
|
||||
public class PdfStructureParser {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(PdfStructureParser.class);
|
||||
|
||||
// Right column is considered empty (single-column page) if it has < 20% of left column's content
|
||||
private static final double TWO_COLUMN_THRESHOLD = 0.2;
|
||||
|
||||
private final ChapterRepository chapterRepository;
|
||||
private final SectionRepository sectionRepository;
|
||||
|
||||
public PdfStructureParser(ChapterRepository chapterRepository,
|
||||
SectionRepository sectionRepository) {
|
||||
this.chapterRepository = chapterRepository;
|
||||
this.sectionRepository = sectionRepository;
|
||||
}
|
||||
|
||||
@Transactional
|
||||
public List<SectionEntity> parse(UUID bookId, String bookTitle, Path pdfPath) {
|
||||
log.info("Parsing PDF structure for book {}", bookId);
|
||||
|
||||
String chapterId = bookId + "-ch1";
|
||||
ChapterEntity chapter = new ChapterEntity(chapterId, bookId, 1, bookTitle, 1);
|
||||
chapterRepository.save(chapter);
|
||||
|
||||
List<SectionEntity> sections = new ArrayList<>();
|
||||
|
||||
try (PDDocument doc = Loader.loadPDF(pdfPath.toFile())) {
|
||||
List<PDPage> pages = new ArrayList<>();
|
||||
doc.getPages().forEach(pages::add);
|
||||
|
||||
for (int i = 0; i < 25; i++) {
|
||||
int pageNum = i + 1;
|
||||
String text = normalizeWhitespace(extractPageText(pages.get(i)));
|
||||
if (text.isBlank()) continue;
|
||||
|
||||
String sectionId = bookId + "-p" + pageNum;
|
||||
SectionEntity section = new SectionEntity(
|
||||
sectionId, chapterId, bookId,
|
||||
String.valueOf(pageNum),
|
||||
"Page " + pageNum,
|
||||
pageNum, pageNum,
|
||||
text
|
||||
);
|
||||
sections.add(sectionRepository.save(section));
|
||||
}
|
||||
} catch (IOException e) {
|
||||
throw new RuntimeException("Failed to parse PDF for book " + bookId, e);
|
||||
}
|
||||
|
||||
log.info("Parsed {} sections for book {}", sections.size(), bookId);
|
||||
return sections;
|
||||
}
|
||||
|
||||
/**
|
||||
* Extracts text from a single page using column-aware region extraction.
|
||||
* Splits the page at the horizontal midpoint. If the right region has fewer
|
||||
* than 20% of the characters of the left region, treats the page as single-column.
|
||||
*/
|
||||
private String extractPageText(PDPage page) throws IOException {
|
||||
PDRectangle mediaBox = page.getMediaBox();
|
||||
int width = (int) mediaBox.getWidth();
|
||||
int height = (int) mediaBox.getHeight();
|
||||
int mid = width / 2;
|
||||
|
||||
PDFTextStripperByArea stripper = new PDFTextStripperByArea();
|
||||
stripper.setSortByPosition(true);
|
||||
stripper.addRegion("left", new Rectangle(0, 0, mid, height));
|
||||
stripper.addRegion("right", new Rectangle(mid, 0, width - mid, height));
|
||||
stripper.extractRegions(page);
|
||||
|
||||
String left = stripper.getTextForRegion("left").strip();
|
||||
String right = stripper.getTextForRegion("right").strip();
|
||||
|
||||
if (right.length() < left.length() * TWO_COLUMN_THRESHOLD) {
|
||||
// Single-column page — left holds all (or nearly all) content
|
||||
return left.isEmpty() ? right : left;
|
||||
}
|
||||
return left + "\n\n" + right;
|
||||
}
|
||||
|
||||
/** Collapses multi-space/tab runs and excessive blank lines. */
|
||||
private String normalizeWhitespace(String text) {
|
||||
return text
|
||||
.replaceAll("[ \t]{2,}", " ")
|
||||
.replaceAll("\n{3,}", "\n\n")
|
||||
.trim();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,97 @@
|
||||
package com.aiteacher.document;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.stereotype.Service;
|
||||
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
|
||||
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
|
||||
import software.amazon.awssdk.core.sync.RequestBody;
|
||||
import software.amazon.awssdk.regions.Region;
|
||||
import software.amazon.awssdk.services.s3.S3Client;
|
||||
import software.amazon.awssdk.services.s3.S3Configuration;
|
||||
import software.amazon.awssdk.services.s3.model.*;
|
||||
|
||||
import java.net.URI;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
|
||||
@Service
|
||||
public class S3MarkdownStorageService implements MarkdownStorageService {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(S3MarkdownStorageService.class);
|
||||
|
||||
private final S3Client s3;
|
||||
private final String bucket;
|
||||
|
||||
public S3MarkdownStorageService(
|
||||
@Value("${app.figure-storage.endpoint}") String endpoint,
|
||||
@Value("${app.figure-storage.region}") String region,
|
||||
@Value("${app.figure-storage.bucket}") String bucket,
|
||||
@Value("${app.figure-storage.access-key-id}") String accessKeyId,
|
||||
@Value("${app.figure-storage.secret-access-key}") String secretKey) {
|
||||
this.bucket = bucket;
|
||||
URI endpointUri = URI.create(endpoint);
|
||||
StaticCredentialsProvider credentials = StaticCredentialsProvider.create(
|
||||
AwsBasicCredentials.create(accessKeyId, secretKey));
|
||||
Region awsRegion = Region.of(region);
|
||||
S3Configuration s3Config = S3Configuration.builder().pathStyleAccessEnabled(true).build();
|
||||
|
||||
this.s3 = S3Client.builder()
|
||||
.endpointOverride(endpointUri)
|
||||
.region(awsRegion)
|
||||
.credentialsProvider(credentials)
|
||||
.serviceConfiguration(s3Config)
|
||||
.build();
|
||||
}
|
||||
|
||||
@Override
|
||||
public String save(UUID bookId, int pageNumber, String markdown) {
|
||||
String key = key(bookId, pageNumber);
|
||||
byte[] bytes = markdown.getBytes(StandardCharsets.UTF_8);
|
||||
s3.putObject(
|
||||
PutObjectRequest.builder().bucket(bucket).key(key)
|
||||
.contentType("text/html; charset=utf-8")
|
||||
.contentLength((long) bytes.length).build(),
|
||||
RequestBody.fromBytes(bytes));
|
||||
return key;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getText(UUID bookId, int pageNumber) {
|
||||
byte[] bytes = s3.getObjectAsBytes(
|
||||
GetObjectRequest.builder().bucket(bucket).key(key(bookId, pageNumber)).build()
|
||||
).asByteArray();
|
||||
return new String(bytes, StandardCharsets.UTF_8);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void deleteAll(UUID bookId) {
|
||||
String prefix = "html/" + bookId + "/";
|
||||
try {
|
||||
List<ObjectIdentifier> toDelete = new ArrayList<>();
|
||||
s3.listObjectsV2Paginator(ListObjectsV2Request.builder()
|
||||
.bucket(bucket).prefix(prefix).build()).stream()
|
||||
.flatMap(page -> page.contents().stream())
|
||||
.map(S3Object::key)
|
||||
.map(k -> ObjectIdentifier.builder().key(k).build())
|
||||
.forEach(toDelete::add);
|
||||
|
||||
if (toDelete.isEmpty()) return;
|
||||
|
||||
s3.deleteObjects(DeleteObjectsRequest.builder()
|
||||
.bucket(bucket)
|
||||
.delete(Delete.builder().objects(toDelete).build())
|
||||
.build());
|
||||
log.info("Deleted {} markdown files from S3 for book {}", toDelete.size(), bookId);
|
||||
} catch (S3Exception ex) {
|
||||
log.warn("Could not fully delete markdown for book {} from S3: {}", bookId, ex.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
private static String key(UUID bookId, int pageNumber) {
|
||||
return "html/" + bookId + "/page-" + pageNumber + ".html";
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
package com.aiteacher.document;
|
||||
|
||||
import jakarta.persistence.*;
|
||||
import java.time.Instant;
|
||||
import java.util.UUID;
|
||||
|
||||
@Entity
|
||||
@Table(name = "section")
|
||||
public class SectionEntity {
|
||||
|
||||
@Id
|
||||
@Column(name = "id", length = 200)
|
||||
private String id;
|
||||
|
||||
@Column(name = "chapter_id", nullable = false, length = 200)
|
||||
private String chapterId;
|
||||
|
||||
@Column(name = "book_id", nullable = false)
|
||||
private UUID bookId;
|
||||
|
||||
@Column(name = "number", length = 50)
|
||||
private String number;
|
||||
|
||||
@Column(name = "title", length = 500)
|
||||
private String title;
|
||||
|
||||
@Column(name = "page_start", nullable = false)
|
||||
private int pageStart;
|
||||
|
||||
@Column(name = "page_end", nullable = false)
|
||||
private int pageEnd;
|
||||
|
||||
@Column(name = "full_text", nullable = false, columnDefinition = "TEXT")
|
||||
private String fullText;
|
||||
|
||||
@Column(name = "created_at", nullable = false)
|
||||
private Instant createdAt;
|
||||
|
||||
public SectionEntity() {}
|
||||
|
||||
public SectionEntity(String id, String chapterId, UUID bookId, String number,
|
||||
String title, int pageStart, int pageEnd, String fullText) {
|
||||
this.id = id;
|
||||
this.chapterId = chapterId;
|
||||
this.bookId = bookId;
|
||||
this.number = number;
|
||||
this.title = title;
|
||||
this.pageStart = pageStart;
|
||||
this.pageEnd = pageEnd;
|
||||
this.fullText = fullText;
|
||||
this.createdAt = Instant.now();
|
||||
}
|
||||
|
||||
public String getId() { return id; }
|
||||
public String getChapterId() { return chapterId; }
|
||||
public UUID getBookId() { return bookId; }
|
||||
public String getNumber() { return number; }
|
||||
public String getTitle() { return title; }
|
||||
public int getPageStart() { return pageStart; }
|
||||
public int getPageEnd() { return pageEnd; }
|
||||
public String getFullText() { return fullText; }
|
||||
public Instant getCreatedAt() { return createdAt; }
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
package com.aiteacher.document;
|
||||
|
||||
import org.springframework.data.jpa.repository.JpaRepository;
|
||||
import org.springframework.data.jpa.repository.Query;
|
||||
import org.springframework.data.repository.query.Param;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
|
||||
public interface SectionRepository extends JpaRepository<SectionEntity, String> {
|
||||
List<SectionEntity> findAllByBookId(UUID bookId);
|
||||
void deleteAllByBookId(UUID bookId);
|
||||
|
||||
@Query("SELECT s FROM SectionEntity s WHERE s.bookId = :bookId AND s.pageStart <= :windowEnd AND s.pageEnd >= :windowStart ORDER BY s.pageStart")
|
||||
List<SectionEntity> findByBookIdAndPageOverlap(
|
||||
@Param("bookId") UUID bookId,
|
||||
@Param("windowStart") int windowStart,
|
||||
@Param("windowEnd") int windowEnd);
|
||||
}
|
||||
@@ -0,0 +1,103 @@
|
||||
package com.aiteacher.document;
|
||||
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.UUID;
|
||||
|
||||
/**
|
||||
* Splits a SectionEntity's full text into overlapping chunks for vector embedding.
|
||||
* Target size: ~1800 characters (~450 tokens); overlap: 200 characters.
|
||||
*/
|
||||
@Service
|
||||
public class TextChunkingService {
|
||||
|
||||
private static final int TARGET_CHARS = 1800;
|
||||
private static final int OVERLAP_CHARS = 200;
|
||||
|
||||
public List<Document> chunk(SectionEntity section, String bookTitle) {
|
||||
String text = section.getFullText();
|
||||
if (text == null || text.isBlank()) return List.of();
|
||||
|
||||
List<String> windows = split(text);
|
||||
List<Document> documents = new ArrayList<>();
|
||||
|
||||
for (int i = 0; i < windows.size(); i++) {
|
||||
String chunkId = UUID.randomUUID().toString();
|
||||
Map<String, Object> metadata = buildMetadata(section, bookTitle, i, windows.size(), chunkId);
|
||||
documents.add(new Document(chunkId, windows.get(i), metadata));
|
||||
}
|
||||
return documents;
|
||||
}
|
||||
|
||||
private List<String> split(String text) {
|
||||
List<String> windows = new ArrayList<>();
|
||||
int start = 0;
|
||||
while (start < text.length()) {
|
||||
int hardEnd = Math.min(start + TARGET_CHARS, text.length());
|
||||
if (hardEnd == text.length()) {
|
||||
String last = text.substring(start).strip();
|
||||
if (!last.isEmpty()) windows.add(last);
|
||||
break;
|
||||
}
|
||||
int splitAt = findSplitPoint(text, start, hardEnd);
|
||||
String chunk = text.substring(start, splitAt).strip();
|
||||
if (!chunk.isEmpty()) windows.add(chunk);
|
||||
// Overlap: back up from split point, align to a word start
|
||||
int overlapStart = Math.max(start + 1, splitAt - OVERLAP_CHARS);
|
||||
while (overlapStart < splitAt && text.charAt(overlapStart) != ' ') overlapStart++;
|
||||
start = overlapStart < splitAt ? overlapStart + 1 : splitAt;
|
||||
}
|
||||
return windows;
|
||||
}
|
||||
|
||||
/**
|
||||
* Finds the best split point at or before hardEnd, preferring (in order):
|
||||
* paragraph boundary, sentence boundary, word boundary, hard cut.
|
||||
*/
|
||||
private int findSplitPoint(String text, int start, int hardEnd) {
|
||||
int lookback = Math.min(400, (hardEnd - start) / 2);
|
||||
|
||||
// 1. Paragraph boundary
|
||||
int paraIdx = text.lastIndexOf("\n\n", hardEnd);
|
||||
if (paraIdx > hardEnd - lookback && paraIdx > start) return paraIdx + 2;
|
||||
|
||||
// 2. Sentence boundary (. ! ?) followed by space or newline
|
||||
for (int i = hardEnd - 1; i > hardEnd - lookback && i > start; i--) {
|
||||
char c = text.charAt(i);
|
||||
if ((c == '.' || c == '!' || c == '?') && i + 1 < text.length()) {
|
||||
char next = text.charAt(i + 1);
|
||||
if (next == ' ' || next == '\n') return i + 1;
|
||||
}
|
||||
}
|
||||
|
||||
// 3. Word boundary
|
||||
for (int i = hardEnd - 1; i > hardEnd - 100 && i > start; i--) {
|
||||
if (text.charAt(i) == ' ') return i + 1;
|
||||
}
|
||||
|
||||
// 4. Hard cut
|
||||
return hardEnd;
|
||||
}
|
||||
|
||||
private Map<String, Object> buildMetadata(SectionEntity section, String bookTitle,
|
||||
int index, int total, String chunkId) {
|
||||
Map<String, Object> m = new HashMap<>();
|
||||
m.put("type", "TEXT");
|
||||
m.put("book_id", section.getBookId().toString());
|
||||
m.put("book_title", bookTitle);
|
||||
m.put("chapter_id", section.getChapterId());
|
||||
m.put("section_id", section.getId());
|
||||
m.put("section_title", section.getTitle() != null ? section.getTitle() : "");
|
||||
m.put("page_start", section.getPageStart());
|
||||
m.put("page_end", section.getPageEnd());
|
||||
m.put("chunk_index", index);
|
||||
m.put("total_chunks", total);
|
||||
m.put("chunk_id", chunkId);
|
||||
return m;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,108 @@
|
||||
package com.aiteacher.document;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.ai.chat.client.ChatClient;
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.core.io.ByteArrayResource;
|
||||
import org.springframework.stereotype.Service;
|
||||
import org.springframework.util.MimeTypeUtils;
|
||||
|
||||
/**
|
||||
* Analyses an extracted figure image using the OpenAI vision model.
|
||||
*
|
||||
* <p>Returns an {@link ImageAnalysis} record containing:
|
||||
* <ul>
|
||||
* <li>{@code description} — 2-3 sentence clinical description of the image</li>
|
||||
* <li>{@code imageText} — all visible text, labels, and annotations copied verbatim
|
||||
* from the image (empty string when none present)</li>
|
||||
* </ul>
|
||||
*
|
||||
* <p>Both fields are stored: {@code description} drives the embedding; {@code imageText}
|
||||
* is added to chunk metadata so queries can match exact labels (e.g., "Circle of Willis").
|
||||
*/
|
||||
@Service
|
||||
public class VisionDescriptionService {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(VisionDescriptionService.class);
|
||||
|
||||
private static final String PROMPT = """
|
||||
You are a neurosurgery educator analysing a medical image.
|
||||
Respond in EXACTLY this format — no other text, no markdown:
|
||||
DESCRIPTION: <2-3 sentence clinical description focusing on anatomical structures, surgical landmarks, and clinical significance>
|
||||
IMAGE_TEXT: <all visible text, labels, measurements, and annotations copied verbatim, comma-separated; write NONE if no text visible>
|
||||
""";
|
||||
|
||||
/** Minimum ms between vision API calls. Configurable via app.vision.min-interval-ms. */
|
||||
private final long minIntervalMs;
|
||||
private final ChatClient chatClient;
|
||||
private volatile long lastCallAt = 0;
|
||||
|
||||
public VisionDescriptionService(
|
||||
ChatClient chatClient,
|
||||
@Value("${app.vision.min-interval-ms:2000}") long minIntervalMs) {
|
||||
this.chatClient = chatClient;
|
||||
this.minIntervalMs = minIntervalMs;
|
||||
}
|
||||
|
||||
/**
|
||||
* Holds the structured output of a vision model call on one figure image.
|
||||
*
|
||||
* @param description clinical description of the image content
|
||||
* @param imageText verbatim text visible inside the image; empty string if none
|
||||
*/
|
||||
public record ImageAnalysis(String description, String imageText) {}
|
||||
|
||||
/**
|
||||
* Analyses the image bytes and returns an {@link ImageAnalysis}.
|
||||
* Falls back gracefully: if the vision call fails, the caption is used as description
|
||||
* and imageText is left empty.
|
||||
*
|
||||
* @param imageBytes PNG bytes of the extracted figure
|
||||
* @param captionFallback caption detected from surrounding text, may be null
|
||||
*/
|
||||
public ImageAnalysis analyze(byte[] imageBytes, String captionFallback) {
|
||||
throttle();
|
||||
try {
|
||||
String raw = chatClient.prompt()
|
||||
.user(u -> u
|
||||
.text(PROMPT)
|
||||
.media(MimeTypeUtils.IMAGE_PNG, new ByteArrayResource(imageBytes)))
|
||||
.call()
|
||||
.content();
|
||||
return parse(raw, captionFallback);
|
||||
} catch (Exception ex) {
|
||||
log.warn("Vision analysis failed: {} — using caption as fallback", ex.getMessage());
|
||||
return new ImageAnalysis(
|
||||
captionFallback != null ? captionFallback : "Figure",
|
||||
"");
|
||||
}
|
||||
}
|
||||
|
||||
private synchronized void throttle() {
|
||||
long now = System.currentTimeMillis();
|
||||
long wait = minIntervalMs - (now - lastCallAt);
|
||||
if (wait > 0) {
|
||||
try { Thread.sleep(wait); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
|
||||
}
|
||||
lastCallAt = System.currentTimeMillis();
|
||||
}
|
||||
|
||||
private ImageAnalysis parse(String raw, String captionFallback) {
|
||||
String description = captionFallback != null ? captionFallback : "Figure";
|
||||
String imageText = "";
|
||||
|
||||
if (raw != null) {
|
||||
for (String line : raw.split("\n")) {
|
||||
if (line.startsWith("DESCRIPTION:")) {
|
||||
String val = line.substring("DESCRIPTION:".length()).strip();
|
||||
if (!val.isEmpty()) description = val;
|
||||
} else if (line.startsWith("IMAGE_TEXT:")) {
|
||||
String val = line.substring("IMAGE_TEXT:".length()).strip();
|
||||
if (!val.isEmpty() && !"NONE".equalsIgnoreCase(val)) imageText = val;
|
||||
}
|
||||
}
|
||||
}
|
||||
return new ImageAnalysis(description, imageText);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
package com.aiteacher.enrichment;
|
||||
|
||||
import com.aiteacher.document.SectionEntity;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.UUID;
|
||||
|
||||
@Service
|
||||
public class ChunkEnrichmentPipeline {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(ChunkEnrichmentPipeline.class);
|
||||
|
||||
private final ChunkEnrichmentService enrichmentService;
|
||||
private final ChunkMetadataRepository metadataRepository;
|
||||
|
||||
public ChunkEnrichmentPipeline(ChunkEnrichmentService enrichmentService,
|
||||
ChunkMetadataRepository metadataRepository) {
|
||||
this.enrichmentService = enrichmentService;
|
||||
this.metadataRepository = metadataRepository;
|
||||
}
|
||||
|
||||
public void enrichAndPersist(List<Document> chunks,
|
||||
Map<String, SectionEntity> sectionsById,
|
||||
String bookTitle) {
|
||||
int total = chunks.size();
|
||||
int done = 0;
|
||||
for (Document chunk : chunks) {
|
||||
String sectionId = (String) chunk.getMetadata().get("section_id");
|
||||
SectionEntity section = sectionId != null ? sectionsById.get(sectionId) : null;
|
||||
UUID chunkId;
|
||||
try {
|
||||
chunkId = UUID.fromString(chunk.getId());
|
||||
} catch (IllegalArgumentException ex) {
|
||||
log.warn("Skipping chunk with non-UUID id '{}'", chunk.getId());
|
||||
continue;
|
||||
}
|
||||
UUID bookId = extractBookId(chunk);
|
||||
if (bookId == null || sectionId == null) {
|
||||
log.warn("Skipping chunk {} missing book_id or section_id metadata", chunkId);
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
ChunkEnrichmentResult result = enrichmentService.enrich(chunk.getText(), section, bookTitle);
|
||||
ChunkMetadataEntity entity = new ChunkMetadataEntity(
|
||||
chunkId, bookId, sectionId,
|
||||
result.facet(), result.entities(), result.summary(),
|
||||
ChunkEnrichmentService.MODEL_VERSION, Instant.now());
|
||||
metadataRepository.save(entity);
|
||||
} catch (Exception ex) {
|
||||
log.warn("Enrichment failed for chunk {}: {}", chunkId, ex.getMessage());
|
||||
}
|
||||
done++;
|
||||
if (done % 25 == 0) {
|
||||
log.info("Enrichment progress: {}/{} chunks", done, total);
|
||||
}
|
||||
}
|
||||
log.info("Enrichment complete: {}/{} chunks enriched", done, total);
|
||||
}
|
||||
|
||||
private UUID extractBookId(Document chunk) {
|
||||
Object raw = chunk.getMetadata().get("book_id");
|
||||
if (raw == null) return null;
|
||||
try {
|
||||
return UUID.fromString(raw.toString());
|
||||
} catch (IllegalArgumentException ex) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
package com.aiteacher.enrichment;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
public record ChunkEnrichmentResult(
|
||||
List<String> entities,
|
||||
ConceptFacet facet,
|
||||
String summary
|
||||
) {}
|
||||
@@ -0,0 +1,135 @@
|
||||
package com.aiteacher.enrichment;
|
||||
|
||||
import com.aiteacher.document.SectionEntity;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.ai.chat.client.ChatClient;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
|
||||
@Service
|
||||
public class ChunkEnrichmentService {
|
||||
|
||||
public static final String MODEL_VERSION = "v1";
|
||||
private static final int MAX_ENTITIES = 8;
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(ChunkEnrichmentService.class);
|
||||
|
||||
private static final String SYSTEM_PROMPT = """
|
||||
You are a medical indexing assistant that classifies neurosurgery textbook excerpts.
|
||||
For each excerpt you receive, extract three fields:
|
||||
- entities: the medical concepts, conditions, procedures, tools, or anatomical
|
||||
structures the excerpt is ABOUT. Normalise each to lowercase, singular canonical
|
||||
English form. Expand abbreviations (e.g. "SAH" -> "subarachnoid hemorrhage").
|
||||
Avoid generic words ("patient", "technique"). Cap at %d entities.
|
||||
|
||||
- facet: exactly one of the following. Pick the SINGLE best fit based on the
|
||||
excerpt's PRIMARY teaching purpose. Use OTHER only when nothing else applies.
|
||||
|
||||
DEFINITION — defines the entity / syndrome / concept ("what is X").
|
||||
ANATOMY — neuroanatomy, vascular/tract relationships, operative
|
||||
landmarks, anatomical variants.
|
||||
PATHOPHYSIOLOGY — mechanism of disease, etiology, natural history,
|
||||
molecular/cellular basis.
|
||||
EPIDEMIOLOGY — incidence, prevalence, demographics, risk factors.
|
||||
CLINICAL_PRESENTATION — symptoms, signs, neurological exam findings, syndromes
|
||||
as they present in patients.
|
||||
IMAGING — CT / MRI / angiography / DSA / ultrasound features and
|
||||
interpretation. If the excerpt describes HOW something
|
||||
looks on imaging, use IMAGING.
|
||||
CLASSIFICATION — named grading scales, staging systems, subtype
|
||||
taxonomies (Hunt-Hess, WFNS, Fisher, Spetzler-Martin,
|
||||
GCS, Karnofsky, mRS, Simpson, etc.). If the excerpt
|
||||
defines or applies a named scale, use CLASSIFICATION
|
||||
even if it is grounded in imaging or clinical exam.
|
||||
INDICATIONS — when to operate / treat / observe; patient selection
|
||||
criteria; contraindications.
|
||||
SURGICAL_TECHNIQUE — operative approach, positioning, steps, landmarks,
|
||||
instruments, implants, intraoperative monitoring.
|
||||
NONSURGICAL_MANAGEMENT — medical therapy, endovascular treatment, stereotactic
|
||||
radiosurgery, conservative / observational management.
|
||||
COMPLICATIONS — intra- or postoperative complications, adverse events.
|
||||
OUTCOMES_FOLLOWUP — prognosis, morbidity/mortality rates, recurrence,
|
||||
surveillance schedules, follow-up care.
|
||||
OTHER — history, philosophy, ethics, or anything not covered.
|
||||
|
||||
Disambiguation rules:
|
||||
* A named grading scale => CLASSIFICATION (even when grounded in imaging/exam).
|
||||
* Tools and implants described as part of an operation => SURGICAL_TECHNIQUE,
|
||||
not a standalone facet.
|
||||
* Illustrative case reports => CLINICAL_PRESENTATION.
|
||||
* Imaging findings of complications => COMPLICATIONS, not IMAGING.
|
||||
|
||||
- summary: one or two sentences describing what the excerpt teaches.
|
||||
|
||||
Respond with the structured JSON requested. Do not fabricate content not present in
|
||||
the excerpt.
|
||||
""".formatted(MAX_ENTITIES);
|
||||
|
||||
private final ChatClient chatClient;
|
||||
|
||||
public ChunkEnrichmentService(ChatClient chatClient) {
|
||||
this.chatClient = chatClient;
|
||||
}
|
||||
|
||||
public ChunkEnrichmentResult enrich(String chunkText, SectionEntity section, String bookTitle) {
|
||||
String userPrompt = buildUserPrompt(chunkText, section, bookTitle);
|
||||
|
||||
LlmOutput raw = chatClient.prompt()
|
||||
.system(SYSTEM_PROMPT)
|
||||
.user(userPrompt)
|
||||
.call()
|
||||
.entity(LlmOutput.class);
|
||||
|
||||
if (raw == null) {
|
||||
log.warn("LLM returned null enrichment; defaulting to OTHER");
|
||||
return new ChunkEnrichmentResult(List.of(), ConceptFacet.OTHER, "");
|
||||
}
|
||||
|
||||
List<String> entities = normaliseEntities(raw.entities());
|
||||
ConceptFacet facet = parseFacet(raw.facet());
|
||||
String summary = raw.summary() != null ? raw.summary().strip() : "";
|
||||
return new ChunkEnrichmentResult(entities, facet, summary);
|
||||
}
|
||||
|
||||
private String buildUserPrompt(String chunkText, SectionEntity section, String bookTitle) {
|
||||
String sectionTitle = section != null && section.getTitle() != null ? section.getTitle() : "";
|
||||
return """
|
||||
BOOK: %s
|
||||
SECTION: %s
|
||||
EXCERPT:
|
||||
---
|
||||
%s
|
||||
---
|
||||
""".formatted(bookTitle, sectionTitle, chunkText);
|
||||
}
|
||||
|
||||
private List<String> normaliseEntities(List<String> raw) {
|
||||
if (raw == null) return List.of();
|
||||
List<String> out = new ArrayList<>();
|
||||
for (String e : raw) {
|
||||
if (e == null) continue;
|
||||
String canonical = e.trim().toLowerCase(Locale.ROOT);
|
||||
if (canonical.isEmpty()) continue;
|
||||
if (!out.contains(canonical)) out.add(canonical);
|
||||
if (out.size() >= MAX_ENTITIES) break;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
private ConceptFacet parseFacet(String raw) {
|
||||
if (raw == null) return ConceptFacet.OTHER;
|
||||
try {
|
||||
return ConceptFacet.valueOf(raw.trim().toUpperCase(Locale.ROOT));
|
||||
} catch (IllegalArgumentException ex) {
|
||||
log.warn("LLM returned unknown facet '{}', defaulting to OTHER", raw);
|
||||
return ConceptFacet.OTHER;
|
||||
}
|
||||
}
|
||||
|
||||
// DTO for Spring AI structured output; facet is read as String so we can defend against bad values
|
||||
public record LlmOutput(List<String> entities, String facet, String summary) {}
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
package com.aiteacher.enrichment;
|
||||
|
||||
import jakarta.persistence.*;
|
||||
import org.hibernate.annotations.JdbcTypeCode;
|
||||
import org.hibernate.type.SqlTypes;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
|
||||
@Entity
|
||||
@Table(name = "chunk_metadata")
|
||||
@org.hibernate.annotations.Check(
|
||||
name = "chunk_metadata_facet_check",
|
||||
constraints = "facet IN ('DEFINITION','ANATOMY','PATHOPHYSIOLOGY','EPIDEMIOLOGY'," +
|
||||
"'CLINICAL_PRESENTATION','IMAGING','CLASSIFICATION','INDICATIONS'," +
|
||||
"'SURGICAL_TECHNIQUE','NONSURGICAL_MANAGEMENT','COMPLICATIONS'," +
|
||||
"'OUTCOMES_FOLLOWUP','OTHER')")
|
||||
public class ChunkMetadataEntity {
|
||||
|
||||
@Id
|
||||
@Column(name = "chunk_id", nullable = false)
|
||||
private UUID chunkId;
|
||||
|
||||
@Column(name = "book_id", nullable = false)
|
||||
private UUID bookId;
|
||||
|
||||
@Column(name = "section_id", nullable = false, length = 200)
|
||||
private String sectionId;
|
||||
|
||||
@Enumerated(EnumType.STRING)
|
||||
@Column(name = "facet", nullable = false, length = 32)
|
||||
private ConceptFacet facet;
|
||||
|
||||
@JdbcTypeCode(SqlTypes.JSON)
|
||||
@Column(name = "entities", nullable = false, columnDefinition = "jsonb")
|
||||
private List<String> entities;
|
||||
|
||||
@Column(name = "summary", nullable = false, columnDefinition = "TEXT")
|
||||
private String summary;
|
||||
|
||||
@Column(name = "model_version", nullable = false, length = 32)
|
||||
private String modelVersion;
|
||||
|
||||
@Column(name = "enriched_at", nullable = false)
|
||||
private Instant enrichedAt;
|
||||
|
||||
protected ChunkMetadataEntity() {}
|
||||
|
||||
public ChunkMetadataEntity(UUID chunkId, UUID bookId, String sectionId,
|
||||
ConceptFacet facet, List<String> entities, String summary,
|
||||
String modelVersion, Instant enrichedAt) {
|
||||
this.chunkId = chunkId;
|
||||
this.bookId = bookId;
|
||||
this.sectionId = sectionId;
|
||||
this.facet = facet;
|
||||
this.entities = entities;
|
||||
this.summary = summary;
|
||||
this.modelVersion = modelVersion;
|
||||
this.enrichedAt = enrichedAt;
|
||||
}
|
||||
|
||||
public UUID getChunkId() { return chunkId; }
|
||||
public UUID getBookId() { return bookId; }
|
||||
public String getSectionId() { return sectionId; }
|
||||
public ConceptFacet getFacet() { return facet; }
|
||||
public List<String> getEntities() { return entities; }
|
||||
public String getSummary() { return summary; }
|
||||
public String getModelVersion() { return modelVersion; }
|
||||
public Instant getEnrichedAt() { return enrichedAt; }
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
package com.aiteacher.enrichment;
|
||||
|
||||
import org.springframework.data.jpa.repository.JpaRepository;
|
||||
import org.springframework.data.jpa.repository.Query;
|
||||
import org.springframework.data.repository.query.Param;
|
||||
import org.springframework.stereotype.Repository;
|
||||
import org.springframework.transaction.annotation.Transactional;
|
||||
|
||||
import java.util.Collection;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
|
||||
@Repository
|
||||
public interface ChunkMetadataRepository extends JpaRepository<ChunkMetadataEntity, UUID> {
|
||||
|
||||
long countByBookId(UUID bookId);
|
||||
|
||||
@Query(value = """
|
||||
SELECT * FROM chunk_metadata
|
||||
WHERE book_id = :bookId
|
||||
AND entities @> to_jsonb(CAST(:entity AS text))
|
||||
""", nativeQuery = true)
|
||||
List<ChunkMetadataEntity> findByBookIdAndEntityContains(@Param("bookId") UUID bookId,
|
||||
@Param("entity") String entity);
|
||||
|
||||
@Query(value = """
|
||||
SELECT * FROM chunk_metadata
|
||||
WHERE entities @> to_jsonb(CAST(:entity AS text))
|
||||
""", nativeQuery = true)
|
||||
List<ChunkMetadataEntity> findByEntityContains(@Param("entity") String entity);
|
||||
|
||||
List<ChunkMetadataEntity> findByChunkIdIn(Collection<UUID> chunkIds);
|
||||
|
||||
@Transactional
|
||||
void deleteByBookId(UUID bookId);
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
package com.aiteacher.enrichment;
|
||||
|
||||
public enum ConceptFacet {
|
||||
DEFINITION("Definition & Overview"),
|
||||
ANATOMY("Anatomy"),
|
||||
PATHOPHYSIOLOGY("Pathophysiology"),
|
||||
EPIDEMIOLOGY("Epidemiology"),
|
||||
CLINICAL_PRESENTATION("Clinical Presentation"),
|
||||
IMAGING("Imaging"),
|
||||
CLASSIFICATION("Classification & Grading"),
|
||||
INDICATIONS("Indications & Patient Selection"),
|
||||
SURGICAL_TECHNIQUE("Surgical Technique"),
|
||||
NONSURGICAL_MANAGEMENT("Non-surgical Management"),
|
||||
COMPLICATIONS("Complications"),
|
||||
OUTCOMES_FOLLOWUP("Outcomes & Follow-up"),
|
||||
OTHER("Other");
|
||||
|
||||
private final String displayTitle;
|
||||
|
||||
ConceptFacet(String displayTitle) {
|
||||
this.displayTitle = displayTitle;
|
||||
}
|
||||
|
||||
public String displayTitle() {
|
||||
return displayTitle;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,138 @@
|
||||
package com.aiteacher.enrichment;
|
||||
|
||||
import com.aiteacher.document.SectionEntity;
|
||||
import com.aiteacher.document.SectionRepository;
|
||||
import com.fasterxml.jackson.core.JsonProcessingException;
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
import org.springframework.scheduling.annotation.Async;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import java.util.UUID;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
|
||||
@Service
|
||||
public class EnrichmentBackfillService {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(EnrichmentBackfillService.class);
|
||||
|
||||
private final JdbcTemplate jdbcTemplate;
|
||||
private final ChunkEnrichmentService enrichmentService;
|
||||
private final ChunkMetadataRepository metadataRepository;
|
||||
private final SectionRepository sectionRepository;
|
||||
private final ObjectMapper objectMapper;
|
||||
private final Map<UUID, BackfillProgress> progressByBook = new ConcurrentHashMap<>();
|
||||
|
||||
public EnrichmentBackfillService(JdbcTemplate jdbcTemplate,
|
||||
ChunkEnrichmentService enrichmentService,
|
||||
ChunkMetadataRepository metadataRepository,
|
||||
SectionRepository sectionRepository,
|
||||
ObjectMapper objectMapper) {
|
||||
this.jdbcTemplate = jdbcTemplate;
|
||||
this.enrichmentService = enrichmentService;
|
||||
this.metadataRepository = metadataRepository;
|
||||
this.sectionRepository = sectionRepository;
|
||||
this.objectMapper = objectMapper;
|
||||
}
|
||||
|
||||
public BackfillProgress getProgress(UUID bookId) {
|
||||
return progressByBook.getOrDefault(bookId, BackfillProgress.idle());
|
||||
}
|
||||
|
||||
@Async
|
||||
public void backfillBook(UUID bookId, String bookTitle) {
|
||||
List<Document> pending = listUnenrichedChunks(bookId);
|
||||
int total = pending.size();
|
||||
progressByBook.put(bookId, new BackfillProgress("RUNNING", total, 0, null));
|
||||
log.info("Backfill starting for book {} — {} chunks pending", bookId, total);
|
||||
|
||||
int done = 0;
|
||||
Map<String, SectionEntity> sectionCache = new HashMap<>();
|
||||
for (Document chunk : pending) {
|
||||
try {
|
||||
String sectionId = (String) chunk.getMetadata().get("section_id");
|
||||
SectionEntity section = sectionId != null
|
||||
? sectionCache.computeIfAbsent(sectionId,
|
||||
id -> sectionRepository.findById(id).orElse(null))
|
||||
: null;
|
||||
ChunkEnrichmentResult result = enrichmentService.enrich(chunk.getText(), section, bookTitle);
|
||||
UUID chunkId = UUID.fromString(chunk.getId());
|
||||
metadataRepository.save(new ChunkMetadataEntity(
|
||||
chunkId, bookId, sectionId != null ? sectionId : "",
|
||||
result.facet(), result.entities(), result.summary(),
|
||||
ChunkEnrichmentService.MODEL_VERSION, Instant.now()));
|
||||
} catch (Exception ex) {
|
||||
log.warn("Backfill failed for chunk {} of book {}: {}", chunk.getId(), bookId, ex.getMessage());
|
||||
}
|
||||
done++;
|
||||
progressByBook.put(bookId, new BackfillProgress("RUNNING", total, done, null));
|
||||
}
|
||||
progressByBook.put(bookId, new BackfillProgress("COMPLETED", total, done, null));
|
||||
log.info("Backfill finished for book {} — {}/{} enriched", bookId, done, total);
|
||||
}
|
||||
|
||||
private List<Document> listUnenrichedChunks(UUID bookId) {
|
||||
// Left anti-join against chunk_metadata so re-runs are cheap.
|
||||
String sql = """
|
||||
SELECT vs.id, vs.content, vs.metadata::text AS metadata_text
|
||||
FROM vector_store vs
|
||||
LEFT JOIN chunk_metadata cm ON cm.chunk_id = vs.id
|
||||
WHERE vs.metadata->>'book_id' = ?
|
||||
AND vs.metadata->>'type' = 'TEXT'
|
||||
AND cm.chunk_id IS NULL
|
||||
""";
|
||||
return jdbcTemplate.query(sql, (rs, rowNum) -> {
|
||||
String id = rs.getString("id");
|
||||
String content = rs.getString("content");
|
||||
String metaJson = rs.getString("metadata_text");
|
||||
Map<String, Object> meta = parseMetadata(metaJson);
|
||||
return new Document(id, content != null ? content : "", meta);
|
||||
}, bookId.toString());
|
||||
}
|
||||
|
||||
private Map<String, Object> parseMetadata(String json) {
|
||||
if (json == null || json.isBlank()) return Map.of();
|
||||
try {
|
||||
JsonNode node = objectMapper.readTree(json);
|
||||
Map<String, Object> out = new HashMap<>();
|
||||
node.properties().forEach(e -> {
|
||||
JsonNode v = e.getValue();
|
||||
if (v.isTextual()) out.put(e.getKey(), v.asText());
|
||||
else if (v.isInt()) out.put(e.getKey(), v.asInt());
|
||||
else if (v.isLong()) out.put(e.getKey(), v.asLong());
|
||||
else if (v.isBoolean()) out.put(e.getKey(), v.asBoolean());
|
||||
else out.put(e.getKey(), v.toString());
|
||||
});
|
||||
return out;
|
||||
} catch (JsonProcessingException ex) {
|
||||
log.warn("Failed to parse vector_store metadata JSON: {}", ex.getMessage());
|
||||
return Map.of();
|
||||
}
|
||||
}
|
||||
|
||||
public Optional<Integer> countEnrichedChunks(UUID bookId) {
|
||||
return Optional.of((int) metadataRepository.countByBookId(bookId));
|
||||
}
|
||||
|
||||
public int countTotalTextChunks(UUID bookId) {
|
||||
Integer n = jdbcTemplate.queryForObject(
|
||||
"SELECT COUNT(*) FROM vector_store WHERE metadata->>'book_id' = ? AND metadata->>'type' = 'TEXT'",
|
||||
Integer.class, bookId.toString());
|
||||
return n != null ? n : 0;
|
||||
}
|
||||
|
||||
public record BackfillProgress(String status, int chunksTotal, int chunksEnriched, String errorMessage) {
|
||||
public static BackfillProgress idle() {
|
||||
return new BackfillProgress("IDLE", 0, 0, null);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
package com.aiteacher.enrichment;
|
||||
|
||||
import com.aiteacher.book.Book;
|
||||
import com.aiteacher.book.BookRepository;
|
||||
import org.springframework.http.HttpStatus;
|
||||
import org.springframework.http.ResponseEntity;
|
||||
import org.springframework.web.bind.annotation.*;
|
||||
|
||||
import java.util.NoSuchElementException;
|
||||
import java.util.UUID;
|
||||
|
||||
@RestController
|
||||
@RequestMapping("/api/v1/admin/books/{id}/enrich")
|
||||
public class EnrichmentController {
|
||||
|
||||
private final BookRepository bookRepository;
|
||||
private final EnrichmentBackfillService backfillService;
|
||||
|
||||
public EnrichmentController(BookRepository bookRepository,
|
||||
EnrichmentBackfillService backfillService) {
|
||||
this.bookRepository = bookRepository;
|
||||
this.backfillService = backfillService;
|
||||
}
|
||||
|
||||
@PostMapping
|
||||
public ResponseEntity<EnrichmentBackfillService.BackfillProgress> start(@PathVariable UUID id) {
|
||||
Book book = bookRepository.findById(id)
|
||||
.orElseThrow(() -> new NoSuchElementException("Book not found."));
|
||||
backfillService.backfillBook(id, book.getTitle());
|
||||
int total = backfillService.countTotalTextChunks(id);
|
||||
int enriched = backfillService.countEnrichedChunks(id).orElse(0);
|
||||
return ResponseEntity.status(HttpStatus.ACCEPTED)
|
||||
.body(new EnrichmentBackfillService.BackfillProgress("RUNNING", total, enriched, null));
|
||||
}
|
||||
|
||||
@GetMapping
|
||||
public ResponseEntity<EnrichmentBackfillService.BackfillProgress> status(@PathVariable UUID id) {
|
||||
bookRepository.findById(id)
|
||||
.orElseThrow(() -> new NoSuchElementException("Book not found."));
|
||||
EnrichmentBackfillService.BackfillProgress progress = backfillService.getProgress(id);
|
||||
if ("IDLE".equals(progress.status())) {
|
||||
int total = backfillService.countTotalTextChunks(id);
|
||||
int enriched = backfillService.countEnrichedChunks(id).orElse(0);
|
||||
progress = new EnrichmentBackfillService.BackfillProgress(
|
||||
enriched >= total && total > 0 ? "COMPLETED" : "IDLE",
|
||||
total, enriched, null);
|
||||
}
|
||||
return ResponseEntity.ok(progress);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
package com.aiteacher.figure;
|
||||
|
||||
import java.awt.image.BufferedImage;
|
||||
import java.util.UUID;
|
||||
|
||||
public interface FigureStorageService {
|
||||
|
||||
/**
|
||||
* Saves an extracted image to S3 and returns the object key stored in the database.
|
||||
*/
|
||||
String save(UUID bookId, String figureId, BufferedImage image);
|
||||
|
||||
/**
|
||||
* Downloads the image bytes for the given S3 object key.
|
||||
*/
|
||||
byte[] getBytes(String key);
|
||||
|
||||
/**
|
||||
* Returns a presigned GET URL valid for 1 hour for the given S3 object key.
|
||||
*/
|
||||
String presignedUrl(String key);
|
||||
|
||||
/**
|
||||
* Deletes all figure objects for the given book.
|
||||
*/
|
||||
void deleteAll(UUID bookId);
|
||||
}
|
||||
@@ -0,0 +1,132 @@
|
||||
package com.aiteacher.figure;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.stereotype.Service;
|
||||
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
|
||||
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
|
||||
import software.amazon.awssdk.core.sync.RequestBody;
|
||||
import software.amazon.awssdk.regions.Region;
|
||||
import software.amazon.awssdk.services.s3.S3Client;
|
||||
import software.amazon.awssdk.services.s3.S3Configuration;
|
||||
import software.amazon.awssdk.services.s3.model.*;
|
||||
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
|
||||
import software.amazon.awssdk.services.s3.presigner.model.GetObjectPresignRequest;
|
||||
import software.amazon.awssdk.services.s3.model.S3Object;
|
||||
|
||||
import javax.imageio.ImageIO;
|
||||
import java.awt.image.BufferedImage;
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.io.IOException;
|
||||
import java.net.URI;
|
||||
import java.time.Duration;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
|
||||
@Service
|
||||
public class S3FigureStorageService implements FigureStorageService {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(S3FigureStorageService.class);
|
||||
|
||||
private final S3Client s3;
|
||||
private final S3Presigner presigner;
|
||||
private final String bucket;
|
||||
|
||||
public S3FigureStorageService(
|
||||
@Value("${app.figure-storage.endpoint}") String endpoint,
|
||||
@Value("${app.figure-storage.region}") String region,
|
||||
@Value("${app.figure-storage.bucket}") String bucket,
|
||||
@Value("${app.figure-storage.access-key-id}") String accessKeyId,
|
||||
@Value("${app.figure-storage.secret-access-key}") String secretKey) {
|
||||
this.bucket = bucket;
|
||||
URI endpointUri = URI.create(endpoint);
|
||||
StaticCredentialsProvider credentials = StaticCredentialsProvider.create(
|
||||
AwsBasicCredentials.create(accessKeyId, secretKey));
|
||||
Region awsRegion = Region.of(region);
|
||||
|
||||
S3Configuration s3Config = S3Configuration.builder()
|
||||
.pathStyleAccessEnabled(true)
|
||||
.build();
|
||||
|
||||
this.s3 = S3Client.builder()
|
||||
.endpointOverride(endpointUri)
|
||||
.region(awsRegion)
|
||||
.credentialsProvider(credentials)
|
||||
.serviceConfiguration(s3Config)
|
||||
.build();
|
||||
|
||||
this.presigner = S3Presigner.builder()
|
||||
.endpointOverride(endpointUri)
|
||||
.region(awsRegion)
|
||||
.credentialsProvider(credentials)
|
||||
.serviceConfiguration(s3Config)
|
||||
.build();
|
||||
}
|
||||
|
||||
@Override
|
||||
public String save(UUID bookId, String figureId, BufferedImage image) {
|
||||
String key = "figures/" + bookId + "/" + figureId + ".png";
|
||||
try {
|
||||
ByteArrayOutputStream out = new ByteArrayOutputStream();
|
||||
ImageIO.write(image, "PNG", out);
|
||||
byte[] bytes = out.toByteArray();
|
||||
|
||||
s3.putObject(
|
||||
PutObjectRequest.builder().bucket(bucket).key(key)
|
||||
.contentType("image/png").contentLength((long) bytes.length).build(),
|
||||
RequestBody.fromBytes(bytes));
|
||||
return key;
|
||||
} catch (IOException ex) {
|
||||
throw new RuntimeException("Failed to encode figure " + figureId, ex);
|
||||
} catch (S3Exception ex) {
|
||||
throw new RuntimeException("Failed to upload figure " + figureId + " to S3", ex);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public byte[] getBytes(String key) {
|
||||
try {
|
||||
return s3.getObjectAsBytes(
|
||||
GetObjectRequest.builder().bucket(bucket).key(key).build()).asByteArray();
|
||||
} catch (S3Exception ex) {
|
||||
throw new RuntimeException("Failed to download figure from S3: " + key, ex);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public String presignedUrl(String key) {
|
||||
GetObjectPresignRequest request = GetObjectPresignRequest.builder()
|
||||
.signatureDuration(Duration.ofHours(1))
|
||||
.getObjectRequest(r -> r.bucket(bucket).key(key))
|
||||
.build();
|
||||
return presigner.presignGetObject(request).url().toString();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void deleteAll(UUID bookId) {
|
||||
String prefix = "figures/" + bookId + "/";
|
||||
try {
|
||||
List<ObjectIdentifier> toDelete = new ArrayList<>();
|
||||
ListObjectsV2Request listRequest = ListObjectsV2Request.builder()
|
||||
.bucket(bucket).prefix(prefix).build();
|
||||
|
||||
s3.listObjectsV2Paginator(listRequest).stream()
|
||||
.flatMap(page -> page.contents().stream())
|
||||
.map(S3Object::key)
|
||||
.map(k -> ObjectIdentifier.builder().key(k).build())
|
||||
.forEach(toDelete::add);
|
||||
|
||||
if (toDelete.isEmpty()) return;
|
||||
|
||||
s3.deleteObjects(DeleteObjectsRequest.builder()
|
||||
.bucket(bucket)
|
||||
.delete(Delete.builder().objects(toDelete).build())
|
||||
.build());
|
||||
log.info("Deleted {} figures from S3 for book {}", toDelete.size(), bookId);
|
||||
} catch (S3Exception ex) {
|
||||
log.warn("Could not fully delete figures for book {} from S3: {}", bookId, ex.getMessage());
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
package com.aiteacher.retrieval;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Set;
|
||||
import java.util.regex.Matcher;
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
/**
|
||||
* Post-processes generated answers to strip citation labels that do not
|
||||
* correspond to any passage retrieved for the current query, preventing
|
||||
* hallucinated source references from reaching the user.
|
||||
*/
|
||||
@Service
|
||||
public class CitationValidatorService {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(CitationValidatorService.class);
|
||||
|
||||
/** Matches citation labels of the form [S1], [F2], [S12], etc. */
|
||||
private static final Pattern CITATION_PATTERN = Pattern.compile("\\[(S|F)\\d+\\]");
|
||||
|
||||
/**
|
||||
* Removes any {@code [Sx]} / {@code [Fx]} citation in {@code generatedAnswer}
|
||||
* whose label is not contained in {@code validLabels}.
|
||||
*
|
||||
* @param generatedAnswer raw model output
|
||||
* @param validLabels set of labels present in the retrieved context
|
||||
* @return cleaned answer text with hallucinated citations removed
|
||||
*/
|
||||
public String validate(String generatedAnswer, Set<String> validLabels) {
|
||||
if (generatedAnswer == null) return "";
|
||||
|
||||
Matcher matcher = CITATION_PATTERN.matcher(generatedAnswer);
|
||||
List<String> removed = new ArrayList<>();
|
||||
StringBuffer sb = new StringBuffer();
|
||||
|
||||
while (matcher.find()) {
|
||||
String label = matcher.group();
|
||||
String inner = label.substring(1, label.length() - 1); // strip [ ]
|
||||
if (validLabels.contains(inner)) {
|
||||
matcher.appendReplacement(sb, Matcher.quoteReplacement(label));
|
||||
} else {
|
||||
removed.add(inner);
|
||||
matcher.appendReplacement(sb, "");
|
||||
}
|
||||
}
|
||||
matcher.appendTail(sb);
|
||||
|
||||
if (!removed.isEmpty()) {
|
||||
log.warn("Stripped hallucinated citations: {}", removed);
|
||||
}
|
||||
|
||||
return sb.toString();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
package com.aiteacher.retrieval;
|
||||
|
||||
/**
|
||||
* Value object holding the original user query alongside its clinically
|
||||
* rewritten variant used for vector-store retrieval.
|
||||
*/
|
||||
public record ExpandedQuery(String original, String rewritten) {}
|
||||
@@ -0,0 +1,27 @@
|
||||
package com.aiteacher.retrieval;
|
||||
|
||||
import com.aiteacher.document.FigureEntity;
|
||||
import com.aiteacher.document.SectionEntity;
|
||||
|
||||
import java.util.HashSet;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
|
||||
/**
|
||||
* Value object produced when building the LLM context prompt.
|
||||
* Maps short ref-labels (S1, S2… / F1, F2…) to their source entities
|
||||
* and carries the fully formatted prompt text.
|
||||
*/
|
||||
public record LabelledContext(
|
||||
Map<String, SectionEntity> sectionLabels,
|
||||
Map<String, FigureEntity> figureLabels,
|
||||
String promptText) {
|
||||
|
||||
/** Returns the union of all valid citation labels for this context. */
|
||||
public Set<String> allLabels() {
|
||||
Set<String> labels = new HashSet<>();
|
||||
labels.addAll(sectionLabels.keySet());
|
||||
labels.addAll(figureLabels.keySet());
|
||||
return labels;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,111 @@
|
||||
package com.aiteacher.retrieval;
|
||||
|
||||
import com.aiteacher.document.*;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.vectorstore.SearchRequest;
|
||||
import org.springframework.ai.vectorstore.VectorStore;
|
||||
import org.springframework.ai.vectorstore.filter.FilterExpressionBuilder;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.util.*;
|
||||
|
||||
/**
|
||||
* Dual-modality retriever: searches text chunks and figure captions independently,
|
||||
* then expands text hits to their parent sections and merges linked figures.
|
||||
*/
|
||||
@Service
|
||||
public class NeurosurgeryRetriever {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(NeurosurgeryRetriever.class);
|
||||
|
||||
private static final int TEXT_TOP_K = 5;
|
||||
private static final int FIGURE_TOP_K = 3;
|
||||
|
||||
private final VectorStore vectorStore;
|
||||
private final SectionRepository sectionRepository;
|
||||
private final FigureRepository figureRepository;
|
||||
private final ChunkFigureRefRepository chunkFigureRefRepository;
|
||||
|
||||
public NeurosurgeryRetriever(VectorStore vectorStore,
|
||||
SectionRepository sectionRepository,
|
||||
FigureRepository figureRepository,
|
||||
ChunkFigureRefRepository chunkFigureRefRepository) {
|
||||
this.vectorStore = vectorStore;
|
||||
this.sectionRepository = sectionRepository;
|
||||
this.figureRepository = figureRepository;
|
||||
this.chunkFigureRefRepository = chunkFigureRefRepository;
|
||||
}
|
||||
|
||||
public RetrievalResult retrieve(String query, UUID bookId) {
|
||||
FilterExpressionBuilder b = new FilterExpressionBuilder();
|
||||
|
||||
// 1. Text chunk search
|
||||
List<Document> textHits = vectorStore.similaritySearch(
|
||||
SearchRequest.builder()
|
||||
.query(query)
|
||||
.topK(TEXT_TOP_K)
|
||||
.filterExpression(b.and(
|
||||
b.eq("type", "TEXT"),
|
||||
b.eq("book_id", bookId.toString())
|
||||
).build())
|
||||
.build()
|
||||
);
|
||||
|
||||
// 2. Figure caption search (independent topK)
|
||||
List<Document> figureHits = vectorStore.similaritySearch(
|
||||
SearchRequest.builder()
|
||||
.query(query)
|
||||
.topK(FIGURE_TOP_K)
|
||||
.filterExpression(b.and(
|
||||
b.eq("type", "FIGURE"),
|
||||
b.eq("book_id", bookId.toString())
|
||||
).build())
|
||||
.build()
|
||||
);
|
||||
|
||||
// 3. Expand text chunks to parent sections from Postgres
|
||||
List<String> sectionIds = textHits.stream()
|
||||
.map(d -> (String) d.getMetadata().get("section_id"))
|
||||
.filter(Objects::nonNull)
|
||||
.distinct()
|
||||
.toList();
|
||||
List<SectionEntity> sections = sectionIds.isEmpty()
|
||||
? List.of()
|
||||
: sectionRepository.findAllById(sectionIds);
|
||||
|
||||
// 4. Fetch figures explicitly linked to retrieved chunks
|
||||
List<UUID> chunkIds = textHits.stream()
|
||||
.map(d -> {
|
||||
try { return UUID.fromString(d.getId()); }
|
||||
catch (Exception e) { return null; }
|
||||
})
|
||||
.filter(Objects::nonNull)
|
||||
.toList();
|
||||
List<String> linkedFigureIds = chunkIds.isEmpty()
|
||||
? List.of()
|
||||
: chunkFigureRefRepository.findByChunkIdIn(chunkIds)
|
||||
.stream().map(ChunkFigureRefEntity::getFigureId).distinct().toList();
|
||||
List<FigureEntity> linkedFigures = linkedFigureIds.isEmpty()
|
||||
? List.of()
|
||||
: figureRepository.findAllById(linkedFigureIds);
|
||||
|
||||
// 5. Collect figures from semantic figure search
|
||||
List<String> semanticFigureIds = figureHits.stream()
|
||||
.map(d -> (String) d.getMetadata().get("figure_id"))
|
||||
.filter(Objects::nonNull)
|
||||
.toList();
|
||||
List<FigureEntity> semanticFigures = semanticFigureIds.isEmpty()
|
||||
? List.of()
|
||||
: figureRepository.findAllById(semanticFigureIds);
|
||||
|
||||
// 6. Merge and deduplicate figures by figureId (linked figures take precedence)
|
||||
Map<String, FigureEntity> merged = new LinkedHashMap<>();
|
||||
linkedFigures.forEach(f -> merged.put(f.getId(), f));
|
||||
semanticFigures.forEach(f -> merged.putIfAbsent(f.getId(), f));
|
||||
|
||||
log.debug("Retrieved {} sections, {} figures for query", sections.size(), merged.size());
|
||||
return new RetrievalResult(sections, new ArrayList<>(merged.values()));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
package com.aiteacher.retrieval;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.ai.chat.client.ChatClient;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
/**
|
||||
* Rewrites a user query into precise clinical/surgical terminology so that
|
||||
* vector-store retrieval can match textbook language even when the user's
|
||||
* phrasing differs from the documentation vocabulary.
|
||||
*/
|
||||
@Service
|
||||
public class QueryExpansionService {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(QueryExpansionService.class);
|
||||
|
||||
private static final String EXPANSION_PROMPT = """
|
||||
Rewrite the following question using precise medical and surgical terminology \
|
||||
as it would appear in a neurosurgery textbook index. \
|
||||
Output only the rewritten question, nothing else.
|
||||
Question: %s""";
|
||||
|
||||
private final ChatClient chatClient;
|
||||
|
||||
public QueryExpansionService(ChatClient chatClient) {
|
||||
this.chatClient = chatClient;
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns an {@link ExpandedQuery} whose {@code rewritten} field contains
|
||||
* the clinically rephrased version of {@code query}.
|
||||
*/
|
||||
public ExpandedQuery expand(String query) {
|
||||
String rewritten = chatClient.prompt()
|
||||
.user(EXPANSION_PROMPT.formatted(query))
|
||||
.call()
|
||||
.content();
|
||||
|
||||
if (rewritten == null || rewritten.isBlank()) {
|
||||
rewritten = query;
|
||||
}
|
||||
|
||||
log.debug("Query expanded: '{}' → '{}'", query, rewritten);
|
||||
return new ExpandedQuery(query, rewritten);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
package com.aiteacher.retrieval;
|
||||
|
||||
import com.aiteacher.document.FigureEntity;
|
||||
import com.aiteacher.document.SectionEntity;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
public record RetrievalResult(
|
||||
List<SectionEntity> parentSections,
|
||||
List<FigureEntity> figures
|
||||
) {}
|
||||
@@ -0,0 +1,7 @@
|
||||
package com.aiteacher.topic;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.UUID;
|
||||
|
||||
public record SavedSummaryItem(UUID id, int summaryNumber, Instant generatedAt) {
|
||||
}
|
||||
@@ -5,6 +5,7 @@ import org.springframework.web.bind.annotation.*;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.NoSuchElementException;
|
||||
import java.util.UUID;
|
||||
|
||||
@RestController
|
||||
@RequestMapping("/api/v1/topics")
|
||||
@@ -25,11 +26,30 @@ public class TopicController {
|
||||
}
|
||||
|
||||
@PostMapping("/{id}/summary")
|
||||
public ResponseEntity<TopicSummaryResponse> generateSummary(@PathVariable String id) {
|
||||
public ResponseEntity<TopicSummaryResponse> generateSummary(
|
||||
@PathVariable String id,
|
||||
@RequestParam(defaultValue = "en") String language) {
|
||||
Topic topic = topicRepository.findById(id)
|
||||
.orElseThrow(() -> new NoSuchElementException("Topic not found."));
|
||||
|
||||
TopicSummaryResponse response = topicSummaryService.generateSummary(topic);
|
||||
TopicSummaryResponse response = topicSummaryService.generateSummary(topic, language);
|
||||
return ResponseEntity.ok(response);
|
||||
}
|
||||
|
||||
@GetMapping("/{id}/summaries")
|
||||
public ResponseEntity<List<SavedSummaryItem>> listSummaries(@PathVariable String id) {
|
||||
topicRepository.findById(id)
|
||||
.orElseThrow(() -> new NoSuchElementException("Topic not found."));
|
||||
|
||||
return ResponseEntity.ok(topicSummaryService.listSummaries(id));
|
||||
}
|
||||
|
||||
@GetMapping("/{id}/summaries/{summaryId}")
|
||||
public ResponseEntity<TopicSummaryResponse> getSummary(@PathVariable String id,
|
||||
@PathVariable UUID summaryId) {
|
||||
topicRepository.findById(id)
|
||||
.orElseThrow(() -> new NoSuchElementException("Topic not found."));
|
||||
|
||||
return ResponseEntity.ok(topicSummaryService.getSummary(summaryId));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
package com.aiteacher.topic;
|
||||
|
||||
import jakarta.persistence.Column;
|
||||
import jakarta.persistence.Entity;
|
||||
import jakarta.persistence.GeneratedValue;
|
||||
import jakarta.persistence.GenerationType;
|
||||
import jakarta.persistence.Id;
|
||||
import jakarta.persistence.Table;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.UUID;
|
||||
|
||||
@Entity
|
||||
@Table(name = "topic_summary")
|
||||
public class TopicSummaryEntity {
|
||||
|
||||
@Id
|
||||
@GeneratedValue(strategy = GenerationType.UUID)
|
||||
private UUID id;
|
||||
|
||||
@Column(name = "topic_id", nullable = false)
|
||||
private String topicId;
|
||||
|
||||
@Column(name = "summary_number", nullable = false)
|
||||
private int summaryNumber;
|
||||
|
||||
@Column(nullable = false, columnDefinition = "TEXT")
|
||||
private String summary;
|
||||
|
||||
@Column(name = "sources_json", nullable = false, columnDefinition = "TEXT")
|
||||
private String sourcesJson;
|
||||
|
||||
@Column(name = "generated_at", nullable = false)
|
||||
private Instant generatedAt;
|
||||
|
||||
protected TopicSummaryEntity() {}
|
||||
|
||||
public TopicSummaryEntity(String topicId, int summaryNumber, String summary,
|
||||
String sourcesJson, Instant generatedAt) {
|
||||
this.topicId = topicId;
|
||||
this.summaryNumber = summaryNumber;
|
||||
this.summary = summary;
|
||||
this.sourcesJson = sourcesJson;
|
||||
this.generatedAt = generatedAt;
|
||||
}
|
||||
|
||||
public UUID getId() { return id; }
|
||||
public String getTopicId() { return topicId; }
|
||||
public int getSummaryNumber() { return summaryNumber; }
|
||||
public String getSummary() { return summary; }
|
||||
public String getSourcesJson() { return sourcesJson; }
|
||||
public Instant getGeneratedAt() { return generatedAt; }
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
package com.aiteacher.topic;
|
||||
|
||||
import org.springframework.data.jpa.repository.JpaRepository;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
|
||||
public interface TopicSummaryRepository extends JpaRepository<TopicSummaryEntity, UUID> {
|
||||
|
||||
List<TopicSummaryEntity> findByTopicIdOrderBySummaryNumberAsc(String topicId);
|
||||
|
||||
long countByTopicId(String topicId);
|
||||
}
|
||||
@@ -2,8 +2,11 @@ package com.aiteacher.topic;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
|
||||
public record TopicSummaryResponse(
|
||||
UUID id,
|
||||
int summaryNumber,
|
||||
String topicId,
|
||||
String topicName,
|
||||
String summary,
|
||||
@@ -11,8 +14,17 @@ public record TopicSummaryResponse(
|
||||
Instant generatedAt
|
||||
) {
|
||||
public record SourceReference(
|
||||
String type,
|
||||
String refLabel,
|
||||
String bookId,
|
||||
String bookTitle,
|
||||
Integer page
|
||||
Integer page,
|
||||
String chunkText,
|
||||
String figureId,
|
||||
String label,
|
||||
String caption,
|
||||
String figureType,
|
||||
String imageUrl
|
||||
) {
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,21 +1,25 @@
|
||||
package com.aiteacher.topic;
|
||||
|
||||
import com.aiteacher.book.Book;
|
||||
import com.aiteacher.book.BookRepository;
|
||||
import com.aiteacher.book.BookStatus;
|
||||
import com.aiteacher.book.NoKnowledgeSourceException;
|
||||
import com.aiteacher.document.FigureEntity;
|
||||
import com.aiteacher.document.SectionEntity;
|
||||
import com.aiteacher.retrieval.NeurosurgeryRetriever;
|
||||
import com.aiteacher.retrieval.RetrievalResult;
|
||||
import com.fasterxml.jackson.core.JsonProcessingException;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.ai.chat.client.ChatClient;
|
||||
import org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor;
|
||||
import org.springframework.ai.chat.model.ChatResponse;
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.vectorstore.VectorStore;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.NoSuchElementException;
|
||||
import java.util.UUID;
|
||||
|
||||
@Service
|
||||
public class TopicSummaryService {
|
||||
@@ -23,86 +27,222 @@ public class TopicSummaryService {
|
||||
private static final Logger log = LoggerFactory.getLogger(TopicSummaryService.class);
|
||||
|
||||
private static final String SYSTEM_PROMPT = """
|
||||
You are an expert neurosurgery educator. Your role is to provide accurate,
|
||||
clinically relevant summaries based ONLY on the content retrieved from the
|
||||
uploaded medical textbooks. Do not use any knowledge outside the provided context.
|
||||
You are an expert neurosurgery educator. Your role is to provide accurate, detailed but synthetically concise educational reports on neurosurgery topics, based on the content retrieved from the uploaded medical textbooks. Your audience is highly experienced neurosurgeons, who are looking for a comprehensive yet digestible overview of a specific topic.
|
||||
When generating reports, your primary goal is to distill the most important and clinically relevant information about the topic. This includes key concepts, anatomical details, surgical techniques, clinical considerations, and any other information that would be essential for a neurosurgeon to understand the topic thoroughly.
|
||||
Base your reports on uploaded medical textbooks. Do not use any knowledge outside the provided context.
|
||||
|
||||
When answering:
|
||||
- Structure your response clearly with key points
|
||||
- If the context mentions specific book titles and page numbers, reference them
|
||||
- Cite claims using ONLY the reference labels provided in the context (e.g. [S1], [F2]).
|
||||
Do not invent page numbers, section titles, or labels not present in the CONTEXT block.
|
||||
- Figures (labeled [F1], [F2], etc.) are actual images and drawings from the textbook — they will be rendered as inline illustrations in your response. Use them actively to support your explanations: reference a figure when it visually demonstrates anatomy, a surgical step, or a clinical concept you are describing.
|
||||
- If the retrieved context does not contain sufficient information on the topic,
|
||||
explicitly state: "The uploaded books do not contain sufficient information on this topic."
|
||||
- Never hallucinate or fabricate clinical information
|
||||
""";
|
||||
|
||||
private final ChatClient chatClient;
|
||||
private final VectorStore vectorStore;
|
||||
private final BookRepository bookRepository;
|
||||
private final NeurosurgeryRetriever retriever;
|
||||
private final TopicSummaryRepository summaryRepository;
|
||||
private final ObjectMapper objectMapper;
|
||||
|
||||
public TopicSummaryService(ChatClient chatClient, VectorStore vectorStore,
|
||||
BookRepository bookRepository) {
|
||||
public TopicSummaryService(ChatClient chatClient,
|
||||
BookRepository bookRepository,
|
||||
NeurosurgeryRetriever retriever,
|
||||
TopicSummaryRepository summaryRepository,
|
||||
ObjectMapper objectMapper) {
|
||||
this.chatClient = chatClient;
|
||||
this.vectorStore = vectorStore;
|
||||
this.bookRepository = bookRepository;
|
||||
this.retriever = retriever;
|
||||
this.summaryRepository = summaryRepository;
|
||||
this.objectMapper = objectMapper;
|
||||
}
|
||||
|
||||
public TopicSummaryResponse generateSummary(Topic topic) {
|
||||
if (!bookRepository.existsByStatus(BookStatus.READY)) {
|
||||
public TopicSummaryResponse generateSummary(Topic topic, String language) {
|
||||
List<Book> readyBooks = bookRepository.findAll().stream()
|
||||
.filter(b -> b.getStatus() == BookStatus.READY)
|
||||
.toList();
|
||||
|
||||
if (readyBooks.isEmpty()) {
|
||||
throw new NoKnowledgeSourceException(
|
||||
"No books are available as knowledge sources. Please upload and process at least one book.");
|
||||
}
|
||||
|
||||
String question = buildQuestion(topic);
|
||||
|
||||
ChatResponse response = chatClient.prompt()
|
||||
.system(SYSTEM_PROMPT)
|
||||
.advisors(QuestionAnswerAdvisor.builder(vectorStore).build())
|
||||
.user(question)
|
||||
.call()
|
||||
.chatResponse();
|
||||
List<SectionEntity> allSections = new ArrayList<>();
|
||||
List<FigureEntity> allFigures = new ArrayList<>();
|
||||
for (Book book : readyBooks) {
|
||||
RetrievalResult result = retriever.retrieve(question, book.getId());
|
||||
allSections.addAll(result.parentSections());
|
||||
allFigures.addAll(result.figures());
|
||||
}
|
||||
|
||||
String summary = response.getResult().getOutput().getText();
|
||||
List<TopicSummaryResponse.SourceReference> sources = extractSources(response);
|
||||
log.debug("Topic reports for '{}': {} sections, {} figures retrieved",
|
||||
topic.getName(), allSections.size(), allFigures.size());
|
||||
|
||||
String contextPrompt = buildContextPrompt(question, allSections, allFigures, language);
|
||||
String summary = chatClient.prompt()
|
||||
.system(SYSTEM_PROMPT)
|
||||
.user(contextPrompt)
|
||||
.call()
|
||||
.content();
|
||||
|
||||
List<TopicSummaryResponse.SourceReference> sources = buildSources(allSections, allFigures, readyBooks);
|
||||
Instant generatedAt = Instant.now();
|
||||
|
||||
int summaryNumber = (int) summaryRepository.countByTopicId(topic.getId()) + 1;
|
||||
String sourcesJson = serializeSources(sources);
|
||||
TopicSummaryEntity entity = new TopicSummaryEntity(
|
||||
topic.getId(), summaryNumber, summary, sourcesJson, generatedAt);
|
||||
entity = summaryRepository.save(entity);
|
||||
|
||||
return new TopicSummaryResponse(
|
||||
entity.getId(),
|
||||
summaryNumber,
|
||||
topic.getId(),
|
||||
topic.getName(),
|
||||
summary,
|
||||
sources,
|
||||
Instant.now()
|
||||
generatedAt
|
||||
);
|
||||
}
|
||||
|
||||
public List<SavedSummaryItem> listSummaries(String topicId) {
|
||||
return summaryRepository.findByTopicIdOrderBySummaryNumberAsc(topicId).stream()
|
||||
.map(e -> new SavedSummaryItem(e.getId(), e.getSummaryNumber(), e.getGeneratedAt()))
|
||||
.toList();
|
||||
}
|
||||
|
||||
public TopicSummaryResponse getSummary(UUID summaryId) {
|
||||
TopicSummaryEntity entity = summaryRepository.findById(summaryId)
|
||||
.orElseThrow(() -> new NoSuchElementException("Summary not found."));
|
||||
|
||||
List<TopicSummaryResponse.SourceReference> sources = deserializeSources(entity.getSourcesJson());
|
||||
|
||||
return new TopicSummaryResponse(
|
||||
entity.getId(),
|
||||
entity.getSummaryNumber(),
|
||||
entity.getTopicId(),
|
||||
entity.getTopicId(),
|
||||
entity.getSummary(),
|
||||
sources,
|
||||
entity.getGeneratedAt()
|
||||
);
|
||||
}
|
||||
|
||||
private String buildQuestion(Topic topic) {
|
||||
return String.format(
|
||||
"Please provide a comprehensive educational summary of the following neurosurgery topic: " +
|
||||
"%s. Topic description: %s. " +
|
||||
"Include key concepts, clinical considerations, and important details that a neurosurgeon should know.",
|
||||
"Provide a comprehensive educational report of the following neurosurgery topic: " +
|
||||
"%s. Topic description: %s. ",
|
||||
topic.getName(), topic.getDescription()
|
||||
);
|
||||
}
|
||||
|
||||
private List<TopicSummaryResponse.SourceReference> extractSources(ChatResponse response) {
|
||||
List<TopicSummaryResponse.SourceReference> sources = new ArrayList<>();
|
||||
private String buildContextPrompt(String question,
|
||||
List<SectionEntity> sections,
|
||||
List<FigureEntity> figures,
|
||||
String language) {
|
||||
StringBuilder sb = new StringBuilder();
|
||||
|
||||
if (response.getMetadata() != null) {
|
||||
Object retrieved = response.getMetadata().get(QuestionAnswerAdvisor.RETRIEVED_DOCUMENTS);
|
||||
if (retrieved instanceof List<?> docs) {
|
||||
for (Object docObj : docs) {
|
||||
if (docObj instanceof Document doc) {
|
||||
Map<String, Object> metadata = doc.getMetadata();
|
||||
String bookTitle = (String) metadata.get("book_title");
|
||||
Object pageObj = metadata.get("page_number");
|
||||
Integer page = pageObj instanceof Number n ? n.intValue() : null;
|
||||
if (bookTitle != null) {
|
||||
sources.add(new TopicSummaryResponse.SourceReference(bookTitle, page));
|
||||
}
|
||||
}
|
||||
}
|
||||
if (!sections.isEmpty()) {
|
||||
sb.append("CONTEXT:\n\n");
|
||||
for (int i = 0; i < sections.size(); i++) {
|
||||
SectionEntity s = sections.get(i);
|
||||
sb.append("[S").append(i + 1).append("] ")
|
||||
.append(s.getTitle()).append(", p.").append(s.getPageStart()).append("\n");
|
||||
sb.append(s.getFullText()).append("\n\n");
|
||||
}
|
||||
}
|
||||
|
||||
// Deduplicate by bookTitle + page
|
||||
return sources.stream().distinct().toList();
|
||||
if (!figures.isEmpty()) {
|
||||
sb.append("AVAILABLE FIGURES:\n");
|
||||
for (int i = 0; i < figures.size(); i++) {
|
||||
FigureEntity f = figures.get(i);
|
||||
sb.append("[F").append(i + 1).append("] ")
|
||||
.append(f.getLabel() != null ? f.getLabel() : "Figure")
|
||||
.append(" (p.").append(f.getPage()).append("): ")
|
||||
.append(f.getCaption() != null ? f.getCaption() : "")
|
||||
.append("\n");
|
||||
}
|
||||
sb.append("\nWhen referencing diagrams, use their label from the context (e.g. [F1]).\n\n");
|
||||
}
|
||||
|
||||
sb.append("QUESTION:\n").append(question);
|
||||
|
||||
if ("th".equalsIgnoreCase(language)) {
|
||||
sb.append("\n\nIMPORTANT: Write the narrative in Thai. ")
|
||||
.append("Keep all medical, anatomical, surgical, pharmacological, and clinical ")
|
||||
.append("terminology in English (e.g., cerebellopontine angle, glioblastoma, craniotomy, ")
|
||||
.append("dexamethasone). Do NOT translate disease names, anatomical structures, drug names, ")
|
||||
.append("procedures, eponyms, or imaging modalities. Translate only connective prose, ")
|
||||
.append("explanations, and general descriptions. Citation labels [S#]/[F#] stay unchanged. ")
|
||||
.append("The sentinel string for insufficient context must remain exactly: ")
|
||||
.append("\"The uploaded books do not contain sufficient information on this topic.\"");
|
||||
}
|
||||
return sb.toString();
|
||||
}
|
||||
|
||||
private List<TopicSummaryResponse.SourceReference> buildSources(List<SectionEntity> sections,
|
||||
List<FigureEntity> figures,
|
||||
List<Book> readyBooks) {
|
||||
List<TopicSummaryResponse.SourceReference> sources = new ArrayList<>();
|
||||
|
||||
for (int i = 0; i < sections.size(); i++) {
|
||||
SectionEntity s = sections.get(i);
|
||||
Book book = readyBooks.stream()
|
||||
.filter(b -> b.getId().equals(s.getBookId()))
|
||||
.findFirst()
|
||||
.orElse(null);
|
||||
String title = book != null ? book.getTitle() : "Book";
|
||||
String bookId = book != null ? book.getId().toString() : null;
|
||||
sources.add(new TopicSummaryResponse.SourceReference(
|
||||
"TEXT", "S" + (i + 1), bookId, title, s.getPageStart(),
|
||||
truncate(s.getFullText(), 500), null, null, null, null, null));
|
||||
}
|
||||
|
||||
for (int i = 0; i < figures.size(); i++) {
|
||||
FigureEntity f = figures.get(i);
|
||||
Book book = readyBooks.stream()
|
||||
.filter(b -> b.getId().equals(f.getBookId()))
|
||||
.findFirst()
|
||||
.orElse(null);
|
||||
String title = book != null ? book.getTitle() : "Book";
|
||||
String bookId = book != null ? book.getId().toString() : null;
|
||||
String filename = f.getImagePath().substring(f.getImagePath().lastIndexOf('/') + 1);
|
||||
String imageUrl = "/api/v1/figures/" + f.getBookId() + "/" + filename;
|
||||
sources.add(new TopicSummaryResponse.SourceReference(
|
||||
"FIGURE", "F" + (i + 1), bookId, title, f.getPage(),
|
||||
null, f.getId(), f.getLabel(), f.getCaption(),
|
||||
f.getFigureType().name(), imageUrl));
|
||||
}
|
||||
|
||||
return sources;
|
||||
}
|
||||
|
||||
private String serializeSources(List<TopicSummaryResponse.SourceReference> sources) {
|
||||
try {
|
||||
return objectMapper.writeValueAsString(sources);
|
||||
} catch (JsonProcessingException e) {
|
||||
log.warn("Failed to serialize sources, storing empty array", e);
|
||||
return "[]";
|
||||
}
|
||||
}
|
||||
|
||||
private String truncate(String text, int maxChars) {
|
||||
if (text == null) return "";
|
||||
return text.length() <= maxChars ? text : text.substring(0, maxChars) + "…";
|
||||
}
|
||||
|
||||
private List<TopicSummaryResponse.SourceReference> deserializeSources(String json) {
|
||||
try {
|
||||
return objectMapper.readValue(json,
|
||||
objectMapper.getTypeFactory().constructCollectionType(
|
||||
List.class, TopicSummaryResponse.SourceReference.class));
|
||||
} catch (JsonProcessingException e) {
|
||||
log.warn("Failed to deserialize sources from stored JSON", e);
|
||||
return List.of();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7,7 +7,7 @@ spring:
|
||||
|
||||
jpa:
|
||||
hibernate:
|
||||
ddl-auto: update
|
||||
ddl-auto: none
|
||||
show-sql: false
|
||||
properties:
|
||||
hibernate:
|
||||
@@ -27,10 +27,11 @@ spring:
|
||||
index-type: HNSW
|
||||
initialize-schema: false
|
||||
openai:
|
||||
api-key: ${OPENAI_API_KEY}
|
||||
api-key: ${OPENAI_API_KEY:}
|
||||
chat:
|
||||
options:
|
||||
model: gpt-4o
|
||||
model: o4-mini
|
||||
reasoning-effort: high
|
||||
embedding:
|
||||
options:
|
||||
model: "text-embedding-3-small"
|
||||
@@ -47,6 +48,29 @@ spring:
|
||||
max-size: 8
|
||||
queue-capacity: 50
|
||||
|
||||
logging:
|
||||
level:
|
||||
"[org.apache.pdfbox]": ERROR
|
||||
|
||||
app:
|
||||
features:
|
||||
upload-enabled: ${UPLOAD_ENABLED:true}
|
||||
delete-enabled: ${DELETE_ENABLED:true}
|
||||
auth:
|
||||
username: ${APP_AUTH_USERNAME:neurosurgeon}
|
||||
password: ${APP_PASSWORD:changeme}
|
||||
figure-storage:
|
||||
endpoint: ${S3_ENDPOINT:https://s3.immich-ad.ovh}
|
||||
region: ${S3_REGION:garage}
|
||||
bucket: ${S3_BUCKET:aiteacher}
|
||||
access-key-id: ${S3_ACCESS_KEY_ID:}
|
||||
secret-access-key: ${S3_SECRET_ACCESS_KEY:}
|
||||
min-image-size-px: 100
|
||||
embedding:
|
||||
batch-size: 20
|
||||
batch-delay-ms: 2000
|
||||
skip-embedding: false
|
||||
marker:
|
||||
base-url: ${MARKER_BASE_URL:http://192.168.1.105:8000}
|
||||
vision:
|
||||
min-interval-ms: ${VISION_MIN_INTERVAL_MS:2000}
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
-- ============================================================
|
||||
-- V4: Document hierarchy — chapter and section tables
|
||||
-- Supports parent-child retrieval pattern for RAG precision.
|
||||
-- ============================================================
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chapter (
|
||||
id VARCHAR(200) PRIMARY KEY,
|
||||
book_id UUID NOT NULL REFERENCES book(id) ON DELETE CASCADE,
|
||||
number INT NOT NULL DEFAULT 1,
|
||||
title VARCHAR(500),
|
||||
page_start INT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS section (
|
||||
id VARCHAR(200) PRIMARY KEY,
|
||||
chapter_id VARCHAR(200) NOT NULL REFERENCES chapter(id) ON DELETE CASCADE,
|
||||
book_id UUID NOT NULL REFERENCES book(id) ON DELETE CASCADE,
|
||||
number VARCHAR(50),
|
||||
title VARCHAR(500),
|
||||
page_start INT NOT NULL,
|
||||
page_end INT NOT NULL,
|
||||
full_text TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_section_book ON section(book_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_section_chapter ON section(chapter_id);
|
||||
@@ -0,0 +1,29 @@
|
||||
-- ============================================================
|
||||
-- V5: Figures and chunk-to-figure reference table
|
||||
-- figure: metadata + file path for each extracted image
|
||||
-- chunk_figure_ref: links vector-store chunks to figures
|
||||
-- ============================================================
|
||||
|
||||
CREATE TABLE IF NOT EXISTS figure (
|
||||
id VARCHAR(200) PRIMARY KEY,
|
||||
book_id UUID NOT NULL REFERENCES book(id) ON DELETE CASCADE,
|
||||
section_id VARCHAR(200) REFERENCES section(id) ON DELETE SET NULL,
|
||||
chapter_id VARCHAR(200) REFERENCES chapter(id) ON DELETE SET NULL,
|
||||
label VARCHAR(100),
|
||||
caption TEXT,
|
||||
figure_type VARCHAR(50) NOT NULL,
|
||||
page INT NOT NULL,
|
||||
image_path VARCHAR(1000) NOT NULL,
|
||||
caption_embedding_id UUID,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chunk_figure_ref (
|
||||
chunk_id UUID NOT NULL,
|
||||
figure_id VARCHAR(200) NOT NULL REFERENCES figure(id) ON DELETE CASCADE,
|
||||
mention_page INT,
|
||||
PRIMARY KEY (chunk_id, figure_id)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_figure_book ON figure(book_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_cfr_chunk ON chunk_figure_ref(chunk_id);
|
||||
@@ -0,0 +1,10 @@
|
||||
CREATE TABLE topic_summary (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
topic_id VARCHAR(100) NOT NULL,
|
||||
summary_number INT NOT NULL,
|
||||
summary TEXT NOT NULL,
|
||||
sources_json TEXT NOT NULL,
|
||||
generated_at TIMESTAMPTZ NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX idx_topic_summary_topic_id ON topic_summary(topic_id, summary_number);
|
||||
@@ -0,0 +1,14 @@
|
||||
CREATE TABLE chunk_metadata (
|
||||
chunk_id UUID PRIMARY KEY,
|
||||
book_id UUID NOT NULL,
|
||||
section_id VARCHAR(200) NOT NULL,
|
||||
facet VARCHAR(32) NOT NULL,
|
||||
entities JSONB NOT NULL,
|
||||
summary TEXT NOT NULL,
|
||||
model_version VARCHAR(32) NOT NULL,
|
||||
enriched_at TIMESTAMPTZ NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX idx_chunk_metadata_book ON chunk_metadata(book_id);
|
||||
CREATE INDEX idx_chunk_metadata_book_facet ON chunk_metadata(book_id, facet);
|
||||
CREATE INDEX idx_chunk_metadata_entities_gin ON chunk_metadata USING GIN (entities jsonb_path_ops);
|
||||
@@ -0,0 +1,11 @@
|
||||
CREATE TABLE concept_report (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
topic_id VARCHAR(100) NOT NULL,
|
||||
report_number INT NOT NULL,
|
||||
facets_json TEXT NOT NULL,
|
||||
sources_json TEXT NOT NULL,
|
||||
generated_at TIMESTAMPTZ NOT NULL,
|
||||
UNIQUE (topic_id, report_number)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_concept_report_topic ON concept_report(topic_id, report_number);
|
||||
@@ -0,0 +1,19 @@
|
||||
ALTER TABLE chunk_metadata DROP CONSTRAINT IF EXISTS chunk_metadata_facet_check;
|
||||
|
||||
ALTER TABLE chunk_metadata
|
||||
ADD CONSTRAINT chunk_metadata_facet_check
|
||||
CHECK (facet IN (
|
||||
'DEFINITION',
|
||||
'ANATOMY',
|
||||
'PATHOPHYSIOLOGY',
|
||||
'EPIDEMIOLOGY',
|
||||
'CLINICAL_PRESENTATION',
|
||||
'IMAGING',
|
||||
'CLASSIFICATION',
|
||||
'INDICATIONS',
|
||||
'SURGICAL_TECHNIQUE',
|
||||
'NONSURGICAL_MANAGEMENT',
|
||||
'COMPLICATIONS',
|
||||
'OUTCOMES_FOLLOWUP',
|
||||
'OTHER'
|
||||
));
|
||||
@@ -0,0 +1,172 @@
|
||||
# Concept Retrieval via Indexing-Time Chunk Enrichment
|
||||
|
||||
## Context
|
||||
|
||||
Vector similarity alone can't answer "tell me everything about aneurysms." It surfaces the chunks most *linguistically* similar to the query, not the set of all chunks that *concern* the concept — and it has no notion of whether each chunk is a definition, a case, a technique, or a complication.
|
||||
|
||||
The unlock is to move intelligence from query time to indexing time: for every text chunk, use an LLM to extract **structured metadata** (entities, facet, summary). At retrieval time, concept lookup becomes an SQL filter (`entities @> ['aneurysm']`) bucketed by facet — deterministic, exhaustive, and organized by default. Vector search remains as a fallback for typos / synonyms and for ranking within a facet.
|
||||
|
||||
This plan covers: (1) defining the metadata schema, (2) enriching chunks during new book ingestion, (3) back-filling the already-embedded corpus via an admin endpoint, (4) a new concept retrieval path, and (5) a Topics-page UI to surface the result.
|
||||
|
||||
## Approach
|
||||
|
||||
### 1. Data model — new `chunk_metadata` table
|
||||
|
||||
Flyway migration `backend/src/main/resources/db/migration/V7__chunk_metadata.sql`:
|
||||
|
||||
```sql
|
||||
CREATE TABLE chunk_metadata (
|
||||
chunk_id VARCHAR(64) PRIMARY KEY, -- same UUID that TextChunkingService issues and stores in vectorstore
|
||||
book_id UUID NOT NULL,
|
||||
section_id VARCHAR(255) NOT NULL,
|
||||
facet VARCHAR(32) NOT NULL, -- enum (see ConceptFacet)
|
||||
entities JSONB NOT NULL, -- canonical lowercase string[]
|
||||
summary TEXT NOT NULL,
|
||||
model_version VARCHAR(32) NOT NULL, -- records which LLM/prompt version tagged this chunk
|
||||
enriched_at TIMESTAMPTZ NOT NULL
|
||||
);
|
||||
CREATE INDEX idx_chunk_metadata_book ON chunk_metadata(book_id);
|
||||
CREATE INDEX idx_chunk_metadata_book_facet ON chunk_metadata(book_id, facet);
|
||||
CREATE INDEX idx_chunk_metadata_entities_gin ON chunk_metadata USING GIN (entities jsonb_path_ops);
|
||||
```
|
||||
|
||||
Why `chunk_id` is the natural key: `TextChunkingService` already generates a UUID per chunk, uses it as the pgvector Document id, stores it in metadata, and it's the key in `ChunkFigureRefEntity` — so the table joins cleanly to everything already in place.
|
||||
|
||||
### 2. Enrichment service & facet taxonomy
|
||||
|
||||
New package `com.aiteacher.enrichment`:
|
||||
|
||||
- `ConceptFacet` enum — 13 values tailored to neurosurgery textbooks: `DEFINITION, ANATOMY, PATHOPHYSIOLOGY, EPIDEMIOLOGY, CLINICAL_PRESENTATION, IMAGING, CLASSIFICATION, INDICATIONS, SURGICAL_TECHNIQUE, NONSURGICAL_MANAGEMENT, COMPLICATIONS, OUTCOMES_FOLLOWUP, OTHER`. `OTHER` is mandatory so the LLM always has an out (no hallucinated bucketing). The prompt carries explicit disambiguation rules (named grading scales → `CLASSIFICATION`; imaging of a complication → `COMPLICATIONS`; tools inside an operation → `SURGICAL_TECHNIQUE`).
|
||||
- `ChunkEnrichmentResult` — record `(List<String> entities, ConceptFacet facet, String summary)`
|
||||
- `ChunkEnrichmentService` — single method `enrich(String chunkText, SectionEntity section, String bookTitle) → ChunkEnrichmentResult`. Uses Spring AI `ChatClient.prompt().call().entity(Class)` for structured output. The prompt gives: book title, section title, chunk text, the fixed facet enum list, and instructs the model to return JSON with entities normalised to lowercase singular canonical form (e.g. "aneurysms" → "aneurysm"; "SAH" → "subarachnoid hemorrhage"). Caps entities at ~8 per chunk.
|
||||
- `ChunkMetadataEntity` + `ChunkMetadataRepository` — JPA entity/repo mirroring the table.
|
||||
|
||||
Model version string (e.g. `"v1"`) lives on the service and is stamped into each row so a future prompt rev can be rolled out by filtering `model_version <> 'v2'` in the backfill job.
|
||||
|
||||
### 3. Hook into new book ingestion
|
||||
|
||||
Modify `BookEmbeddingService.embedBook`:
|
||||
|
||||
```java
|
||||
// Step 3: Chunk and embed text
|
||||
List<Document> allChunks = new ArrayList<>();
|
||||
for (SectionEntity section : sections) {
|
||||
allChunks.addAll(textChunkingService.chunk(section, bookTitle));
|
||||
}
|
||||
if (skipEmbedding) { ... } else {
|
||||
embedInBatches(allChunks, bookId);
|
||||
chunkEnrichmentPipeline.enrichAndPersist(allChunks, sectionsById, bookTitle); // NEW
|
||||
}
|
||||
```
|
||||
|
||||
- `ChunkEnrichmentPipeline` — new orchestrator that iterates chunks, calls `ChunkEnrichmentService.enrich(...)` per chunk, saves `ChunkMetadataEntity` rows in batches, with the same throttle pattern as `embedInBatches`.
|
||||
- Runs *after* embedding, not in place of it, so a failure in enrichment doesn't corrupt the vector store. On failure, log and continue — the backfill endpoint is the universal recovery path.
|
||||
- Extend `deleteBookChunks` to also delete `chunk_metadata` rows so deletion stays consistent.
|
||||
|
||||
### 4. Backfill endpoint for already-embedded books
|
||||
|
||||
New `EnrichmentController` in `com.aiteacher.enrichment`:
|
||||
|
||||
- `POST /api/v1/admin/books/{id}/enrich` → kicks off async backfill, returns 202 with `{status, chunksTotal, chunksEnriched}`
|
||||
- `GET /api/v1/admin/books/{id}/enrich` → returns progress
|
||||
|
||||
Backfill flow (`EnrichmentBackfillService.backfillBook(UUID bookId)`):
|
||||
|
||||
1. Query the pgvector storage table directly via `JdbcTemplate` for all chunks of the book:
|
||||
```sql
|
||||
SELECT id, content, metadata
|
||||
FROM vector_store
|
||||
WHERE metadata->>'book_id' = ? AND metadata->>'type' = 'TEXT'
|
||||
```
|
||||
2. Left-anti-join against `chunk_metadata` to skip already-enriched chunks → idempotent, resumable.
|
||||
3. For each missing chunk: look up its `SectionEntity` via `section_id` in metadata, call `ChunkEnrichmentService.enrich`, write a `ChunkMetadataEntity` row.
|
||||
4. Progress tracked in an in-memory `ConcurrentHashMap<UUID, BackfillProgress>` (POC scope — no cross-restart resumability needed because the left-anti-join makes re-runs free).
|
||||
5. `@Async` on the backfill method using the same executor as `embedBook`.
|
||||
|
||||
### 5. Concept retrieval path
|
||||
|
||||
New `com.aiteacher.concept.ConceptRetriever`:
|
||||
|
||||
```java
|
||||
public ConceptRetrievalResult retrieveByConcept(String conceptKeyword, UUID bookId) {
|
||||
String canonical = canonicalise(conceptKeyword); // lowercase, trim, simple plural strip
|
||||
|
||||
// 5a. Primary: SQL entity match, grouped by facet
|
||||
List<ChunkMetadataEntity> hits = chunkMetadataRepository
|
||||
.findByBookIdAndEntityContains(bookId, canonical); // WHERE entities @> to_jsonb(?::text)
|
||||
|
||||
if (hits.isEmpty()) {
|
||||
// 5b. Fallback: vector search, then enrich-join + facet-group
|
||||
List<Document> vectorHits = vectorStore.similaritySearch(/* TEXT filter, book_id filter, topK=30 */);
|
||||
List<String> chunkIds = vectorHits.stream().map(Document::getId).toList();
|
||||
hits = chunkMetadataRepository.findByChunkIdIn(chunkIds);
|
||||
}
|
||||
|
||||
Map<ConceptFacet, List<ChunkMetadataEntity>> byFacet = hits.stream()
|
||||
.collect(groupingBy(ChunkMetadataEntity::getFacet, LinkedHashMap::new, toList()));
|
||||
|
||||
// Hydrate: load SectionEntity for each chunk's section_id; load linked figures
|
||||
// via ChunkFigureRefRepository.findByChunkIdIn(chunkIds) — reuses existing linkage.
|
||||
return assemble(byFacet, ...);
|
||||
}
|
||||
```
|
||||
|
||||
`ConceptRetrievalResult` = `Map<ConceptFacet, FacetBundle>` where each `FacetBundle` holds the parent sections, linked figures, and the per-chunk `summary` strings.
|
||||
|
||||
Cross-book aggregation: caller loops over READY books and merges bundles by facet.
|
||||
|
||||
### 6. Concept Report service & controller
|
||||
|
||||
New `ConceptReportService` in `com.aiteacher.concept` — mirrors the shape of `TopicSummaryService`, but:
|
||||
|
||||
- Calls `ConceptRetriever.retrieveByConcept(topic.getName(), bookId)` per book.
|
||||
- For each facet that has hits, sends **one** LLM synthesis call with the chunks/figures of that facet — producing a structured, facet-labelled report.
|
||||
- Persists in a new `concept_report` table:
|
||||
|
||||
```sql
|
||||
CREATE TABLE concept_report (
|
||||
id UUID PRIMARY KEY,
|
||||
topic_id VARCHAR(255) NOT NULL REFERENCES topic(id),
|
||||
report_number INT NOT NULL,
|
||||
facets_json JSONB NOT NULL, -- [{facetKey,title,markdown,refLabels[]}, ...]
|
||||
sources_json JSONB NOT NULL, -- deduplicated SourceReference[]
|
||||
generated_at TIMESTAMPTZ NOT NULL,
|
||||
UNIQUE (topic_id, report_number)
|
||||
);
|
||||
```
|
||||
|
||||
Controller `ConceptReportController` exposes three endpoints under `/api/v1/topics/{id}/concept-reports` (POST generate, GET list, GET `/{reportId}`).
|
||||
|
||||
Reuses `TopicSummaryResponse.SourceReference` verbatim.
|
||||
|
||||
### 7. Frontend
|
||||
|
||||
- `frontend/src/stores/topicStore.ts`: add parallel state `conceptReportList`, `activeConceptReport`, `conceptReportLoading`, and actions mirroring the existing summary ones.
|
||||
- `frontend/src/views/TopicsView.vue`: add a **Summary / Concept Report** tab toggle at the top of the topic panel. Concept Report reuses the history-chips + Generate button UI. Report body renders each `FacetSection` as `<h3>{title}</h3>` + markdown.
|
||||
- Loading hint: update the "up to 30 seconds" copy to "up to 60 seconds".
|
||||
|
||||
### 8. README update
|
||||
|
||||
Add an **Indexing Pipeline** diagram showing: PDF → parse → chunk → embed → **enrich (new)** → chunk_metadata. Plus a **Concept Retrieval** sequence diagram: query → entity-match SQL → facet-grouped bundle → synthesis → report.
|
||||
|
||||
## Decisions & trade-offs
|
||||
|
||||
- **Storage as separate Postgres table, not vectorstore JSON**: vectorstore has no metadata-only update API, backfill would require delete+reinsert (re-embedding cost). A dedicated table joins cleanly on `chunk_id` and is GIN-indexed.
|
||||
- **Entity-match primary, vector fallback**: deterministic for the main use case, robust against typos/synonyms. Vector search stays the default for normal chat retrieval — this feature is additive.
|
||||
- **Enrichment runs *after* embedding, not before**: keeps the two failure modes independent. The backfill endpoint is the universal recovery lever.
|
||||
- **Fixed 9-value facet enum** (incl. `OTHER`): constrains LLM outputs; `OTHER` prevents forced mis-bucketing.
|
||||
- **Direct `JdbcTemplate` read against `vector_store` for backfill**: Spring AI exposes no listing API. Acceptable for a POC, isolated behind one method.
|
||||
- **Synchronous (sequential) LLM calls**: simplest; parallelism is a later optimisation if needed.
|
||||
- **`model_version` column**: cheap insurance. If the prompt or facet taxonomy changes, backfill can re-enrich only stale rows.
|
||||
|
||||
## Verification
|
||||
|
||||
1. Migration applies V7 and V8. Tables and indexes created.
|
||||
2. New book ingestion: upload PDF → `chunk_metadata` populated with plausible entities/facets/summaries.
|
||||
3. Backfill: POST `/api/v1/admin/books/{id}/enrich` → idempotent, completes, re-run is a no-op.
|
||||
4. Concept retrieval primary path: POST `/api/v1/topics/aneurysm/concept-reports` → 200 with facets populated.
|
||||
5. Fallback path: misspelled topic still returns results via vector fallback.
|
||||
6. Frontend: Concept Report tab renders facet-labelled markdown + sources + inline figures; persists across reloads.
|
||||
7. Deletion: removing a book cascades to `chunk_metadata` rows.
|
||||
8. Regression: existing chat and summary flows still work.
|
||||
9. Lint & tests pass.
|
||||
@@ -0,0 +1,37 @@
|
||||
version: '3.9'
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: pgvector/pgvector:pg16
|
||||
container_name: aiteacher-postgres-native
|
||||
environment:
|
||||
POSTGRES_DB: aiteacher
|
||||
POSTGRES_USER: aiteacher
|
||||
POSTGRES_PASSWORD: aiteacher
|
||||
ports:
|
||||
- "5432:5432"
|
||||
volumes:
|
||||
- pgdata_native:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U aiteacher -d aiteacher"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
backend:
|
||||
image: ai-teacher-backend:latest
|
||||
container_name: aiteacher-backend-native
|
||||
env_file:
|
||||
- .env
|
||||
environment:
|
||||
SPRING_DATASOURCE_URL: jdbc:postgresql://postgres:5432/aiteacher
|
||||
SPRING_DATASOURCE_USERNAME: aiteacher
|
||||
SPRING_DATASOURCE_PASSWORD: aiteacher
|
||||
ports:
|
||||
- "8080:8080"
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
|
||||
volumes:
|
||||
pgdata_native:
|
||||
@@ -3,5 +3,12 @@
|
||||
# In production point it directly at the backend, e.g. https://api.example.com/api/v1
|
||||
VITE_API_URL=/api/v1
|
||||
|
||||
# Shared password for HTTP Basic auth (must match APP_PASSWORD on the backend).
|
||||
VITE_APP_PASSWORD=changeme
|
||||
# Credentials are no longer configured here. Users enter their username and
|
||||
# password via the login form. The backend validates them via HTTP Basic Auth.
|
||||
# Configure the backend credentials with APP_AUTH_USERNAME and APP_PASSWORD.
|
||||
|
||||
# Set to 'false' to hide the upload UI (frontend). Also set UPLOAD_ENABLED=false on the backend to block the endpoint.
|
||||
VITE_UPLOAD_ENABLED=true
|
||||
|
||||
# Set to 'false' to hide the delete button (frontend). Also set DELETE_ENABLED=false on the backend to block the endpoint.
|
||||
VITE_DELETE_ENABLED=true
|
||||
|
||||
+5
-3
@@ -1,5 +1,5 @@
|
||||
# ---- Build stage ----
|
||||
FROM node:20-alpine AS build
|
||||
FROM docker.io/library/node:20-alpine AS build
|
||||
WORKDIR /app
|
||||
COPY package*.json ./
|
||||
RUN npm ci
|
||||
@@ -7,8 +7,10 @@ COPY . .
|
||||
RUN npm run build
|
||||
|
||||
# ---- Runtime stage (nginx) ----
|
||||
FROM nginx:alpine
|
||||
FROM docker.io/library/nginx:alpine
|
||||
COPY --from=build /app/dist /usr/share/nginx/html
|
||||
COPY nginx.conf /etc/nginx/conf.d/default.conf
|
||||
COPY docker-entrypoint.sh /docker-entrypoint.sh
|
||||
RUN chmod +x /docker-entrypoint.sh
|
||||
EXPOSE 80
|
||||
CMD ["nginx", "-g", "daemon off;"]
|
||||
ENTRYPOINT ["/docker-entrypoint.sh"]
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
#!/bin/sh
|
||||
set -e
|
||||
|
||||
# Write runtime env vars into a JS file loaded before the app bundle.
|
||||
# Any VITE_* variable passed via `docker run -e` will be available as
|
||||
# window.__env__.VITE_* inside the browser.
|
||||
cat > /usr/share/nginx/html/env-config.js <<EOF
|
||||
window.__env__ = {
|
||||
VITE_API_URL: "${VITE_API_URL:-}",
|
||||
VITE_APP_PASSWORD: "${VITE_APP_PASSWORD:-}",
|
||||
VITE_UPLOAD_ENABLED: "${VITE_UPLOAD_ENABLED:-}",
|
||||
VITE_DELETE_ENABLED: "${VITE_DELETE_ENABLED:-}"
|
||||
};
|
||||
EOF
|
||||
|
||||
exec nginx -g "daemon off;"
|
||||
@@ -8,6 +8,7 @@
|
||||
</head>
|
||||
<body>
|
||||
<div id="app"></div>
|
||||
<script src="/env-config.js"></script>
|
||||
<script type="module" src="/src/main.ts"></script>
|
||||
</body>
|
||||
</html>
|
||||
|
||||
+156
-21
@@ -6,23 +6,31 @@
|
||||
<span class="brand-name">AI Teacher</span>
|
||||
<span class="brand-subtitle">Neurosurgeon Learning Platform</span>
|
||||
</div>
|
||||
<ul class="navbar-links">
|
||||
<li>
|
||||
<RouterLink to="/" :class="{ active: $route.path === '/' }">
|
||||
<span class="nav-icon">📚</span> Library
|
||||
</RouterLink>
|
||||
</li>
|
||||
<li>
|
||||
<RouterLink to="/topics" :class="{ active: $route.path === '/topics' }">
|
||||
<span class="nav-icon">🗂</span> Topics
|
||||
</RouterLink>
|
||||
</li>
|
||||
<li>
|
||||
<RouterLink to="/chat" :class="{ active: $route.path === '/chat' }">
|
||||
<span class="nav-icon">💬</span> Chat
|
||||
</RouterLink>
|
||||
</li>
|
||||
</ul>
|
||||
<template v-if="authStore.isAuthenticated">
|
||||
<button class="burger" :class="{ open: menuOpen }" @click="menuOpen = !menuOpen" aria-label="Menu">
|
||||
<span></span><span></span><span></span>
|
||||
</button>
|
||||
<div class="nav-drawer" :class="{ open: menuOpen }" @click="menuOpen = false">
|
||||
<ul class="navbar-links">
|
||||
<li>
|
||||
<RouterLink to="/" :class="{ active: $route.path === '/' }">
|
||||
<span class="nav-icon">📚</span> Library
|
||||
</RouterLink>
|
||||
</li>
|
||||
<li>
|
||||
<RouterLink to="/topics" :class="{ active: $route.path === '/topics' }">
|
||||
<span class="nav-icon">🗂</span> Topics
|
||||
</RouterLink>
|
||||
</li>
|
||||
<li>
|
||||
<RouterLink to="/chat" :class="{ active: $route.path === '/chat' }">
|
||||
<span class="nav-icon">💬</span> Chat
|
||||
</RouterLink>
|
||||
</li>
|
||||
</ul>
|
||||
<button class="btn btn-logout" @click.stop="logout">Sign out</button>
|
||||
</div>
|
||||
</template>
|
||||
</nav>
|
||||
|
||||
<main class="main-content">
|
||||
@@ -35,12 +43,26 @@
|
||||
</template>
|
||||
|
||||
<script setup lang="ts">
|
||||
import { ref, provide } from 'vue'
|
||||
import { RouterLink, RouterView } from 'vue-router'
|
||||
import { ref, provide, watch } from 'vue'
|
||||
import { RouterLink, RouterView, useRouter, useRoute } from 'vue-router'
|
||||
import { useAuthStore } from '@/stores/authStore'
|
||||
|
||||
const authStore = useAuthStore()
|
||||
const router = useRouter()
|
||||
const route = useRoute()
|
||||
|
||||
const menuOpen = ref(false)
|
||||
const toastMessage = ref('')
|
||||
const toastType = ref<'toast-error' | 'toast-success'>('toast-error')
|
||||
|
||||
// Close menu on navigation
|
||||
watch(() => route.path, () => { menuOpen.value = false })
|
||||
|
||||
function logout() {
|
||||
authStore.clearCredentials()
|
||||
router.push({ name: 'login' })
|
||||
}
|
||||
|
||||
function showToast(message: string, type: 'error' | 'success' = 'error') {
|
||||
toastMessage.value = message
|
||||
toastType.value = type === 'error' ? 'toast-error' : 'toast-success'
|
||||
@@ -64,11 +86,11 @@ body {
|
||||
Ubuntu, Cantarell, 'Fira Sans', 'Droid Sans', 'Helvetica Neue', sans-serif;
|
||||
background: #f0f4f8;
|
||||
color: #2d3748;
|
||||
min-height: 100vh;
|
||||
height: 100vh;
|
||||
}
|
||||
|
||||
#app {
|
||||
min-height: 100vh;
|
||||
height: 100vh;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
}
|
||||
@@ -82,6 +104,9 @@ body {
|
||||
justify-content: space-between;
|
||||
height: 64px;
|
||||
box-shadow: 0 2px 8px rgba(0, 0, 0, 0.3);
|
||||
position: sticky;
|
||||
top: 0;
|
||||
z-index: 100;
|
||||
}
|
||||
|
||||
.navbar-brand {
|
||||
@@ -106,6 +131,13 @@ body {
|
||||
margin-left: 0.25rem;
|
||||
}
|
||||
|
||||
/* Desktop: links inline */
|
||||
.nav-drawer {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 0.5rem;
|
||||
}
|
||||
|
||||
.navbar-links {
|
||||
list-style: none;
|
||||
display: flex;
|
||||
@@ -131,8 +163,38 @@ body {
|
||||
color: white;
|
||||
}
|
||||
|
||||
/* Burger button — hidden on desktop */
|
||||
.burger {
|
||||
display: none;
|
||||
flex-direction: column;
|
||||
justify-content: center;
|
||||
gap: 5px;
|
||||
width: 36px;
|
||||
height: 36px;
|
||||
background: transparent;
|
||||
border: none;
|
||||
cursor: pointer;
|
||||
padding: 4px;
|
||||
border-radius: 6px;
|
||||
}
|
||||
|
||||
.burger span {
|
||||
display: block;
|
||||
height: 2px;
|
||||
background: #bee3f8;
|
||||
border-radius: 2px;
|
||||
transition: transform 0.2s, opacity 0.2s;
|
||||
}
|
||||
|
||||
.burger.open span:nth-child(1) { transform: translateY(7px) rotate(45deg); }
|
||||
.burger.open span:nth-child(2) { opacity: 0; }
|
||||
.burger.open span:nth-child(3) { transform: translateY(-7px) rotate(-45deg); }
|
||||
|
||||
.main-content {
|
||||
flex: 1;
|
||||
min-height: 0;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
padding: 2rem;
|
||||
max-width: 1200px;
|
||||
margin: 0 auto;
|
||||
@@ -224,6 +286,20 @@ body {
|
||||
background: #cbd5e0;
|
||||
}
|
||||
|
||||
.btn-logout {
|
||||
background: transparent;
|
||||
color: #bee3f8;
|
||||
border: 1px solid #4a90b8;
|
||||
font-size: 0.85rem;
|
||||
padding: 0.4rem 0.9rem;
|
||||
margin-left: 1rem;
|
||||
}
|
||||
|
||||
.btn-logout:hover {
|
||||
background: #2b6cb0;
|
||||
color: white;
|
||||
}
|
||||
|
||||
.spinner {
|
||||
display: inline-block;
|
||||
width: 20px;
|
||||
@@ -284,4 +360,63 @@ body {
|
||||
font-size: 0.9rem;
|
||||
margin-top: 0.5rem;
|
||||
}
|
||||
|
||||
@media (max-width: 768px) {
|
||||
.navbar {
|
||||
padding: 0 1rem;
|
||||
}
|
||||
|
||||
.brand-subtitle {
|
||||
display: none;
|
||||
}
|
||||
|
||||
/* Show burger, hide desktop drawer */
|
||||
.burger {
|
||||
display: flex;
|
||||
}
|
||||
|
||||
.nav-drawer {
|
||||
display: none;
|
||||
position: absolute;
|
||||
top: 64px;
|
||||
right: 0;
|
||||
left: 0;
|
||||
background: #1a365d;
|
||||
flex-direction: column;
|
||||
align-items: stretch;
|
||||
padding: 0.5rem 0 1rem;
|
||||
box-shadow: 0 4px 12px rgba(0, 0, 0, 0.3);
|
||||
z-index: 99;
|
||||
}
|
||||
|
||||
.nav-drawer.open {
|
||||
display: flex;
|
||||
}
|
||||
|
||||
.navbar-links {
|
||||
flex-direction: column;
|
||||
gap: 0;
|
||||
}
|
||||
|
||||
.navbar-links a {
|
||||
padding: 0.85rem 1.5rem;
|
||||
border-radius: 0;
|
||||
font-size: 1rem;
|
||||
}
|
||||
|
||||
.navbar-links a:hover,
|
||||
.navbar-links a.active {
|
||||
background: #2b6cb0;
|
||||
}
|
||||
|
||||
.btn-logout {
|
||||
margin: 0.5rem 1.5rem 0;
|
||||
width: calc(100% - 3rem);
|
||||
justify-content: center;
|
||||
}
|
||||
|
||||
.main-content {
|
||||
padding: 1rem;
|
||||
}
|
||||
}
|
||||
</style>
|
||||
|
||||
@@ -32,8 +32,32 @@
|
||||
<span>{{ book.status === 'PENDING' ? 'Queued for processing...' : 'Embedding in progress...' }}</span>
|
||||
</div>
|
||||
|
||||
<div v-if="enrichProgress && enrichProgress.status === 'RUNNING'" class="processing-indicator">
|
||||
<div class="spinner spinner-dark"></div>
|
||||
<span>Enriching chunks {{ enrichProgress.chunksEnriched }} / {{ enrichProgress.chunksTotal }}</span>
|
||||
</div>
|
||||
|
||||
<div v-if="enrichFeedback" class="enrich-feedback">{{ enrichFeedback }}</div>
|
||||
|
||||
<div class="book-actions">
|
||||
<router-link
|
||||
v-if="book.status === 'READY'"
|
||||
:to="{ name: 'book-reader', params: { id: book.id } }"
|
||||
class="btn btn-secondary"
|
||||
>
|
||||
Read
|
||||
</router-link>
|
||||
<button
|
||||
v-if="book.status === 'READY' && uploadEnabled"
|
||||
class="btn btn-secondary"
|
||||
:disabled="enrichRunning"
|
||||
@click="handleEnrich"
|
||||
title="Enrich chunks with concept metadata"
|
||||
>
|
||||
{{ enrichRunning ? 'Enriching...' : 'Enrich' }}
|
||||
</button>
|
||||
<button
|
||||
v-if="deleteEnabled"
|
||||
class="btn btn-danger"
|
||||
:disabled="book.status === 'PROCESSING' || deleting"
|
||||
@click="$emit('delete', book.id)"
|
||||
@@ -46,18 +70,62 @@
|
||||
</template>
|
||||
|
||||
<script setup lang="ts">
|
||||
import { computed } from 'vue'
|
||||
import type { Book } from '@/stores/bookStore'
|
||||
import { computed, onUnmounted, ref } from 'vue'
|
||||
import type { Book, EnrichmentProgress } from '@/stores/bookStore'
|
||||
import { useBookStore } from '@/stores/bookStore'
|
||||
import { env } from '@/env';
|
||||
|
||||
const props = defineProps<{
|
||||
book: Book
|
||||
deleting?: boolean
|
||||
deleteEnabled?: boolean
|
||||
}>()
|
||||
|
||||
defineEmits<{
|
||||
(e: 'delete', id: string): void
|
||||
}>()
|
||||
|
||||
const bookStore = useBookStore()
|
||||
const enrichProgress = ref<EnrichmentProgress | null>(null)
|
||||
const enrichFeedback = ref<string | null>(null)
|
||||
let pollTimer: ReturnType<typeof setInterval> | null = null
|
||||
|
||||
const enrichRunning = computed(() => enrichProgress.value?.status === 'RUNNING')
|
||||
const uploadEnabled = env('VITE_UPLOAD_ENABLED') !== 'false'
|
||||
|
||||
async function handleEnrich() {
|
||||
enrichFeedback.value = null
|
||||
const started = await bookStore.startEnrichment(props.book.id)
|
||||
if (!started) {
|
||||
enrichFeedback.value = bookStore.error ?? 'Enrichment failed to start.'
|
||||
return
|
||||
}
|
||||
enrichProgress.value = started
|
||||
startPolling()
|
||||
}
|
||||
|
||||
function startPolling() {
|
||||
stopPolling()
|
||||
pollTimer = setInterval(async () => {
|
||||
const status = await bookStore.fetchEnrichmentStatus(props.book.id)
|
||||
if (!status) return
|
||||
enrichProgress.value = status
|
||||
if (status.status === 'COMPLETED') {
|
||||
stopPolling()
|
||||
enrichFeedback.value = `Enriched ${status.chunksEnriched} / ${status.chunksTotal} chunks.`
|
||||
}
|
||||
}, 2000)
|
||||
}
|
||||
|
||||
function stopPolling() {
|
||||
if (pollTimer != null) {
|
||||
clearInterval(pollTimer)
|
||||
pollTimer = null
|
||||
}
|
||||
}
|
||||
|
||||
onUnmounted(stopPolling)
|
||||
|
||||
const statusClass = computed(() => {
|
||||
switch (props.book.status) {
|
||||
case 'READY':
|
||||
@@ -181,6 +249,16 @@ function formatDate(iso: string): string {
|
||||
.book-actions {
|
||||
display: flex;
|
||||
justify-content: flex-end;
|
||||
gap: 0.5rem;
|
||||
margin-top: 0.25rem;
|
||||
}
|
||||
|
||||
.enrich-feedback {
|
||||
font-size: 0.8rem;
|
||||
color: #22543d;
|
||||
background: #f0fff4;
|
||||
border: 1px solid #c6f6d5;
|
||||
border-radius: 6px;
|
||||
padding: 0.4rem 0.6rem;
|
||||
}
|
||||
</style>
|
||||
|
||||
@@ -0,0 +1,239 @@
|
||||
<template>
|
||||
<div class="book-panel">
|
||||
<div class="book-panel-header">
|
||||
<span class="book-panel-title">{{ bookTitle || 'Book' }} — p. {{ page }}</span>
|
||||
<div class="book-panel-nav">
|
||||
<button class="nav-btn" :disabled="page <= 1" @click="emit('navigate', page - 1)">←</button>
|
||||
<button class="nav-btn" @click="emit('navigate', page + 1)">→</button>
|
||||
</div>
|
||||
<button class="close-btn" @click="emit('close')" title="Close">✕</button>
|
||||
</div>
|
||||
|
||||
<div class="book-panel-body">
|
||||
<div v-if="loading" class="panel-loading">
|
||||
<div class="spinner spinner-dark" style="width:24px;height:24px;margin:0 auto 0.5rem;"></div>
|
||||
<p>Loading page {{ page }}…</p>
|
||||
</div>
|
||||
<div v-else-if="error" class="panel-error">{{ error }}</div>
|
||||
<div v-else class="markdown-body" v-html="renderedHtml"></div>
|
||||
</div>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<script setup lang="ts">
|
||||
import { ref, watch, onMounted, onUnmounted } from 'vue'
|
||||
import { api } from '@/services/api'
|
||||
|
||||
const props = defineProps<{
|
||||
bookId: string
|
||||
page: number
|
||||
bookTitle?: string
|
||||
}>()
|
||||
|
||||
const emit = defineEmits<{
|
||||
close: []
|
||||
navigate: [page: number]
|
||||
}>()
|
||||
|
||||
const loading = ref(false)
|
||||
const error = ref<string | null>(null)
|
||||
const renderedHtml = ref('')
|
||||
let activeBlobUrls: string[] = []
|
||||
|
||||
onMounted(() => loadPage(props.page))
|
||||
|
||||
watch(() => [props.bookId, props.page], () => loadPage(props.page))
|
||||
|
||||
onUnmounted(() => {
|
||||
activeBlobUrls.forEach(u => URL.revokeObjectURL(u))
|
||||
})
|
||||
|
||||
async function loadPage(page: number) {
|
||||
loading.value = true
|
||||
error.value = null
|
||||
renderedHtml.value = ''
|
||||
activeBlobUrls.forEach(u => URL.revokeObjectURL(u))
|
||||
activeBlobUrls = []
|
||||
|
||||
try {
|
||||
const res = await api.get<string>(`/books/${props.bookId}/pages/${page}/html`, {
|
||||
headers: { Accept: 'text/html' },
|
||||
responseType: 'text'
|
||||
})
|
||||
renderedHtml.value = await resolveImages(res.data)
|
||||
} catch (e: any) {
|
||||
error.value = e.message ?? 'Failed to load page.'
|
||||
} finally {
|
||||
loading.value = false
|
||||
}
|
||||
}
|
||||
|
||||
async function resolveImages(html: string): Promise<string> {
|
||||
const srcPattern = /src="(\/api\/v1\/figures\/[^"]+)"/g
|
||||
const matches = [...html.matchAll(srcPattern)]
|
||||
if (matches.length === 0) return html
|
||||
|
||||
const unique = [...new Set(matches.map(m => m[1]))]
|
||||
const blobMap: Record<string, string> = {}
|
||||
|
||||
await Promise.all(
|
||||
unique.map(async (src) => {
|
||||
try {
|
||||
const res = await api.get(src.replace(/^\/api\/v1/, ''), { responseType: 'blob' })
|
||||
const blobUrl = URL.createObjectURL(res.data)
|
||||
activeBlobUrls.push(blobUrl)
|
||||
blobMap[src] = blobUrl
|
||||
} catch {
|
||||
// leave original src
|
||||
}
|
||||
})
|
||||
)
|
||||
|
||||
return html.replace(/src="(\/api\/v1\/figures\/[^"]+)"/g, (_, src) =>
|
||||
blobMap[src] ? `src="${blobMap[src]}"` : `src="${src}"`
|
||||
)
|
||||
}
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
.book-panel {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
height: 100%;
|
||||
background: white;
|
||||
border-left: 1px solid #e2e8f0;
|
||||
border-radius: 0 10px 10px 0;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.book-panel-header {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 0.5rem;
|
||||
padding: 0.6rem 0.75rem;
|
||||
background: #f7fafc;
|
||||
border-bottom: 1px solid #e2e8f0;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.book-panel-title {
|
||||
flex: 1;
|
||||
font-size: 0.8rem;
|
||||
font-weight: 600;
|
||||
color: #2b6cb0;
|
||||
white-space: nowrap;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
}
|
||||
|
||||
.book-panel-nav {
|
||||
display: flex;
|
||||
gap: 0.25rem;
|
||||
}
|
||||
|
||||
.nav-btn {
|
||||
width: 1.75rem;
|
||||
height: 1.75rem;
|
||||
border: 1px solid #cbd5e0;
|
||||
border-radius: 5px;
|
||||
background: white;
|
||||
cursor: pointer;
|
||||
font-size: 0.85rem;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
transition: background 0.15s;
|
||||
}
|
||||
.nav-btn:hover:not(:disabled) { background: #ebf8ff; border-color: #3182ce; }
|
||||
.nav-btn:disabled { opacity: 0.4; cursor: not-allowed; }
|
||||
|
||||
.close-btn {
|
||||
width: 1.75rem;
|
||||
height: 1.75rem;
|
||||
border: none;
|
||||
border-radius: 5px;
|
||||
background: none;
|
||||
cursor: pointer;
|
||||
font-size: 1rem;
|
||||
color: #718096;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
transition: background 0.15s, color 0.15s;
|
||||
}
|
||||
.close-btn:hover { background: #fed7d7; color: #742a2a; }
|
||||
|
||||
.book-panel-body {
|
||||
flex: 1;
|
||||
overflow-y: auto;
|
||||
padding: 1rem 1.25rem;
|
||||
}
|
||||
|
||||
.panel-loading {
|
||||
text-align: center;
|
||||
padding: 2rem;
|
||||
color: #718096;
|
||||
font-size: 0.875rem;
|
||||
}
|
||||
|
||||
.panel-error {
|
||||
padding: 1rem;
|
||||
background: #fff5f5;
|
||||
border: 1px solid #fed7d7;
|
||||
color: #742a2a;
|
||||
border-radius: 6px;
|
||||
font-size: 0.875rem;
|
||||
}
|
||||
|
||||
.markdown-body {
|
||||
font-size: 0.9rem;
|
||||
line-height: 1.75;
|
||||
color: #2d3748;
|
||||
}
|
||||
|
||||
.markdown-body :deep(h1),
|
||||
.markdown-body :deep(h2),
|
||||
.markdown-body :deep(h3) {
|
||||
color: #1a365d;
|
||||
font-weight: 600;
|
||||
margin: 1.25rem 0 0.5rem;
|
||||
}
|
||||
.markdown-body :deep(h2) { font-size: 1.05rem; border-bottom: 1px solid #e2e8f0; padding-bottom: 0.3rem; }
|
||||
.markdown-body :deep(h3) { font-size: 0.95rem; }
|
||||
.markdown-body :deep(p) { margin: 0.6rem 0; }
|
||||
.markdown-body :deep(img) {
|
||||
max-width: 100%;
|
||||
border-radius: 6px;
|
||||
display: block;
|
||||
margin: 0.75rem auto;
|
||||
box-shadow: 0 1px 4px rgba(0,0,0,0.12);
|
||||
}
|
||||
.markdown-body :deep(ul),
|
||||
.markdown-body :deep(ol) { padding-left: 1.4rem; margin: 0.5rem 0; }
|
||||
.markdown-body :deep(code) {
|
||||
background: #f7fafc;
|
||||
border: 1px solid #e2e8f0;
|
||||
border-radius: 3px;
|
||||
padding: 0.1em 0.3em;
|
||||
font-size: 0.85em;
|
||||
}
|
||||
.markdown-body :deep(blockquote) {
|
||||
border-left: 3px solid #3182ce;
|
||||
padding-left: 0.75rem;
|
||||
color: #4a5568;
|
||||
margin: 0.5rem 0;
|
||||
}
|
||||
.markdown-body :deep(table) {
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
font-size: 0.875em;
|
||||
margin: 0.75rem 0;
|
||||
}
|
||||
.markdown-body :deep(th),
|
||||
.markdown-body :deep(td) {
|
||||
border: 1px solid #e2e8f0;
|
||||
padding: 0.35rem 0.6rem;
|
||||
text-align: left;
|
||||
}
|
||||
.markdown-body :deep(th) { background: #f7fafc; font-weight: 600; }
|
||||
</style>
|
||||
@@ -3,25 +3,17 @@
|
||||
<div class="message-bubble" :class="isUser ? 'bubble-user' : 'bubble-assistant'">
|
||||
<div class="message-role">{{ isUser ? 'You' : 'AI Teacher' }}</div>
|
||||
<div v-if="isUser" class="message-content">{{ message.content }}</div>
|
||||
<div v-else class="message-content message-content--markdown" v-html="renderedContent"></div>
|
||||
<div v-else class="message-content message-content--markdown" v-html="renderedWithBadges" @click="onContentClick"></div>
|
||||
|
||||
<!-- Source chips for assistant messages -->
|
||||
<!-- Sources for assistant messages -->
|
||||
<div v-if="!isUser && message.sources && message.sources.length > 0" class="message-sources">
|
||||
<div class="sources-label">Sources:</div>
|
||||
<div class="source-list">
|
||||
<div
|
||||
v-for="(source, idx) in message.sources"
|
||||
:key="idx"
|
||||
class="source-item"
|
||||
>
|
||||
<div class="source-chip">
|
||||
<span class="source-book-icon">📖</span>
|
||||
<span class="source-book-title">{{ source.bookTitle }}</span>
|
||||
<span v-if="source.page" class="source-page">p. {{ source.page }}</span>
|
||||
</div>
|
||||
<div v-if="source.chunkText" class="source-chunk">{{ source.chunkText }}</div>
|
||||
</div>
|
||||
</div>
|
||||
<SourceList
|
||||
ref="sourceListEl"
|
||||
:sources="message.sources"
|
||||
:active-ref="activeRef"
|
||||
@open-source="(bookId: string, page: number) => emit('open-source', bookId, page)"
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div class="message-timestamp">{{ formatTime(message.createdAt) }}</div>
|
||||
@@ -30,16 +22,72 @@
|
||||
</template>
|
||||
|
||||
<script setup lang="ts">
|
||||
import { computed } from 'vue'
|
||||
import { computed, ref } from 'vue'
|
||||
import { marked } from 'marked'
|
||||
import type { ChatMessage } from '@/stores/chatStore'
|
||||
import type { ChatMessage, ChatSource } from '@/stores/chatStore'
|
||||
import SourceList from '@/components/SourceList.vue'
|
||||
|
||||
const props = defineProps<{
|
||||
message: ChatMessage
|
||||
}>()
|
||||
|
||||
const emit = defineEmits<{
|
||||
'open-source': [bookId: string, page: number]
|
||||
}>()
|
||||
|
||||
const isUser = computed(() => props.message.role === 'USER')
|
||||
const renderedContent = computed(() => marked.parse(props.message.content) as string)
|
||||
const activeRef = ref<string | null>(null)
|
||||
const sourceListEl = ref<InstanceType<typeof SourceList> | null>(null)
|
||||
|
||||
function escapeHtml(s: string): string {
|
||||
return s.replace(/&/g, '&').replace(/</g, '<').replace(/>/g, '>').replace(/"/g, '"')
|
||||
}
|
||||
|
||||
const renderedWithBadges = computed(() => {
|
||||
const html = marked.parse(props.message.content) as string
|
||||
|
||||
const figureMap = new Map<string, ChatSource>()
|
||||
for (const src of (props.message.sources ?? [])) {
|
||||
if (src.type === 'FIGURE' && src.refLabel) {
|
||||
figureMap.set(src.refLabel, src)
|
||||
}
|
||||
}
|
||||
|
||||
return html.replace(/\[(S|F)\d+\]/g, (match) => {
|
||||
const inner = match.slice(1, -1)
|
||||
const badge = `<span class="citation-badge" data-ref="${inner}" title="Jump to source ${inner}">${match}</span>`
|
||||
|
||||
const fig = figureMap.get(inner)
|
||||
if (fig?.imageUrl) {
|
||||
const alt = escapeHtml(fig.caption || fig.label || 'Figure')
|
||||
const captionText = [fig.label, fig.caption].filter(Boolean).map(escapeHtml).join(' — ')
|
||||
const captionHtml = captionText
|
||||
? `<figcaption class="inline-figure-caption">${captionText}</figcaption>`
|
||||
: ''
|
||||
return `${badge}<figure class="inline-figure"><img src="${fig.imageUrl}" alt="${alt}" class="inline-figure-img" loading="lazy" onerror="this.parentElement.style.display='none'" />${captionHtml}</figure>`
|
||||
}
|
||||
|
||||
return badge
|
||||
})
|
||||
})
|
||||
|
||||
function onContentClick(e: MouseEvent) {
|
||||
const target = e.target as HTMLElement
|
||||
if (!target.classList.contains('citation-badge')) return
|
||||
|
||||
const label = target.getAttribute('data-ref')
|
||||
if (!label) return
|
||||
|
||||
activeRef.value = activeRef.value === label ? null : label
|
||||
|
||||
const sourceEl = sourceListEl.value?.$el?.querySelector(`[data-ref-label="${label}"]`) as HTMLElement | null
|
||||
sourceEl?.scrollIntoView({ behavior: 'smooth', block: 'start' })
|
||||
|
||||
const source = (props.message.sources ?? []).find((s: ChatSource) => s.refLabel === label)
|
||||
if (source?.bookId && source.page) {
|
||||
emit('open-source', source.bookId, source.page)
|
||||
}
|
||||
}
|
||||
|
||||
function formatTime(iso: string): string {
|
||||
return new Date(iso).toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' })
|
||||
@@ -182,6 +230,71 @@ function formatTime(iso: string): string {
|
||||
gap: 0.25rem;
|
||||
}
|
||||
|
||||
.source-item--figure {
|
||||
gap: 0.4rem;
|
||||
}
|
||||
|
||||
.source-chip {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 0.25rem;
|
||||
border-radius: 4px;
|
||||
padding: 0.2rem 0.5rem;
|
||||
font-size: 0.78rem;
|
||||
}
|
||||
|
||||
.source-chip--text {
|
||||
background: #ebf8ff;
|
||||
border: 1px solid #bee3f8;
|
||||
}
|
||||
|
||||
.source-chip--clickable {
|
||||
cursor: pointer;
|
||||
transition: background 0.15s, border-color 0.15s;
|
||||
}
|
||||
|
||||
.source-chip--clickable:hover {
|
||||
background: #bee3f8;
|
||||
border-color: #90cdf4;
|
||||
}
|
||||
|
||||
.source-open-hint {
|
||||
font-size: 0.75rem;
|
||||
color: #3182ce;
|
||||
margin-left: 0.1rem;
|
||||
}
|
||||
|
||||
.source-chip--figure {
|
||||
background: #f0fff4;
|
||||
border: 1px solid #9ae6b4;
|
||||
}
|
||||
|
||||
.source-icon {
|
||||
font-size: 0.8rem;
|
||||
}
|
||||
|
||||
.source-book-title {
|
||||
color: #2b6cb0;
|
||||
font-weight: 500;
|
||||
}
|
||||
|
||||
.source-figure-label {
|
||||
color: #276749;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.source-figure-type {
|
||||
color: #718096;
|
||||
font-size: 0.72rem;
|
||||
background: #e2e8f0;
|
||||
border-radius: 3px;
|
||||
padding: 0 0.3rem;
|
||||
}
|
||||
|
||||
.source-page {
|
||||
color: #718096;
|
||||
}
|
||||
|
||||
.source-chunk {
|
||||
font-size: 0.78rem;
|
||||
color: #4a5568;
|
||||
@@ -194,28 +307,89 @@ function formatTime(iso: string): string {
|
||||
line-height: 1.5;
|
||||
}
|
||||
|
||||
.source-chip {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 0.25rem;
|
||||
background: #ebf8ff;
|
||||
border: 1px solid #bee3f8;
|
||||
border-radius: 4px;
|
||||
padding: 0.2rem 0.5rem;
|
||||
.source-caption {
|
||||
font-size: 0.78rem;
|
||||
color: #4a5568;
|
||||
font-style: italic;
|
||||
}
|
||||
|
||||
.source-book-icon {
|
||||
font-size: 0.8rem;
|
||||
.source-figure-image {
|
||||
max-width: 100%;
|
||||
}
|
||||
|
||||
.source-book-title {
|
||||
.figure-img {
|
||||
max-width: 100%;
|
||||
max-height: 300px;
|
||||
border-radius: 6px;
|
||||
border: 1px solid #e2e8f0;
|
||||
object-fit: contain;
|
||||
}
|
||||
|
||||
.figure-missing {
|
||||
font-size: 0.78rem;
|
||||
color: #a0aec0;
|
||||
font-style: italic;
|
||||
}
|
||||
|
||||
.message-content--markdown :deep(.citation-badge) {
|
||||
display: inline-block;
|
||||
background: #ebf8ff;
|
||||
border: 1px solid #90cdf4;
|
||||
border-radius: 3px;
|
||||
padding: 0 0.3em;
|
||||
font-size: 0.78em;
|
||||
font-weight: 600;
|
||||
color: #2b6cb0;
|
||||
font-weight: 500;
|
||||
cursor: pointer;
|
||||
user-select: none;
|
||||
transition: background 0.15s;
|
||||
}
|
||||
|
||||
.source-page {
|
||||
.message-content--markdown :deep(.citation-badge:hover) {
|
||||
background: #bee3f8;
|
||||
}
|
||||
|
||||
.source-item--active {
|
||||
outline: 2px solid #4299e1;
|
||||
border-radius: 6px;
|
||||
}
|
||||
|
||||
.source-ref-label {
|
||||
font-size: 0.72rem;
|
||||
font-weight: 700;
|
||||
background: #bee3f8;
|
||||
color: #2b6cb0;
|
||||
border-radius: 3px;
|
||||
padding: 0 0.3rem;
|
||||
}
|
||||
|
||||
.source-ref-label--figure {
|
||||
background: #9ae6b4;
|
||||
color: #276749;
|
||||
}
|
||||
|
||||
.message-content--markdown :deep(.inline-figure) {
|
||||
display: block;
|
||||
margin: 0.75rem 0;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.message-content--markdown :deep(.inline-figure-img) {
|
||||
max-width: 100%;
|
||||
max-height: 400px;
|
||||
border-radius: 6px;
|
||||
border: 1px solid #e2e8f0;
|
||||
object-fit: contain;
|
||||
display: block;
|
||||
margin: 0 auto;
|
||||
}
|
||||
|
||||
.message-content--markdown :deep(.inline-figure-caption) {
|
||||
font-size: 0.78rem;
|
||||
color: #718096;
|
||||
font-style: italic;
|
||||
margin-top: 0.3rem;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.message-timestamp {
|
||||
|
||||
@@ -0,0 +1,298 @@
|
||||
<template>
|
||||
<div class="source-list">
|
||||
<!-- TEXT sources -->
|
||||
<div
|
||||
v-for="(source, idx) in textSources"
|
||||
:key="'text-' + idx"
|
||||
class="source-item"
|
||||
:class="{ 'source-item--active': activeRef === source.refLabel }"
|
||||
:data-ref-label="source.refLabel"
|
||||
>
|
||||
<div class="source-chip-wrapper">
|
||||
<div
|
||||
class="source-chip source-chip--text"
|
||||
:class="{ 'source-chip--clickable': source.bookId && source.page }"
|
||||
@click="source.bookId && source.page ? emit('open-source', source.bookId, source.page) : undefined"
|
||||
>
|
||||
<span class="source-icon">📖</span>
|
||||
<span v-if="source.refLabel" class="source-ref-label">{{ source.refLabel }}</span>
|
||||
<span class="source-book-title">{{ source.bookTitle }}</span>
|
||||
<span v-if="source.page" class="source-page">p. {{ source.page }}</span>
|
||||
<span v-if="source.bookId && source.page" class="source-open-hint">↗</span>
|
||||
</div>
|
||||
<div v-if="source.chunkText" class="tooltip tooltip--text">
|
||||
<p class="tooltip-chunk">{{ source.chunkText }}</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- FIGURE sources -->
|
||||
<div
|
||||
v-for="(source, idx) in figureSources"
|
||||
:key="'fig-' + idx"
|
||||
class="source-item source-item--figure"
|
||||
:class="{ 'source-item--active': activeRef === source.refLabel }"
|
||||
:data-ref-label="source.refLabel"
|
||||
>
|
||||
<div class="source-chip-wrapper">
|
||||
<div
|
||||
class="source-chip source-chip--figure"
|
||||
:class="{ 'source-chip--clickable': source.bookId && source.page }"
|
||||
@click="source.bookId && source.page ? emit('open-source', source.bookId, source.page) : undefined"
|
||||
>
|
||||
<span class="source-icon">🖼️</span>
|
||||
<span v-if="source.refLabel" class="source-ref-label source-ref-label--figure">{{ source.refLabel }}</span>
|
||||
<span class="source-figure-label">{{ source.label || 'Figure' }}</span>
|
||||
<span v-if="source.page" class="source-page">p. {{ source.page }}</span>
|
||||
<span v-if="source.figureType" class="source-figure-type">{{ formatFigureType(source.figureType) }}</span>
|
||||
<span v-if="source.bookId && source.page" class="source-open-hint">↗</span>
|
||||
</div>
|
||||
<div v-if="source.imageUrl || source.caption" class="tooltip tooltip--figure">
|
||||
<img
|
||||
v-if="source.imageUrl"
|
||||
:src="source.imageUrl"
|
||||
:alt="source.caption || source.label || 'Figure'"
|
||||
class="tooltip-figure-img"
|
||||
loading="lazy"
|
||||
@error="onImageError"
|
||||
/>
|
||||
<p v-if="source.caption" class="tooltip-caption">{{ source.caption }}</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<script setup lang="ts">
|
||||
import { computed } from 'vue'
|
||||
|
||||
export interface SourceItem {
|
||||
type?: 'TEXT' | 'FIGURE'
|
||||
refLabel?: string
|
||||
bookId?: string | null
|
||||
bookTitle: string
|
||||
page?: number | null
|
||||
chunkText?: string
|
||||
figureId?: string
|
||||
label?: string
|
||||
caption?: string
|
||||
figureType?: string
|
||||
imageUrl?: string
|
||||
}
|
||||
|
||||
const props = defineProps<{
|
||||
sources: SourceItem[]
|
||||
activeRef?: string | null
|
||||
}>()
|
||||
|
||||
const emit = defineEmits<{
|
||||
'open-source': [bookId: string, page: number]
|
||||
}>()
|
||||
|
||||
const textSources = computed(() =>
|
||||
props.sources.filter(s => s.type === 'TEXT' || !s.type)
|
||||
)
|
||||
|
||||
const figureSources = computed(() =>
|
||||
props.sources.filter(s => s.type === 'FIGURE')
|
||||
)
|
||||
|
||||
function formatFigureType(type: string): string {
|
||||
const labels: Record<string, string> = {
|
||||
ANATOMICAL_DIAGRAM: 'Anatomical Diagram',
|
||||
SURGICAL_PHOTOGRAPH: 'Surgical Photo',
|
||||
MRI_CT_SCAN: 'MRI / CT',
|
||||
TABLE: 'Table',
|
||||
CHART: 'Chart',
|
||||
INTRAOPERATIVE_IMAGE: 'Intraoperative'
|
||||
}
|
||||
return labels[type] ?? type
|
||||
}
|
||||
|
||||
function onImageError(e: Event) {
|
||||
const img = e.target as HTMLImageElement
|
||||
img.style.display = 'none'
|
||||
const wrapper = img.parentElement
|
||||
if (wrapper) {
|
||||
const missing = document.createElement('span')
|
||||
missing.className = 'figure-missing'
|
||||
missing.textContent = 'Image unavailable'
|
||||
wrapper.appendChild(missing)
|
||||
}
|
||||
}
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
.source-list {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 0.5rem;
|
||||
}
|
||||
|
||||
.source-item {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 0.25rem;
|
||||
}
|
||||
|
||||
.source-item--active {
|
||||
outline: 2px solid #4299e1;
|
||||
border-radius: 6px;
|
||||
}
|
||||
|
||||
/* Wrapper provides the positioning context for the tooltip */
|
||||
.source-chip-wrapper {
|
||||
position: relative;
|
||||
display: inline-block;
|
||||
}
|
||||
|
||||
/* ── Chip base ── */
|
||||
.source-chip {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 0.25rem;
|
||||
border-radius: 4px;
|
||||
padding: 0.2rem 0.5rem;
|
||||
font-size: 0.78rem;
|
||||
}
|
||||
|
||||
.source-chip--text {
|
||||
background: #ebf8ff;
|
||||
border: 1px solid #bee3f8;
|
||||
}
|
||||
|
||||
.source-chip--figure {
|
||||
background: #f0fff4;
|
||||
border: 1px solid #9ae6b4;
|
||||
}
|
||||
|
||||
.source-chip--clickable {
|
||||
cursor: pointer;
|
||||
transition: background 0.15s, border-color 0.15s;
|
||||
}
|
||||
|
||||
.source-chip--clickable:hover {
|
||||
background: #bee3f8;
|
||||
border-color: #90cdf4;
|
||||
}
|
||||
|
||||
.source-chip--figure.source-chip--clickable:hover {
|
||||
background: #c6f6d5;
|
||||
border-color: #68d391;
|
||||
}
|
||||
|
||||
/* ── Tooltip ── */
|
||||
.tooltip {
|
||||
display: none;
|
||||
position: absolute;
|
||||
left: 0;
|
||||
top: calc(100% + 6px);
|
||||
z-index: 100;
|
||||
background: #1a202c;
|
||||
border-radius: 6px;
|
||||
padding: 0.6rem 0.75rem;
|
||||
box-shadow: 0 4px 16px rgba(0, 0, 0, 0.2);
|
||||
/* Keep it from overflowing too far */
|
||||
max-width: min(340px, 80vw);
|
||||
pointer-events: none;
|
||||
}
|
||||
|
||||
/* Show on chip hover */
|
||||
.source-chip-wrapper:hover .tooltip {
|
||||
display: block;
|
||||
}
|
||||
|
||||
/* Small arrow pointing up */
|
||||
.tooltip::before {
|
||||
content: '';
|
||||
position: absolute;
|
||||
top: -5px;
|
||||
left: 14px;
|
||||
border-left: 5px solid transparent;
|
||||
border-right: 5px solid transparent;
|
||||
border-bottom: 5px solid #1a202c;
|
||||
}
|
||||
|
||||
.tooltip--text .tooltip-chunk {
|
||||
margin: 0;
|
||||
font-size: 0.78rem;
|
||||
color: #e2e8f0;
|
||||
line-height: 1.5;
|
||||
white-space: pre-wrap;
|
||||
word-break: break-word;
|
||||
}
|
||||
|
||||
.tooltip--figure {
|
||||
max-width: min(300px, 80vw);
|
||||
}
|
||||
|
||||
.tooltip-figure-img {
|
||||
display: block;
|
||||
max-width: 100%;
|
||||
max-height: 220px;
|
||||
border-radius: 4px;
|
||||
object-fit: contain;
|
||||
margin-bottom: 0.4rem;
|
||||
}
|
||||
|
||||
.tooltip-caption {
|
||||
margin: 0;
|
||||
font-size: 0.75rem;
|
||||
color: #cbd5e0;
|
||||
font-style: italic;
|
||||
line-height: 1.4;
|
||||
}
|
||||
|
||||
/* ── Chip internals ── */
|
||||
.source-icon {
|
||||
font-size: 0.8rem;
|
||||
}
|
||||
|
||||
.source-ref-label {
|
||||
font-size: 0.72rem;
|
||||
font-weight: 700;
|
||||
background: #bee3f8;
|
||||
color: #2b6cb0;
|
||||
border-radius: 3px;
|
||||
padding: 0 0.3rem;
|
||||
}
|
||||
|
||||
.source-ref-label--figure {
|
||||
background: #9ae6b4;
|
||||
color: #276749;
|
||||
}
|
||||
|
||||
.source-book-title {
|
||||
color: #2b6cb0;
|
||||
font-weight: 500;
|
||||
}
|
||||
|
||||
.source-figure-label {
|
||||
color: #276749;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.source-figure-type {
|
||||
color: #718096;
|
||||
font-size: 0.72rem;
|
||||
background: #e2e8f0;
|
||||
border-radius: 3px;
|
||||
padding: 0 0.3rem;
|
||||
}
|
||||
|
||||
.source-page {
|
||||
color: #718096;
|
||||
}
|
||||
|
||||
.source-open-hint {
|
||||
font-size: 0.75rem;
|
||||
color: #3182ce;
|
||||
margin-left: 0.1rem;
|
||||
}
|
||||
|
||||
.figure-missing {
|
||||
font-size: 0.78rem;
|
||||
color: #a0aec0;
|
||||
font-style: italic;
|
||||
}
|
||||
</style>
|
||||
Vendored
+2
@@ -3,6 +3,8 @@
|
||||
interface ImportMetaEnv {
|
||||
readonly VITE_API_URL: string
|
||||
readonly VITE_APP_PASSWORD: string
|
||||
readonly VITE_UPLOAD_ENABLED: string
|
||||
readonly VITE_DELETE_ENABLED: string
|
||||
}
|
||||
|
||||
interface ImportMeta {
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
/**
|
||||
* Read a VITE_ env variable.
|
||||
* At runtime in Docker, values come from window.__env__ (injected by docker-entrypoint.sh).
|
||||
* At build time (dev / CI), values come from import.meta.env.
|
||||
*/
|
||||
export function env(key: string): string | undefined {
|
||||
const runtime = (window as Record<string, any>).__env__?.[key]
|
||||
if (runtime) return runtime
|
||||
return (import.meta as any).env?.[key]
|
||||
}
|
||||
+16
-1
@@ -4,6 +4,21 @@ import App from './App.vue'
|
||||
import router from './router'
|
||||
|
||||
const app = createApp(App)
|
||||
app.use(createPinia())
|
||||
const pinia = createPinia()
|
||||
app.use(pinia)
|
||||
app.use(router)
|
||||
|
||||
// Verify any session restored from sessionStorage is still valid.
|
||||
// If the backend rejects the credentials (e.g. password changed), clear them
|
||||
// before the router guard fires so the user lands on /login cleanly.
|
||||
import { useAuthStore } from '@/stores/authStore'
|
||||
import { api } from '@/services/api'
|
||||
|
||||
const auth = useAuthStore()
|
||||
if (auth.isAuthenticated) {
|
||||
api.get('/auth/check').catch(() => {
|
||||
auth.clearCredentials()
|
||||
})
|
||||
}
|
||||
|
||||
app.mount('#app')
|
||||
|
||||
@@ -1,11 +1,19 @@
|
||||
import { createRouter, createWebHistory } from 'vue-router'
|
||||
import { useAuthStore } from '@/stores/authStore'
|
||||
import LoginView from '@/views/LoginView.vue'
|
||||
import UploadView from '@/views/UploadView.vue'
|
||||
import TopicsView from '@/views/TopicsView.vue'
|
||||
import ChatView from '@/views/ChatView.vue'
|
||||
import BookReaderView from '@/views/BookReaderView.vue'
|
||||
|
||||
const router = createRouter({
|
||||
history: createWebHistory(import.meta.env.BASE_URL),
|
||||
routes: [
|
||||
{
|
||||
path: '/login',
|
||||
name: 'login',
|
||||
component: LoginView
|
||||
},
|
||||
{
|
||||
path: '/',
|
||||
name: 'upload',
|
||||
@@ -20,8 +28,20 @@ const router = createRouter({
|
||||
path: '/chat',
|
||||
name: 'chat',
|
||||
component: ChatView
|
||||
},
|
||||
{
|
||||
path: '/books/:id/read',
|
||||
name: 'book-reader',
|
||||
component: BookReaderView
|
||||
}
|
||||
]
|
||||
})
|
||||
|
||||
router.beforeEach((to) => {
|
||||
const auth = useAuthStore()
|
||||
if (to.name !== 'login' && !auth.isAuthenticated) {
|
||||
return { name: 'login' }
|
||||
}
|
||||
})
|
||||
|
||||
export default router
|
||||
|
||||
@@ -1,20 +1,30 @@
|
||||
import axios from 'axios'
|
||||
import { useAuthStore } from '@/stores/authStore'
|
||||
import { env } from '@/env'
|
||||
|
||||
export const api = axios.create({
|
||||
baseURL: import.meta.env.VITE_API_URL ?? '/api/v1',
|
||||
auth: {
|
||||
username: 'neurosurgeon',
|
||||
password: import.meta.env.VITE_APP_PASSWORD ?? 'changeme'
|
||||
},
|
||||
baseURL: env('VITE_API_URL') ?? '/api/v1',
|
||||
headers: {
|
||||
'Content-Type': 'application/json'
|
||||
}
|
||||
})
|
||||
|
||||
// Response interceptor for error normalisation
|
||||
api.interceptors.request.use((config) => {
|
||||
const auth = useAuthStore()
|
||||
if (auth.username && auth.password) {
|
||||
config.auth = { username: auth.username, password: auth.password }
|
||||
}
|
||||
return config
|
||||
})
|
||||
|
||||
api.interceptors.response.use(
|
||||
(response) => response,
|
||||
(error) => {
|
||||
if (error.response?.status === 401) {
|
||||
useAuthStore().clearCredentials()
|
||||
window.location.href = '/login'
|
||||
return Promise.reject(new Error('Session expired. Please sign in again.'))
|
||||
}
|
||||
const message =
|
||||
error.response?.data?.error ??
|
||||
error.message ??
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
import { defineStore } from 'pinia'
|
||||
import { ref, computed } from 'vue'
|
||||
|
||||
const SESSION_KEY = 'auth'
|
||||
|
||||
export const useAuthStore = defineStore('auth', () => {
|
||||
const stored = sessionStorage.getItem(SESSION_KEY)
|
||||
const parsed = stored ? (JSON.parse(stored) as { username: string; password: string }) : null
|
||||
|
||||
const username = ref<string | null>(parsed?.username ?? null)
|
||||
const password = ref<string | null>(parsed?.password ?? null)
|
||||
|
||||
const isAuthenticated = computed(() => !!username.value && !!password.value)
|
||||
|
||||
function setCredentials(u: string, p: string) {
|
||||
username.value = u
|
||||
password.value = p
|
||||
sessionStorage.setItem(SESSION_KEY, JSON.stringify({ username: u, password: p }))
|
||||
}
|
||||
|
||||
function clearCredentials() {
|
||||
username.value = null
|
||||
password.value = null
|
||||
sessionStorage.removeItem(SESSION_KEY)
|
||||
}
|
||||
|
||||
return { username, password, isAuthenticated, setCredentials, clearCredentials }
|
||||
})
|
||||
@@ -77,5 +77,42 @@ export const useBookStore = defineStore('books', () => {
|
||||
}
|
||||
}
|
||||
|
||||
return { books, loading, uploading, error, fetchBooks, uploadBook, refreshBook, deleteBook }
|
||||
async function startEnrichment(id: string): Promise<EnrichmentProgress | null> {
|
||||
try {
|
||||
const response = await api.post<EnrichmentProgress>(`/admin/books/${id}/enrich`)
|
||||
return response.data
|
||||
} catch (err: any) {
|
||||
error.value = err.message
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
async function fetchEnrichmentStatus(id: string): Promise<EnrichmentProgress | null> {
|
||||
try {
|
||||
const response = await api.get<EnrichmentProgress>(`/admin/books/${id}/enrich`)
|
||||
return response.data
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
books,
|
||||
loading,
|
||||
uploading,
|
||||
error,
|
||||
fetchBooks,
|
||||
uploadBook,
|
||||
refreshBook,
|
||||
deleteBook,
|
||||
startEnrichment,
|
||||
fetchEnrichmentStatus
|
||||
}
|
||||
})
|
||||
|
||||
export interface EnrichmentProgress {
|
||||
status: 'IDLE' | 'RUNNING' | 'COMPLETED'
|
||||
chunksTotal: number
|
||||
chunksEnriched: number
|
||||
errorMessage: string | null
|
||||
}
|
||||
|
||||
@@ -2,11 +2,27 @@ import { defineStore } from 'pinia'
|
||||
import { ref } from 'vue'
|
||||
import { api } from '@/services/api'
|
||||
|
||||
export interface ChatSource {
|
||||
type: 'TEXT' | 'FIGURE'
|
||||
bookId?: string
|
||||
bookTitle: string
|
||||
page: number | null
|
||||
refLabel?: string
|
||||
// TEXT-specific
|
||||
chunkText?: string
|
||||
// FIGURE-specific
|
||||
figureId?: string
|
||||
label?: string
|
||||
caption?: string
|
||||
figureType?: string
|
||||
imageUrl?: string
|
||||
}
|
||||
|
||||
export interface ChatMessage {
|
||||
id: string
|
||||
role: 'USER' | 'ASSISTANT'
|
||||
content: string
|
||||
sources: Array<{ bookTitle: string; page: number | null; chunkText?: string }>
|
||||
sources: ChatSource[]
|
||||
createdAt: string
|
||||
}
|
||||
|
||||
|
||||
@@ -10,11 +10,22 @@ export interface Topic {
|
||||
}
|
||||
|
||||
export interface SourceReference {
|
||||
type?: 'TEXT' | 'FIGURE'
|
||||
refLabel?: string
|
||||
bookId: string | null
|
||||
bookTitle: string
|
||||
page: number | null
|
||||
chunkText?: string
|
||||
figureId?: string
|
||||
label?: string
|
||||
caption?: string
|
||||
figureType?: string
|
||||
imageUrl?: string
|
||||
}
|
||||
|
||||
export interface TopicSummary {
|
||||
id: string
|
||||
summaryNumber: number
|
||||
topicId: string
|
||||
topicName: string
|
||||
summary: string
|
||||
@@ -22,14 +33,50 @@ export interface TopicSummary {
|
||||
generatedAt: string
|
||||
}
|
||||
|
||||
export interface SavedSummaryItem {
|
||||
id: string
|
||||
summaryNumber: number
|
||||
generatedAt: string
|
||||
}
|
||||
|
||||
export interface FacetSection {
|
||||
facetKey: string
|
||||
title: string
|
||||
markdown: string
|
||||
refLabels: string[]
|
||||
}
|
||||
|
||||
export interface ConceptReport {
|
||||
id: string
|
||||
reportNumber: number
|
||||
topicId: string
|
||||
topicName: string
|
||||
facets: FacetSection[]
|
||||
sources: SourceReference[]
|
||||
generatedAt: string
|
||||
}
|
||||
|
||||
export interface SavedConceptReportItem {
|
||||
id: string
|
||||
reportNumber: number
|
||||
generatedAt: string
|
||||
}
|
||||
|
||||
export const useTopicStore = defineStore('topics', () => {
|
||||
const topics = ref<Topic[]>([])
|
||||
const activeSummary = ref<TopicSummary | null>(null)
|
||||
const activeSummaryTopicId = ref<string | null>(null)
|
||||
const summaryList = ref<SavedSummaryItem[]>([])
|
||||
const loading = ref(false)
|
||||
const summaryLoading = ref(false)
|
||||
const summaryListLoading = ref(false)
|
||||
const error = ref<string | null>(null)
|
||||
|
||||
const activeConceptReport = ref<ConceptReport | null>(null)
|
||||
const conceptReportList = ref<SavedConceptReportItem[]>([])
|
||||
const conceptReportLoading = ref(false)
|
||||
const conceptReportListLoading = ref(false)
|
||||
|
||||
async function fetchTopics() {
|
||||
loading.value = true
|
||||
error.value = null
|
||||
@@ -43,13 +90,47 @@ export const useTopicStore = defineStore('topics', () => {
|
||||
}
|
||||
}
|
||||
|
||||
async function generateSummary(topicId: string): Promise<TopicSummary | null> {
|
||||
async function fetchSummaries(topicId: string) {
|
||||
summaryListLoading.value = true
|
||||
summaryList.value = []
|
||||
error.value = null
|
||||
try {
|
||||
const response = await api.get<SavedSummaryItem[]>(`/topics/${topicId}/summaries`)
|
||||
summaryList.value = response.data
|
||||
} catch (err: any) {
|
||||
error.value = err.message
|
||||
} finally {
|
||||
summaryListLoading.value = false
|
||||
}
|
||||
}
|
||||
|
||||
async function fetchSummaryDetail(topicId: string, summaryId: string): Promise<TopicSummary | null> {
|
||||
summaryLoading.value = true
|
||||
activeSummary.value = null
|
||||
error.value = null
|
||||
try {
|
||||
const response = await api.get<TopicSummary>(`/topics/${topicId}/summaries/${summaryId}`)
|
||||
activeSummary.value = response.data
|
||||
return response.data
|
||||
} catch (err: any) {
|
||||
error.value = err.message
|
||||
return null
|
||||
} finally {
|
||||
summaryLoading.value = false
|
||||
}
|
||||
}
|
||||
|
||||
async function generateSummary(topicId: string, language: 'en' | 'th' = 'en'): Promise<TopicSummary | null> {
|
||||
summaryLoading.value = true
|
||||
activeSummaryTopicId.value = topicId
|
||||
activeSummary.value = null
|
||||
error.value = null
|
||||
try {
|
||||
const response = await api.post<TopicSummary>(`/topics/${topicId}/summary`)
|
||||
const response = await api.post<TopicSummary>(
|
||||
`/topics/${topicId}/summary`,
|
||||
null,
|
||||
{ params: { language } }
|
||||
)
|
||||
activeSummary.value = response.data
|
||||
return response.data
|
||||
} catch (err: any) {
|
||||
@@ -61,14 +142,75 @@ export const useTopicStore = defineStore('topics', () => {
|
||||
}
|
||||
}
|
||||
|
||||
async function fetchConceptReports(topicId: string) {
|
||||
conceptReportListLoading.value = true
|
||||
conceptReportList.value = []
|
||||
error.value = null
|
||||
try {
|
||||
const response = await api.get<SavedConceptReportItem[]>(`/topics/${topicId}/concept-reports`)
|
||||
conceptReportList.value = response.data
|
||||
} catch (err: any) {
|
||||
error.value = err.message
|
||||
} finally {
|
||||
conceptReportListLoading.value = false
|
||||
}
|
||||
}
|
||||
|
||||
async function fetchConceptReportDetail(topicId: string, reportId: string): Promise<ConceptReport | null> {
|
||||
conceptReportLoading.value = true
|
||||
activeConceptReport.value = null
|
||||
error.value = null
|
||||
try {
|
||||
const response = await api.get<ConceptReport>(`/topics/${topicId}/concept-reports/${reportId}`)
|
||||
activeConceptReport.value = response.data
|
||||
return response.data
|
||||
} catch (err: any) {
|
||||
error.value = err.message
|
||||
return null
|
||||
} finally {
|
||||
conceptReportLoading.value = false
|
||||
}
|
||||
}
|
||||
|
||||
async function generateConceptReport(topicId: string, language: 'en' | 'th' = 'en'): Promise<ConceptReport | null> {
|
||||
conceptReportLoading.value = true
|
||||
activeConceptReport.value = null
|
||||
error.value = null
|
||||
try {
|
||||
const response = await api.post<ConceptReport>(
|
||||
`/topics/${topicId}/concept-reports`,
|
||||
null,
|
||||
{ params: { language } }
|
||||
)
|
||||
activeConceptReport.value = response.data
|
||||
return response.data
|
||||
} catch (err: any) {
|
||||
error.value = err.message
|
||||
return null
|
||||
} finally {
|
||||
conceptReportLoading.value = false
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
topics,
|
||||
activeSummary,
|
||||
activeSummaryTopicId,
|
||||
summaryList,
|
||||
loading,
|
||||
summaryLoading,
|
||||
summaryListLoading,
|
||||
error,
|
||||
activeConceptReport,
|
||||
conceptReportList,
|
||||
conceptReportLoading,
|
||||
conceptReportListLoading,
|
||||
fetchTopics,
|
||||
generateSummary
|
||||
fetchSummaries,
|
||||
fetchSummaryDetail,
|
||||
generateSummary,
|
||||
fetchConceptReports,
|
||||
fetchConceptReportDetail,
|
||||
generateConceptReport
|
||||
}
|
||||
})
|
||||
|
||||
@@ -0,0 +1,335 @@
|
||||
<template>
|
||||
<div class="reader-view">
|
||||
<!-- Header -->
|
||||
<div class="reader-header">
|
||||
<router-link to="/" class="back-link">← Library</router-link>
|
||||
<div class="reader-title">
|
||||
<h1 class="book-title">{{ book?.title ?? 'Loading…' }}</h1>
|
||||
</div>
|
||||
<div class="page-nav">
|
||||
<button class="nav-btn" :disabled="currentPage <= 1" @click="goTo(currentPage - 1)">←</button>
|
||||
<form class="page-jump" @submit.prevent="onJump">
|
||||
<input
|
||||
v-model.number="jumpInput"
|
||||
type="number"
|
||||
:min="1"
|
||||
:max="book?.pageCount ?? 1"
|
||||
class="page-input"
|
||||
/>
|
||||
<span class="page-sep">/ {{ book?.pageCount ?? '…' }}</span>
|
||||
</form>
|
||||
<button class="nav-btn" :disabled="!book || currentPage >= book.pageCount!" @click="goTo(currentPage + 1)">→</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Content -->
|
||||
<div class="reader-body">
|
||||
<div v-if="loading" class="reader-loading">
|
||||
<div class="spinner spinner-dark" style="width:28px;height:28px;margin:0 auto 0.75rem;"></div>
|
||||
<p>Loading page {{ currentPage }}…</p>
|
||||
</div>
|
||||
|
||||
<div v-else-if="error" class="reader-error card">
|
||||
<strong>Could not load page {{ currentPage }}</strong><br />
|
||||
{{ error }}
|
||||
</div>
|
||||
|
||||
<div v-else class="reader-content card">
|
||||
<div class="markdown-body" v-html="renderedHtml"></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<script setup lang="ts">
|
||||
import { ref, watch, onMounted } from 'vue'
|
||||
import { useRoute } from 'vue-router'
|
||||
import { api } from '@/services/api'
|
||||
import { useBookStore } from '@/stores/bookStore'
|
||||
import type { Book } from '@/stores/bookStore'
|
||||
|
||||
const route = useRoute()
|
||||
const bookStore = useBookStore()
|
||||
|
||||
const bookId = route.params.id as string
|
||||
const book = ref<Book | null>(null)
|
||||
const currentPage = ref(1)
|
||||
const jumpInput = ref(1)
|
||||
const loading = ref(false)
|
||||
const error = ref<string | null>(null)
|
||||
const renderedHtml = ref('')
|
||||
|
||||
// Blob URLs created this session — revoked on next page load
|
||||
let activeBlobUrls: string[] = []
|
||||
|
||||
onMounted(async () => {
|
||||
book.value = bookStore.books.find(b => b.id === bookId) ?? null
|
||||
if (!book.value) {
|
||||
try {
|
||||
const res = await api.get<Book>(`/books/${bookId}`)
|
||||
book.value = res.data
|
||||
} catch {
|
||||
error.value = 'Book not found.'
|
||||
return
|
||||
}
|
||||
}
|
||||
await loadPage(1)
|
||||
})
|
||||
|
||||
watch(currentPage, (page) => {
|
||||
jumpInput.value = page
|
||||
loadPage(page)
|
||||
})
|
||||
|
||||
async function goTo(page: number) {
|
||||
if (!book.value) return
|
||||
const clamped = Math.max(1, Math.min(page, book.value.pageCount ?? 1))
|
||||
if (clamped !== currentPage.value) {
|
||||
currentPage.value = clamped
|
||||
}
|
||||
}
|
||||
|
||||
function onJump() {
|
||||
goTo(jumpInput.value)
|
||||
}
|
||||
|
||||
async function loadPage(page: number) {
|
||||
loading.value = true
|
||||
error.value = null
|
||||
renderedHtml.value = ''
|
||||
|
||||
// Revoke previous blob URLs to free memory
|
||||
activeBlobUrls.forEach(u => URL.revokeObjectURL(u))
|
||||
activeBlobUrls = []
|
||||
|
||||
try {
|
||||
const res = await api.get<string>(`/books/${bookId}/pages/${page}/html`, {
|
||||
headers: { Accept: 'text/html' },
|
||||
responseType: 'text'
|
||||
})
|
||||
let html = await resolveImages(res.data)
|
||||
renderedHtml.value = html
|
||||
} catch (e: any) {
|
||||
error.value = e.message ?? 'Failed to load page.'
|
||||
} finally {
|
||||
loading.value = false
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Finds <img src="/api/v1/figures/..."> in the HTML, fetches each image
|
||||
* through the authenticated axios instance, and replaces the src with a
|
||||
* temporary blob URL so the browser can render it without re-authenticating.
|
||||
*/
|
||||
async function resolveImages(html: string): Promise<string> {
|
||||
const srcPattern = /src="(\/api\/v1\/figures\/[^"]+)"/g
|
||||
const matches = [...html.matchAll(srcPattern)]
|
||||
if (matches.length === 0) return html
|
||||
|
||||
const unique = [...new Set(matches.map(m => m[1]))]
|
||||
const blobMap: Record<string, string> = {}
|
||||
|
||||
await Promise.all(
|
||||
unique.map(async (src) => {
|
||||
try {
|
||||
const res = await api.get(src.replace(/^\/api\/v1/, ''), { responseType: 'blob' })
|
||||
const blobUrl = URL.createObjectURL(res.data)
|
||||
activeBlobUrls.push(blobUrl)
|
||||
blobMap[src] = blobUrl
|
||||
} catch {
|
||||
// leave original src — browser will attempt (and likely fail silently)
|
||||
}
|
||||
})
|
||||
)
|
||||
|
||||
return html.replace(/src="(\/api\/v1\/figures\/[^"]+)"/g, (_, src) =>
|
||||
blobMap[src] ? `src="${blobMap[src]}"` : `src="${src}"`
|
||||
)
|
||||
}
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
.reader-view {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 1rem;
|
||||
max-width: 860px;
|
||||
margin: 0 auto;
|
||||
flex: 1;
|
||||
min-height: 0;
|
||||
}
|
||||
|
||||
.reader-header {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 1rem;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
.back-link {
|
||||
color: #3182ce;
|
||||
text-decoration: none;
|
||||
font-size: 0.9rem;
|
||||
white-space: nowrap;
|
||||
}
|
||||
.back-link:hover { text-decoration: underline; }
|
||||
|
||||
.reader-title {
|
||||
flex: 1;
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.book-title {
|
||||
font-size: 1.1rem;
|
||||
font-weight: 600;
|
||||
color: #1a365d;
|
||||
white-space: nowrap;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
}
|
||||
|
||||
.page-nav {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 0.5rem;
|
||||
}
|
||||
|
||||
.nav-btn {
|
||||
width: 2rem;
|
||||
height: 2rem;
|
||||
border: 1px solid #cbd5e0;
|
||||
border-radius: 6px;
|
||||
background: #fff;
|
||||
cursor: pointer;
|
||||
font-size: 1rem;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
transition: background 0.15s;
|
||||
}
|
||||
.nav-btn:hover:not(:disabled) { background: #ebf8ff; border-color: #3182ce; }
|
||||
.nav-btn:disabled { opacity: 0.4; cursor: not-allowed; }
|
||||
|
||||
.page-jump {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 0.35rem;
|
||||
}
|
||||
|
||||
.page-input {
|
||||
width: 3.5rem;
|
||||
text-align: center;
|
||||
border: 1px solid #cbd5e0;
|
||||
border-radius: 6px;
|
||||
padding: 0.25rem 0.4rem;
|
||||
font-size: 0.9rem;
|
||||
color: #2d3748;
|
||||
}
|
||||
.page-input:focus { outline: none; border-color: #3182ce; }
|
||||
|
||||
.page-sep {
|
||||
font-size: 0.85rem;
|
||||
color: #718096;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.reader-body {
|
||||
flex: 1;
|
||||
min-height: 0;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
}
|
||||
|
||||
.reader-loading {
|
||||
text-align: center;
|
||||
padding: 3rem;
|
||||
color: #718096;
|
||||
}
|
||||
|
||||
.reader-error {
|
||||
padding: 1.25rem;
|
||||
background: #fff5f5;
|
||||
border: 1px solid #fed7d7;
|
||||
color: #742a2a;
|
||||
border-radius: 8px;
|
||||
}
|
||||
|
||||
.reader-content {
|
||||
flex: 1;
|
||||
min-height: 0;
|
||||
overflow-y: auto;
|
||||
padding: 2rem;
|
||||
}
|
||||
|
||||
/* Markdown rendering */
|
||||
.markdown-body {
|
||||
font-size: 0.95rem;
|
||||
line-height: 1.75;
|
||||
color: #2d3748;
|
||||
}
|
||||
|
||||
.markdown-body :deep(h1),
|
||||
.markdown-body :deep(h2),
|
||||
.markdown-body :deep(h3) {
|
||||
color: #1a365d;
|
||||
font-weight: 600;
|
||||
margin: 1.5rem 0 0.75rem;
|
||||
}
|
||||
.markdown-body :deep(h2) { font-size: 1.15rem; border-bottom: 1px solid #e2e8f0; padding-bottom: 0.4rem; }
|
||||
.markdown-body :deep(h3) { font-size: 1rem; }
|
||||
|
||||
.markdown-body :deep(p) { margin: 0.75rem 0; }
|
||||
|
||||
.markdown-body :deep(img) {
|
||||
max-width: 100%;
|
||||
border-radius: 6px;
|
||||
display: block;
|
||||
margin: 1rem auto;
|
||||
box-shadow: 0 1px 4px rgba(0,0,0,0.12);
|
||||
}
|
||||
|
||||
.markdown-body :deep(ul),
|
||||
.markdown-body :deep(ol) {
|
||||
padding-left: 1.5rem;
|
||||
margin: 0.75rem 0;
|
||||
}
|
||||
|
||||
.markdown-body :deep(code) {
|
||||
background: #f7fafc;
|
||||
border: 1px solid #e2e8f0;
|
||||
border-radius: 3px;
|
||||
padding: 0.1em 0.35em;
|
||||
font-size: 0.88em;
|
||||
}
|
||||
|
||||
.markdown-body :deep(blockquote) {
|
||||
border-left: 3px solid #3182ce;
|
||||
padding-left: 1rem;
|
||||
color: #4a5568;
|
||||
margin: 0.75rem 0;
|
||||
}
|
||||
|
||||
.markdown-body :deep(table) {
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
font-size: 0.9em;
|
||||
margin: 1rem 0;
|
||||
}
|
||||
.markdown-body :deep(th),
|
||||
.markdown-body :deep(td) {
|
||||
border: 1px solid #e2e8f0;
|
||||
padding: 0.4rem 0.75rem;
|
||||
text-align: left;
|
||||
}
|
||||
.markdown-body :deep(th) { background: #f7fafc; font-weight: 600; }
|
||||
|
||||
@media (max-width: 768px) {
|
||||
.reader-view {
|
||||
max-width: 100%;
|
||||
}
|
||||
|
||||
.reader-content {
|
||||
padding: 1rem;
|
||||
}
|
||||
}
|
||||
</style>
|
||||
+125
-123
@@ -3,27 +3,10 @@
|
||||
<h1 class="page-title">Knowledge Chat</h1>
|
||||
<p class="page-subtitle">Ask questions grounded in your uploaded medical textbooks.</p>
|
||||
|
||||
<!-- Step 1: Topic Selection -->
|
||||
<div v-if="!chatStore.session && !selectedTopic" class="topic-selection">
|
||||
<h2 class="section-title">Select a Topic</h2>
|
||||
<div class="topic-grid">
|
||||
<button
|
||||
v-for="topic in topicStore.topics"
|
||||
:key="topic.id"
|
||||
:class="['topic-tile', { 'topic-tile-freeform': topic.id === 'free-form' }]"
|
||||
@click="handleTopicSelect(topic)"
|
||||
>
|
||||
<span class="topic-tile-name">{{ topic.name }}</span>
|
||||
<span v-if="topic.id === 'free-form'" class="topic-tile-hint">Any neurosurgery question</span>
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Step 2: Topic selected — previous sessions + new chat -->
|
||||
<div v-else-if="!chatStore.session && selectedTopic" class="session-setup card">
|
||||
<!-- Session selection -->
|
||||
<div v-if="!chatStore.session" class="session-setup card">
|
||||
<div class="setup-header">
|
||||
<button class="btn-back" @click="handleBack">← Topics</button>
|
||||
<h2 class="section-title">{{ selectedTopic.name }}</h2>
|
||||
<h2 class="section-title">Free-form Chat</h2>
|
||||
</div>
|
||||
|
||||
<div v-if="chatStore.error" class="error-banner">{{ chatStore.error }}</div>
|
||||
@@ -71,56 +54,74 @@
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Messages Area -->
|
||||
<div class="messages-container" ref="messagesContainer">
|
||||
<div v-if="chatStore.loading && chatStore.messages.length === 0" class="empty-state">
|
||||
<div class="spinner spinner-dark" style="width:32px;height:32px;margin:0 auto 1rem;"></div>
|
||||
<p class="empty-state-text">Loading messages...</p>
|
||||
</div>
|
||||
<!-- Chat + Reader split -->
|
||||
<div class="chat-reader-split">
|
||||
<!-- Messages + Input -->
|
||||
<div class="chat-column">
|
||||
<!-- Messages Area -->
|
||||
<div class="messages-container" ref="messagesContainer">
|
||||
<div v-if="chatStore.loading && chatStore.messages.length === 0" class="empty-state">
|
||||
<div class="spinner spinner-dark" style="width:32px;height:32px;margin:0 auto 1rem;"></div>
|
||||
<p class="empty-state-text">Loading messages...</p>
|
||||
</div>
|
||||
|
||||
<div v-else-if="chatStore.messages.length === 0" class="empty-state">
|
||||
<div class="empty-state-icon">💬</div>
|
||||
<p class="empty-state-text">No messages yet</p>
|
||||
<p class="empty-state-hint">Ask a question about the uploaded books below.</p>
|
||||
</div>
|
||||
<div v-else-if="chatStore.messages.length === 0" class="empty-state">
|
||||
<div class="empty-state-icon">💬</div>
|
||||
<p class="empty-state-text">No messages yet</p>
|
||||
<p class="empty-state-hint">Ask a question about the uploaded books below.</p>
|
||||
</div>
|
||||
|
||||
<div v-else class="messages-list">
|
||||
<ChatMessage
|
||||
v-for="message in chatStore.messages"
|
||||
:key="message.id"
|
||||
:message="message"
|
||||
/>
|
||||
<div v-if="chatStore.sending" class="typing-indicator">
|
||||
<div class="typing-bubble">
|
||||
<span></span><span></span><span></span>
|
||||
<div v-else class="messages-list">
|
||||
<ChatMessage
|
||||
v-for="message in chatStore.messages"
|
||||
:key="message.id"
|
||||
:message="message"
|
||||
@open-source="handleOpenSource"
|
||||
/>
|
||||
<div v-if="chatStore.sending" class="typing-indicator">
|
||||
<div class="typing-bubble">
|
||||
<span></span><span></span><span></span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Input Area -->
|
||||
<div class="input-area card">
|
||||
<div v-if="chatStore.error" class="error-banner">{{ chatStore.error }}</div>
|
||||
<div class="input-row">
|
||||
<textarea
|
||||
v-model="inputText"
|
||||
class="message-input"
|
||||
placeholder="Ask a question about your uploaded books..."
|
||||
rows="2"
|
||||
:disabled="chatStore.sending"
|
||||
@keydown.enter.exact.prevent="handleSend"
|
||||
@keydown.enter.shift.exact="inputText += '\n'"
|
||||
></textarea>
|
||||
<button
|
||||
class="btn btn-primary send-btn"
|
||||
:disabled="!inputText.trim() || chatStore.sending"
|
||||
@click="handleSend"
|
||||
>
|
||||
<span v-if="chatStore.sending" class="spinner"></span>
|
||||
<span v-else>Send</span>
|
||||
</button>
|
||||
<!-- Input Area -->
|
||||
<div class="input-area card">
|
||||
<div v-if="chatStore.error" class="error-banner">{{ chatStore.error }}</div>
|
||||
<div class="input-row">
|
||||
<textarea
|
||||
v-model="inputText"
|
||||
class="message-input"
|
||||
placeholder="Ask a question about your uploaded books..."
|
||||
rows="2"
|
||||
:disabled="chatStore.sending"
|
||||
@keydown.enter.exact.prevent="handleSend"
|
||||
@keydown.enter.shift.exact="inputText += '\n'"
|
||||
></textarea>
|
||||
<button
|
||||
class="btn btn-primary send-btn"
|
||||
:disabled="!inputText.trim() || chatStore.sending"
|
||||
@click="handleSend"
|
||||
>
|
||||
<span v-if="chatStore.sending" class="spinner"></span>
|
||||
<span v-else>Send</span>
|
||||
</button>
|
||||
</div>
|
||||
<p class="input-hint">Press Enter to send, Shift+Enter for new line.</p>
|
||||
</div>
|
||||
</div>
|
||||
<p class="input-hint">Press Enter to send, Shift+Enter for new line.</p>
|
||||
|
||||
<!-- Inline book reader panel -->
|
||||
<BookPagePanel
|
||||
v-if="readerPanel"
|
||||
:book-id="readerPanel.bookId"
|
||||
:page="readerPanel.page"
|
||||
:book-title="readerPanel.bookTitle"
|
||||
class="reader-panel"
|
||||
@close="readerPanel = null"
|
||||
@navigate="(p) => readerPanel && (readerPanel.page = p)"
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -130,8 +131,10 @@
|
||||
import { ref, nextTick, onMounted, watch, inject } from 'vue'
|
||||
import { useChatStore } from '@/stores/chatStore'
|
||||
import { useTopicStore } from '@/stores/topicStore'
|
||||
import { useBookStore } from '@/stores/bookStore'
|
||||
import type { ChatSession } from '@/stores/chatStore'
|
||||
import ChatMessage from '@/components/ChatMessage.vue'
|
||||
import BookPagePanel from '@/components/BookPagePanel.vue'
|
||||
|
||||
interface Topic {
|
||||
id: string
|
||||
@@ -142,6 +145,7 @@ interface Topic {
|
||||
|
||||
const chatStore = useChatStore()
|
||||
const topicStore = useTopicStore()
|
||||
const bookStore = useBookStore()
|
||||
const showToast = inject<(msg: string, type?: 'error' | 'success') => void>('showToast')
|
||||
|
||||
const selectedTopic = ref<Topic | null>(null)
|
||||
@@ -150,10 +154,22 @@ const loadingTopicSessions = ref(false)
|
||||
const inputText = ref('')
|
||||
const messagesContainer = ref<HTMLElement | null>(null)
|
||||
|
||||
interface ReaderPanel { bookId: string; page: number; bookTitle?: string }
|
||||
const readerPanel = ref<ReaderPanel | null>(null)
|
||||
|
||||
function handleOpenSource(bookId: string, page: number) {
|
||||
const book = bookStore.books.find(b => b.id === bookId)
|
||||
readerPanel.value = { bookId, page, bookTitle: book?.title }
|
||||
}
|
||||
|
||||
onMounted(async () => {
|
||||
if (topicStore.topics.length === 0) {
|
||||
await topicStore.fetchTopics()
|
||||
}
|
||||
const freeForm = topicStore.topics.find((t) => t.id === 'free-form')
|
||||
if (freeForm) {
|
||||
await handleTopicSelect(freeForm)
|
||||
}
|
||||
})
|
||||
|
||||
watch(
|
||||
@@ -189,11 +205,6 @@ async function handleTopicSelect(topic: Topic) {
|
||||
loadingTopicSessions.value = false
|
||||
}
|
||||
|
||||
function handleBack() {
|
||||
selectedTopic.value = null
|
||||
topicSessions.value = []
|
||||
}
|
||||
|
||||
async function handleNewChat() {
|
||||
const ok = await chatStore.createSession(selectedTopic.value!.id)
|
||||
if (!ok) {
|
||||
@@ -207,9 +218,7 @@ async function handleResumeSession(session: ChatSession) {
|
||||
}
|
||||
|
||||
function handleLeaveSession() {
|
||||
// Leave without deleting — session stays in DB and will appear in "Previous Chats"
|
||||
chatStore.leaveSession()
|
||||
// Refresh the sessions list for the current topic
|
||||
if (selectedTopic.value) {
|
||||
loadingTopicSessions.value = true
|
||||
chatStore.fetchSessionsByTopic(selectedTopic.value.id).then((sessions) => {
|
||||
@@ -231,12 +240,6 @@ async function handleSend() {
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
.topic-selection {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 1.25rem;
|
||||
}
|
||||
|
||||
.section-title {
|
||||
font-size: 1.1rem;
|
||||
font-weight: 600;
|
||||
@@ -244,52 +247,6 @@ async function handleSend() {
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
.topic-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fill, minmax(220px, 1fr));
|
||||
gap: 0.75rem;
|
||||
}
|
||||
|
||||
.topic-tile {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: flex-start;
|
||||
gap: 0.25rem;
|
||||
padding: 1rem 1.1rem;
|
||||
background: white;
|
||||
border: 1px solid #e2e8f0;
|
||||
border-radius: 8px;
|
||||
cursor: pointer;
|
||||
text-align: left;
|
||||
transition: border-color 0.15s, box-shadow 0.15s;
|
||||
}
|
||||
|
||||
.topic-tile:hover {
|
||||
border-color: #3182ce;
|
||||
box-shadow: 0 2px 8px rgba(49, 130, 206, 0.15);
|
||||
}
|
||||
|
||||
.topic-tile-freeform {
|
||||
border-style: dashed;
|
||||
border-color: #a0aec0;
|
||||
}
|
||||
|
||||
.topic-tile-freeform:hover {
|
||||
border-color: #718096;
|
||||
box-shadow: 0 2px 8px rgba(0, 0, 0, 0.08);
|
||||
}
|
||||
|
||||
.topic-tile-name {
|
||||
font-size: 0.9rem;
|
||||
font-weight: 600;
|
||||
color: #2d3748;
|
||||
}
|
||||
|
||||
.topic-tile-hint {
|
||||
font-size: 0.78rem;
|
||||
color: #a0aec0;
|
||||
}
|
||||
|
||||
.session-setup {
|
||||
max-width: 540px;
|
||||
}
|
||||
@@ -381,6 +338,29 @@ async function handleSend() {
|
||||
min-height: 500px;
|
||||
}
|
||||
|
||||
.chat-reader-split {
|
||||
display: flex;
|
||||
flex: 1;
|
||||
min-height: 0;
|
||||
gap: 0;
|
||||
}
|
||||
|
||||
.chat-column {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
flex: 1;
|
||||
min-width: 0;
|
||||
gap: 1rem;
|
||||
}
|
||||
|
||||
.reader-panel {
|
||||
width: 420px;
|
||||
flex-shrink: 0;
|
||||
border-radius: 10px;
|
||||
margin-left: 1rem;
|
||||
box-shadow: -2px 0 8px rgba(0, 0, 0, 0.07);
|
||||
}
|
||||
|
||||
.session-bar {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
@@ -505,4 +485,26 @@ async function handleSend() {
|
||||
font-size: 0.875rem;
|
||||
margin-bottom: 0.75rem;
|
||||
}
|
||||
|
||||
@media (max-width: 768px) {
|
||||
.chat-layout {
|
||||
height: auto;
|
||||
min-height: unset;
|
||||
}
|
||||
|
||||
.chat-reader-split {
|
||||
flex-direction: column;
|
||||
}
|
||||
|
||||
.chat-column {
|
||||
min-height: 60vh;
|
||||
}
|
||||
|
||||
.reader-panel {
|
||||
width: 100%;
|
||||
margin-left: 0;
|
||||
margin-top: 1rem;
|
||||
box-shadow: none;
|
||||
}
|
||||
}
|
||||
</style>
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user