Indexing pipeline

fetch HTML ──▶ extract text ──▶ chunk ──▶ scan ──▶ embed ──▶ store
 (httpx)      (trafilatura)              (regex)  (fastembed) (SQLite + Chroma)

Save time

When you save a page the daemon fetches it, extracts the main text, chunks it, and runs the injection scan. If the page is an unrendered JavaScript shell, or any chunk is risky, it returns a scan warning and stops — nothing is stored. Otherwise it creates the bookmark and hands the already-fetched HTML and text to a background job so the request returns immediately.

The extension can also pass the page’s HTML it captured in the browser. That matters for sites that block server-side fetches or render entirely with JavaScript — the daemon would get nothing on its own.

Background job

The job re-uses the HTML and text from the save when it has them, then:

1. Chunks the text.
2. Scans each chunk and records its risk score + signals as chunk metadata.
3. Embeds the chunks.
4. Deletes the bookmark’s old vectors, writes the new ones to Chroma, writes the chunk rows to SQLite.
5. Marks the bookmark done (or error with a message if anything failed).

Re-index

POST /bookmarks/{id}/reindex re-runs the job. A re-index can only fetch server-side, so on a bot-blocking or JS-rendered site it falls back to the HTML kept from the original save. A failed re-fetch never overwrites a good earlier capture.