How it works
The ingestion pipeline, supported inputs, collections, and the embedding model.
Enjab RAG processes knowledge, it does not just store it. Every ingest runs the same pipeline, so search results come back with a clean title, summary, tags, and structured fields no matter what you sent in.
The pipeline
Receive & keep the original
The raw file is stored in object storage (Cloudflare R2) so the original is always recoverable. Text and links skip this step.
Extract the text
The content is turned into text based on its type (see the table below), images via OCR, audio via transcription.
Process with a model
One model pass writes a title, a summary, tags, the language, and a flexible metadata object (dates, departments, names, document type, whatever it can confidently extract). Never invents facts.
Chunk (structure-aware)
Splitting respects what the document is, not a blind character count. Tables and spreadsheets keep their header row on every chunk (so each chunk is self-describing); prose keeps its section heading attached; and splitting cascades through real boundaries (headings → paragraphs → sentences → lines). Every chunk is bounded so it always fits the embedding model.
Contextualize & embed
Each chunk gets Contextual Retrieval: a fast model writes a one-line context situating the chunk in its document (which document, section, and entity it concerns), and that context is embedded together with the chunk, so short or ambiguous chunks (a single spreadsheet row, a bare figure) still retrieve correctly. The original chunk is stored as the returned content; the contextualized text is what gets embedded with OpenAI text-embedding-3-large (3072 dimensions).
Index
Chunks and their vectors are stored in Postgres with pgvector (halfvec, cosine, HNSW index). search ranks chunks by cosine similarity within a collection.
Supported inputs
| Input | How it is read |
|---|---|
| Text layer extracted; scanned / image-only PDFs fall back to model OCR. | |
Word (.docx) | Text extracted (mammoth). Legacy binary .doc is not supported, save as .docx or PDF. |
Spreadsheet (.xlsx, .xls, .csv) | Parsed per sheet into rows the model can read. |
Image (png, jpg, webp, …) | OCR'd with a vision model. |
Audio (mp3, wav, m4a, …) | Transcribed with Whisper. |
| Text | Indexed directly. |
| Link | The page is fetched and reduced to readable text. |
Titles & metadata
You never have to name a document. The model writes the best title from the content (you can override it with an explicit title on the API, but the default is automatic). Summaries, tags, and the structured metadata object come from the same pass and are returned on every search result.
Collections
Knowledge lives in collections. Today there is a single global collection (all of the centre's operational knowledge). The collection parameter exists on every endpoint so per-centre or per-sensitivity scoping can be added later without breaking anything, omit it and you get global.
What goes in
Enjab RAG holds operational knowledge, how the clinics run: hours, policies, procedures, FAQs, handbooks, directories. It is not for patient records or clinical data.
Storage & model summary
| Concern | Choice |
|---|---|
| Chunking | Structure-aware (table-header + heading propagation) + Contextual Retrieval |
| Embeddings | OpenAI text-embedding-3-large, 3072-dim |
| Vector store | Supabase Postgres + pgvector (halfvec, HNSW, cosine) |
| Originals | Cloudflare R2 (object storage) |
| Processing | OpenAI (extraction, OCR, transcription, summarize/tag, per-chunk context) |