Poseidon
Grounded search and evaluation infrastructure for a European tech team.
- Role
- System architecture, retrieval design, evaluation tooling, frontend
- System type
- AI retrieval platform
- Disciplines
- Intelligence · Engineering · Operations
- Technology
- TypeScript, Python, Postgres, pgvector, Hybrid search, LLM evaluation, React
Key challenge
Build retrieval the team could trust — which meant building the measurement before the feature, so every tuning decision had evidence behind it.
The situation
A European tech team was shipping features on top of document search that nobody trusted. Answers were sometimes right and sometimes confidently wrong, and there was no shared way to tell the difference before users did.
The problem
Every retrieval change was a vibe check. Chunking, embedding models, ranking weights — each was tuned by whoever last touched it, against whichever queries they happened to try. Improvements and regressions were indistinguishable.
The constraints
Existing traffic could not be disrupted, the corpus mixed languages and formats, and the team was small — whatever got built had to run as infrastructure, not as a research project someone babysat.
The system
Poseidon is a retrieval platform with two halves that reinforce each other: a hybrid search layer combining lexical and semantic matching with reranking, and an evaluation harness that scores every change against a curated set of real questions with known-good passages.
Architecture
Documents flow through a single ingestion pipeline into chunked, embedded passages stored in Postgres with pgvector. Queries fan out across lexical and vector indexes, merge, and rerank. The evaluation harness runs the same query path offline against golden datasets, so what gets measured is exactly what ships.
Interface
A console where a query shows its ranked passages with citations, scores, and which stage of the pipeline promoted or buried each result — alongside a panel comparing the current configuration against the last known-good run. Debugging retrieval became reading, not guessing.
Intelligence
Every answer carries its evidence. Passages are cited end-to-end from retrieval through to the response, so a wrong answer can be traced to the exact chunk, score, or ranking decision that produced it.
Execution
Designed and built the pipeline, the retrieval layer, the evaluation harness, and the console. The evaluation culture outlasted the engagement: the team now treats a failing eval run the way they treat a failing test.
Lessons
The evaluation harness was almost cut as scope. It ended up being the product — the search layer changed shape several times, but the ability to prove a change helped never did.
Outcome
The team stopped arguing about whether search was good and started improving it: regressions are caught in evaluation runs before users ever see them, and quality became a number the whole team can read.
Architectural insight
Retrieval quality is an evaluation problem before it is a model problem — the teams that measure first are the ones that improve fastest.