EvaluationsObservabilityReliabilityClean ArchitectureRAG
An open-source RAG system that answers questions over a large, mixed corpus (audio transcripts, PDFs, EPUBs), where a confidently wrong or unsourced answer erodes trust immediately. Every answer is grounded in cited source passages, and answer quality is measured, not guessed.
- The approach
- The complete pipeline: multi-format ingestion, token-based chunking, embeddings in PostgreSQL/pgvector, hybrid retrieval (semantic and keyword search fused with Reciprocal Rank Fusion, plus reranking), exposed as tools to a Pydantic-AI agent that streams answers grounded in cited source passages.
- What it proves
- An evaluation framework with LLM-as-judge scoring, a custom error taxonomy and Langfuse observability measures and iteratively improves answer quality, rather than guessing it. Every answer is grounded in cited source passages.
- Highlights
- Cited sources on every answer
- LLM-as-judge eval harness and error taxonomy
- Hybrid retrieval: RRF plus reranking
- Langfuse observability
- Stack
- LLM integrationRAG (pgvector + full-text search)Voyage AI embeddings + rerankingEvaluation harnessLLM-as-judgeContext engineeringInstructorPydantic AIOpenRouter APILangfusePythonFastAPIPostgreSQLDocker
Have a process to automate, or an AI feature to ship?
Whether it's manual work between tools that should run itself, or an AI product that needs to reach production reliably, I'll take it from audit to a robust, well-tested build, end to end or alongside your team.