Engineers and support reps were drowning in scattered docs, tickets, and runbooks. We built a RAG layer that answers in plain language with cited sources, cutting resolution time by 60%.
Knowledge lived in five places, Confluence, GitHub, Zendesk, Slack threads, and tribal memory. Finding the one paragraph that solved a recurring incident meant pinging three people and waiting.
Keyword search alone failed: it matched terms, not intent, and never told you whether the answer was current or trustworthy.
Pure vector search hallucinated edge-case matches; pure keyword search missed paraphrased questions. Fusing both, then reranking, gave us recall without noise.
Every generated answer is constrained to retrieved chunks and ships with citations, so users verify the source in one click. No citation, no claim.
| Component | Technology | Why |
|---|---|---|
| Retrieval store | OpenSearch | Hybrid vector + BM25 in one engine with native ACL filtering. |
| Embeddings & LLM | OpenAI | Strong multilingual embeddings and grounded synthesis quality. |
| Orchestration | LangChain | Composable retrieval, reranking, and prompt assembly. |
| Backend | Python | Async ingestion pipelines and eval tooling. |
| Frontend | Next.js | Server-streamed answers with citation UI. |
Vector kNN and BM25 run in parallel and fuse via reciprocal rank fusion for high recall.
A second-stage model reorders candidates so the most relevant chunk lands in context.
Responses include inline source links and refuse to answer when context is missing.
Per-user access controls filter retrieval so private docs never leak into answers.
Next.js streams tokens with sub-second time-to-first-token for an instant feel.
Thumbs up/down signals feed a labeled eval set that tracks accuracy over time.
“No citation, no claim, every answer points back to a source you can open in one click.”
Knowledge Base AI turned five fragmented systems into a single, trustworthy answer box. Engineers self-serve runbooks instead of interrupting on-call, support reps close tickets 60% faster, and the system handles 10K+ queries a day at 85% accuracy, all grounded in citations the team actually trusts. What started as a search problem became the default first stop for any "how do we…" question across the org.