Zendex — RAG Chat with Inference Logging SDK
Multi-turn streaming RAG chat over 468 curated ideas with multi-provider failover across Groq Llama 3.3 70B and Gemini 2.0 Flash. Built a custom inference logging SDK for latency, token usage, errors, and PII redaction per request, plus an observability dashboard with real-time metrics and Langfuse traces.
Shared on Twitter, where Zendex gained 38K+ impressions and was seen by Noah Zender, whose ideas power the RAG knowledge base. Twitter launch post
Problem
RAG demos usually stop at retrieval and chat. Zendex treats the system like production software: every inference needs traceability, failure handling, latency visibility, and enough privacy discipline to be inspected later.
Approach
- –Built a streaming chat path over a curated idea corpus, with retrieval citations kept close to the generated answer.
- –Added model failover so the app can route between Groq Llama 3.3 70B and Gemini 2.0 Flash when a provider is slow or unavailable.
- –Created a lightweight inference logging SDK that records request metadata, latency, token usage, errors, and redacted payloads before the data reaches the dashboard.
System Diagram
Next.js chat UI
Streams assistant responses and keeps citations visible to the user.
FastAPI orchestration
Owns retrieval, provider routing, streaming, and SDK instrumentation.
Pinecone retrieval
Searches the curated idea corpus and returns chunks for grounded responses.
LLM failover
Routes generation across Groq and Gemini based on availability and task fit.
Observability layer
Stores inference logs, Langfuse traces, latency, usage, and error signals.
Key Decisions
Why separate orchestration from the UI
The FastAPI layer keeps retrieval, logging, and provider routing server-side, so the frontend only handles interaction and rendering.
Why build a custom SDK
Logging inside the inference boundary captures useful operational signals before errors or retries hide what actually happened.
Concepts Used
Outcomes
- –Live Dockerized deployment with streaming RAG over 468 curated ideas.
- –Per-request observability for latency, token usage, errors, redaction, and distributed traces.
- –A reusable backend pattern for AI apps that need more than a happy-path chat box.