An end-to-end Agentic Retrieval-Augmented Generation (RAG) system that ingests course content and generates high-quality, grounded Q&A pairs using LLMs.
This system transforms raw course materials (transcripts(txt), PDFs, etc.) into a structured validated outputs (such as Q&A) using:
- Retrieval (Qdrant + BGE-M3)
- Agentic orchestration (Coordinator + specialized agents)
- LLM-based generation (Qwen via vLLM)
- Validation and quality control
It is designed for:
- EdTech platforms
- Course builders
- Assessment generation systems
The system consists of two main flows:
Source Data → Ingestion → Chunking → Embeddings → Storage
-
Clean and preprocess transcripts
-
Chunk content with metadata
-
Generate embeddings (BGE-M3)
-
Store in:
- Qdrant (vectors + payload)
- Postgres (metadata, jobs)
Request → Coordinator → Retrieval → Context → Generation → Validation → Output
-
Coordinator Agent
- Plans execution
- Routes tasks between agents
-
Retrieval Agent
- Query rewriting
- Filtering (course/module/lesson)
- Qdrant search
-
Reranker (BGE)
- Improves relevance of retrieved chunks
-
Context Builder
- Merges and cleans chunks
- Preserves citations
-
Context Critic
- Checks if context is sufficient
- Triggers retrieval retry if needed
-
Generation Agent
- Uses Qwen (via vLLM)
- Produces structured Q&A (JSON)
-
Validation Agent
- Grounding check
- Duplicate removal
- Difficulty validation
- Repair / retry logic
-
Export Agent
- Saves to DB
- Exposes API / export formats
- ✅ Agentic orchestration (not just pipeline RAG)
- ✅ Hybrid retrieval (semantic + rerank)
- ✅ Context quality feedback loop
- ✅ Grounded Q&A with source citations
- ✅ Validation + repair cycles
- ✅ Modular and extensible architecture
- ✅ GPU-optional local development (via mocks)
| Layer | Technology |
|---|---|
| Backend | FastAPI (Python) |
| Vector DB | Qdrant |
| Metadata DB | Postgres |
| LLM | Qwen (via vLLM) |
| Embeddings | BGE-M3 |
| Reranker | BGE Reranker |
| Queue/Workers | Celery / Redis (optional) |
app/
api/ # FastAPI routes
agents/ # Coordinator + specialized agents
tools/ # Embedding, rerank, context builder
services/ # Retrieval, ingestion, question service
models/ # ORM models
schemas/ # Pydantic schemas
db/ # DB connections
workers/ # background jobs
infra/
docker/
scripts/
configs/
tests/
-
Interface-first design
-
Clear separation:
- Agents
- Tools
- Services
- Infrastructure adapters
-
Replaceable model providers (mock → GPU)
-
Async job-based execution
-
Structured JSON outputs
-
Deterministic validation where possible
Run the system locally using mock providers:
- Mock embedding provider
- Mock generation provider
- Mock reranker
This allows:
- full pipeline testing
- faster iteration
- no GPU dependency
See docs/local_development.md for the Docker Compose startup flow.
For split CPU/GPU production deployment guidance, see docs/production_deployment.md.
- Qwen via vLLM
- BGE-M3 embedding service
- BGE reranker
- FastAPI backend
- Postgres
- Qdrant
- Workers / queue
- Agents orchestration
-
User requests:
Generate 20 medium-difficulty questions from Module 3 -
Coordinator:
- Creates plan
- Calls retrieval
-
Retrieval:
- Finds relevant chunks
- Reranks results
-
Context Critic:
- If insufficient → retry retrieval
-
Generation:
- LLM generates Q&A with citations
-
Validation:
- Checks grounding, quality, duplicates
-
Export:
- Saves to Question Bank
{
"question": "What is inorder traversal?",
"options": ["...", "..."],
"answer": "...",
"explanation": "...",
"difficulty": "medium",
"source_chunk_ids": ["chunk_12", "chunk_15"]
}- Multi-agent planning enhancements
- Difficulty calibration models
- UI dashboard for question review
- Analytics on question quality
- Multi-language support
This system combines:
- RAG (retrieval + grounding)
- Agents (decision-making + loops)
- Validation (quality control)
to produce reliable, scalable, and high-quality question generation from raw educational content.
MIT (or your preferred license)
PRs and suggestions welcome.