Skip to content

Latest commit

 

History

History
282 lines (185 loc) · 5.16 KB

README.md

File metadata and controls

282 lines (185 loc) · 5.16 KB

🧠 Agentic RAG Question Generation System

An end-to-end Agentic Retrieval-Augmented Generation (RAG) system that ingests course content and generates high-quality, grounded Q&A pairs using LLMs.


🚀 Overview

This system transforms raw course materials (transcripts(txt), PDFs, etc.) into a structured validated outputs (such as Q&A) using:

  • Retrieval (Qdrant + BGE-M3)
  • Agentic orchestration (Coordinator + specialized agents)
  • LLM-based generation (Qwen via vLLM)
  • Validation and quality control

It is designed for:

  • EdTech platforms
  • Course builders
  • Assessment generation systems

🏗️ Architecture

The system consists of two main flows:

1. Offline Indexing Pipeline

Source Data → Ingestion → Chunking → Embeddings → Storage
  • Clean and preprocess transcripts

  • Chunk content with metadata

  • Generate embeddings (BGE-M3)

  • Store in:

    • Qdrant (vectors + payload)
    • Postgres (metadata, jobs)

2. Online Agentic Generation Pipeline

Request → Coordinator → Retrieval → Context → Generation → Validation → Output

Key Components

  • Coordinator Agent

    • Plans execution
    • Routes tasks between agents
  • Retrieval Agent

    • Query rewriting
    • Filtering (course/module/lesson)
    • Qdrant search
  • Reranker (BGE)

    • Improves relevance of retrieved chunks
  • Context Builder

    • Merges and cleans chunks
    • Preserves citations
  • Context Critic

    • Checks if context is sufficient
    • Triggers retrieval retry if needed
  • Generation Agent

    • Uses Qwen (via vLLM)
    • Produces structured Q&A (JSON)
  • Validation Agent

    • Grounding check
    • Duplicate removal
    • Difficulty validation
    • Repair / retry logic
  • Export Agent

    • Saves to DB
    • Exposes API / export formats

🧠 Key Features

  • ✅ Agentic orchestration (not just pipeline RAG)
  • ✅ Hybrid retrieval (semantic + rerank)
  • ✅ Context quality feedback loop
  • ✅ Grounded Q&A with source citations
  • ✅ Validation + repair cycles
  • ✅ Modular and extensible architecture
  • ✅ GPU-optional local development (via mocks)

🧱 Tech Stack

Layer Technology
Backend FastAPI (Python)
Vector DB Qdrant
Metadata DB Postgres
LLM Qwen (via vLLM)
Embeddings BGE-M3
Reranker BGE Reranker
Queue/Workers Celery / Redis (optional)

🧩 Project Structure

app/
  api/                # FastAPI routes
  agents/             # Coordinator + specialized agents
  tools/              # Embedding, rerank, context builder
  services/           # Retrieval, ingestion, question service
  models/             # ORM models
  schemas/            # Pydantic schemas
  db/                 # DB connections
  workers/            # background jobs

infra/
  docker/
  scripts/

configs/
tests/

🔌 System Design Principles

  • Interface-first design

  • Clear separation:

    • Agents
    • Tools
    • Services
    • Infrastructure adapters
  • Replaceable model providers (mock → GPU)

  • Async job-based execution

  • Structured JSON outputs

  • Deterministic validation where possible


💻 Local Development (No GPU Required)

Run the system locally using mock providers:

  • Mock embedding provider
  • Mock generation provider
  • Mock reranker

This allows:

  • full pipeline testing
  • faster iteration
  • no GPU dependency

See docs/local_development.md for the Docker Compose startup flow.

For split CPU/GPU production deployment guidance, see docs/production_deployment.md.


⚡ Production Deployment

Recommended Setup

GPU Instance (e.g. GH200)

  • Qwen via vLLM
  • BGE-M3 embedding service
  • BGE reranker

CPU Instance

  • FastAPI backend
  • Postgres
  • Qdrant
  • Workers / queue
  • Agents orchestration

🔄 Workflow Example

  1. User requests:

    Generate 20 medium-difficulty questions from Module 3
    
  2. Coordinator:

    • Creates plan
    • Calls retrieval
  3. Retrieval:

    • Finds relevant chunks
    • Reranks results
  4. Context Critic:

    • If insufficient → retry retrieval
  5. Generation:

    • LLM generates Q&A with citations
  6. Validation:

    • Checks grounding, quality, duplicates
  7. Export:

    • Saves to Question Bank

📦 Output Format (Example)

{
  "question": "What is inorder traversal?",
  "options": ["...", "..."],
  "answer": "...",
  "explanation": "...",
  "difficulty": "medium",
  "source_chunk_ids": ["chunk_12", "chunk_15"]
}

📈 Future Improvements

  • Multi-agent planning enhancements
  • Difficulty calibration models
  • UI dashboard for question review
  • Analytics on question quality
  • Multi-language support

🧠 Summary

This system combines:

  • RAG (retrieval + grounding)
  • Agents (decision-making + loops)
  • Validation (quality control)

to produce reliable, scalable, and high-quality question generation from raw educational content.


📄 License

MIT (or your preferred license)


🤝 Contributing

PRs and suggestions welcome.