roman-agent

A fine-tuned personal AI agent — Llama 3.1 8B + QLoRA — that answers questions about Roman (@roman-bytes). Deployable as a FastAPI server with an embeddable vanilla-JS chat widget for any website.

---

How it works

1. You fill in data/knowledge_base.md with real info about yourself
2. Run the helper script to auto-generate Q&A pairs, or write them manually in data/training_data.jsonl
3. Fine-tune Llama 3.1 8B with QLoRA (uses Unsloth for speed + memory efficiency)
4. Serve the model via FastAPI
5. Drop one <script> tag on your website to get a floating chat widget

---

Quick start

1. Fill in your data

Edit data/knowledge_base.md — replace every [PLACEHOLDER] with real info about yourself.

Then edit data/training_data.jsonl — fill in your actual answers for the [YOUR ANSWER] placeholders.

Optionally auto-generate additional pairs from the knowledge base:

python scripts/generate_training_data.py

2. Install dependencies

Requires Python 3.10+ and a CUDA GPU (16GB+ VRAM recommended for training).

python -m venv venv
source venv/bin/activate   # Windows: venv\Scripts\activate
pip install -r requirements.txt

3. Fine-tune the model

python training/train.py

Training config lives in training/config.yaml. With 30 examples and the default config, this takes ~5–15 minutes on an A100.

The fine-tuned model is saved to model/merged/.

4. Start the API server

uvicorn api.main:app --host 0.0.0.0 --port 8000

The server auto-detects whether a fine-tuned model exists. If not, it falls back to base Llama 3.1 with your knowledge base injected as a system prompt (useful for testing before training).

Endpoints

Method	Path	Description
POST	/chat	Send messages, get a reply
GET	/health	Check model load status

Example:

curl -X POST http://localhost:8000/chat \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "What have you built?"}]}'

5. Embed the widget on your website

<script>
  window.ROMAN_AGENT_URL = "https://your-api-host.com";
</script>
<script src="roman-agent.js"></script>

That's it. A floating chat button appears in the bottom-right corner. Copy widget/roman-agent.js to your static assets.

---

Optional: export to GGUF (CPU inference)

Run the model locally without a GPU using llama.cpp or Ollama:

python scripts/export_model.py --quantization q4_k_m
# Output: model/roman-agent.gguf

Then load in Ollama:

ollama create roman-agent -f Modelfile
ollama run roman-agent

---

Project structure

roman-agent/
├── data/
│   ├── training_data.jsonl       # Q&A training pairs
│   └── knowledge_base.md         # Raw info about Roman
├── training/
│   ├── train.py                  # QLoRA fine-tuning (Unsloth)
│   └── config.yaml               # Hyperparameters
├── api/
│   ├── main.py                   # FastAPI server
│   ├── model.py                  # Model loading + inference
│   └── schemas.py                # Pydantic schemas
├── widget/
│   └── roman-agent.js            # Embeddable chat widget
├── model/                        # (gitignored) fine-tuned weights
├── scripts/
│   ├── generate_training_data.py
│   └── export_model.py
├── requirements.txt
└── .gitignore

---

Tips for better results

• More data = better agent. 30 examples is a starting point. Aim for 100+ pairs that cover your full range of experience.
• Be specific in responses. Vague answers like "I've built many projects" produce vague model outputs. Include names, dates, stack details.
• Test without training first. The API falls back to base Llama with your knowledge_base.md as context — this can be surprisingly good and helps you validate your data before spending time training.
• Use generate_training_data.py after fully filling in knowledge_base.md — it can generate 20–40 additional pairs automatically.

---

Hardware requirements

Task	Minimum	Recommended
Training	16GB VRAM (RTX 3090 / A10)	24–40GB (A100)
Inference (GPU)	8GB VRAM	16GB
Inference (CPU via GGUF)	8GB RAM	16GB RAM

Training uses 4-bit quantization via Unsloth, so memory requirements are much lower than standard fine-tuning.