🧠 Second Brain - AI-Powered Knowledge Management System

An intelligent knowledge management system that automatically processes, understands, and connects your documents using AI. Built with a production-grade agentic architecture, vector search, and knowledge graphs.

✨ Features

🤖 Intelligent Processing Pipeline

Automated Document Ingestion: Drop PDFs into the inbox folder and watch them get processed automatically
Multi-Step Agent Architecture: Coordinated agents handle extraction, summarization, tagging, and embedding
Smart Chunking: Documents are intelligently split while preserving context
AI-Powered Summarization: Get concise summaries using local LLMs via Ollama

🔍 Advanced Search & Retrieval

Semantic Search: Find documents by meaning, not just keywords
Hybrid Retrieval: Combines vector similarity and knowledge graph traversal
Context-Aware Answers: Get relevant answers with supporting evidence
Multi-Step Reasoning: AI agent performs step-by-step reasoning over your knowledge base

🕸️ Knowledge Graph

Automatic Relationship Discovery: Connects documents through shared topics and concepts
Graph-Based Exploration: Navigate your knowledge through conceptual links
Topic Clustering: Automatically categorizes content by topics, concepts, and applications

📊 AI Insights

Knowledge Analytics: Discover patterns across your entire knowledge base
Recommendation Engine: Get personalized document recommendations
Insight Generation: AI-powered insights about your learning journey

🏗️ Architecture

┌─────────────────────────────────────────────────────────────┐
│                    File Watcher (Inbox)                      │
└────────────────────────┬────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│              Agent Orchestrator (Planner)                    │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐   │
│  │  Extract │→│ Summarize│→│   Tag    │→│  Embed   │   │
│  └──────────┘  └──────────┘  └──────────┘  └──────────┘   │
│                                   │                          │
│                                   ▼                          │
│                        ┌──────────────────┐                 │
│                        │  Graph Builder   │                 │
│                        └──────────────────┘                 │
└─────────────────────────────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│                    Storage Layer                             │
│  ┌─────────────┐    ┌─────────────┐    ┌──────────────┐   │
│  │  ChromaDB   │    │   Neo4j     │    │ File System  │   │
│  │ (Vectors)   │    │  (Graph)    │    │ (Processed)  │   │
│  └─────────────┘    └─────────────┘    └──────────────┘   │
└─────────────────────────────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│                  Query Agent                                 │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐   │
│  │ Retrieve │→│   Tag    │→│  Graph   │→│  Reason  │   │
│  └──────────┘  └──────────┘  └──────────┘  └──────────┘   │
└─────────────────────────────────────────────────────────────┘

🔧 Prerequisites

Required Software

Python 3.8+
Neo4j 5.0+ - Graph database for knowledge relationships
Ollama - Local LLM inference engine

System Requirements

8GB RAM minimum (16GB recommended)
10GB free disk space for models and data
macOS, Linux, or Windows (WSL recommended)

📦 Installation

1. Clone the Repository

git clone https://github.com/yourusername/second-brain.git
cd second-brain

2. Set Up Python Environment

# Create virtual environment
python -m venv .venv

# Activate virtual environment
# On macOS/Linux:
source .venv/bin/activate
# On Windows:
.venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

3. Install Neo4j

# macOS (using Homebrew)
brew install neo4j
neo4j start

# Linux/Windows - Download from https://neo4j.com/download/

4. Install Ollama

# macOS/Linux
curl -fsSL https://ollama.com/install.sh | sh

# Pull the required model
ollama pull llama3.2:3b

5. Configure Environment Variables

# Create .env file
cp .env.example .env

# Edit .env with your settings

.env file:

# Ollama Model
MODEL_NAME=llama3.2:3b

# Neo4j Configuration
URI=bolt://localhost:7687
USERNAME=neo4j
PASSWORD=your_secure_password_here

6. Set Neo4j Password

# First time setup - reset password
neo4j-admin set-initial-password your_secure_password_here

🚀 Usage

Start the Processing Pipeline

# Activate virtual environment
source .venv/bin/activate

# Start the pipeline
python run_pipeline.py

The pipeline will:

Monitor data/inbox/ for new PDF files
Automatically process them through the AI pipeline
Store embeddings, summaries, and graph relationships
Move processed files to data/processed/

Query Your Knowledge Base

# Interactive query interface
python -m src.scripts.query

Example queries:

Ask anything: What is machine learning?
Ask anything: Explain neural networks
Ask anything: What concepts relate to deep learning?

View AI Insights

# Generate insights about your knowledge base
python -m src.scripts.insights

Reset/Clean Database

# Clear all data and start fresh
python -m src.scripts.reset

📁 Project Structure

second-brain/
├── data/
│   ├── inbox/              # Drop PDF files here for processing
│   ├── processed/          # Processed files and metadata
│   ├── archive/            # Archived files
│   └── db/                 # ChromaDB vector database
│
├── src/
│   ├── agent/              # Agentic orchestration
│   │   ├── executor.py     # Step execution engine
│   │   ├── orchestrator.py # Main orchestration loop
│   │   ├── planner.py      # Planning logic
│   │   └── query/          # Query agent components
│   │
│   ├── processing/         # Document processing
│   │   ├── chunker.py      # Text chunking logic
│   │   ├── embeddings.py   # Vector embeddings
│   │   ├── summarizer.py   # AI summarization
│   │   └── tagger.py       # Topic/concept extraction
│   │
│   ├── graph/              # Knowledge graph
│   │   ├── graph_builder.py
│   │   ├── graph_db.py     # Neo4j interface
│   │   └── graph_query.py  # Graph queries
│   │
│   ├── memory/             # Vector storage
│   │   └── vector_store.py # ChromaDB interface
│   │
│   ├── retrieval/          # Search & retrieval
│   │   ├── search.py       # Semantic search
│   │   ├── ranker.py       # Result ranking
│   │   └── qa.py           # Question answering
│   │
│   ├── intelligence/       # AI insights
│   │   ├── ai_insights.py
│   │   ├── reasoning.py
│   │   └── recommender.py
│   │
│   ├── tools/              # Reusable tools
│   │   ├── base_tool.py    # Tool interface
│   │   ├── extract.py      # PDF extraction
│   │   ├── summarize.py    # Summarization tool
│   │   ├── tag.py          # Tagging tool
│   │   ├── embed.py        # Embedding tool
│   │   ├── graph.py        # Graph tool
│   │   └── reason.py       # Reasoning tool
│   │
│   ├── scripts/            # Utility scripts
│   │   ├── query.py        # Interactive query
│   │   ├── insights.py     # Insights dashboard
│   │   └── reset.py        # Database reset
│   │
│   ├── ingestion/          # File ingestion
│   │   ├── watcher.py      # File system watcher
│   │   ├── worker.py       # Processing worker
│   │   └── queue_manager.py
│   │
│   ├── state/              # State management
│   │   └── processing_state.py
│   │
│   ├── utils/              # Utilities
│   │   └── hashing.py
│   │
│   └── config.py           # Configuration
│
├── logs/                   # Application logs
├── requirements.txt        # Python dependencies
├── run_pipeline.py         # Main entry point
├── .env                    # Environment variables
├── .gitignore
└── README.md

🛠️ Development

Code Quality Standards

This project follows production-level standards:

Type hints: All functions use Python type hints
Logging: Comprehensive logging with different levels
Error handling: Graceful error handling with detailed messages
Modularity: Clean separation of concerns
Documentation: Inline comments and docstrings

Adding New Tools

Tools follow the BaseTool interface:

from src.tools.base_tool import BaseTool

class MyCustomTool(BaseTool):
    name = "my_tool"
    description = "What this tool does"
    
    def run(self, *args, **kwargs):
        # Your logic here
        return {
            "result": "data",
            "metadata": {}
        }

The execute() method automatically wraps responses with:

Status tracking
Error handling
Execution time
Metadata

Running Tests

# Run all tests
pytest

# Run with coverage
pytest --cov=src

🐛 Troubleshooting

Common Issues

1. Neo4j Authentication Error

Error: Neo.ClientError.Security.Unauthorized

Solution:

# Reset Neo4j password
neo4j-admin set-initial-password your_new_password

# Update .env file with new password

2. Ollama Model Not Found

Error: model 'llama3.2:3b' not found

Solution:

ollama pull llama3.2:3b

3. ChromaDB Permission Error

Error: Permission denied: data/db/chroma.sqlite3

Solution:

# Fix permissions
chmod -R 755 data/db/

4. PDF Extraction Fails

Error: No text extracted from file

Solution:

Ensure PDF is not scanned/image-based
Try OCR preprocessing if needed
Check PDF file is not corrupted

Logs

All logs are stored in logs/pipeline.log:

# View logs in real-time
tail -f logs/pipeline.log

# Search for errors
grep ERROR logs/pipeline.log

📊 Performance Tips

Batch Processing: Process multiple files by dropping them all at once
GPU Acceleration: Use Ollama with GPU for faster inference
Chunk Size: Adjust chunk size in src/processing/chunker.py based on your use case
Vector Index: ChromaDB automatically optimizes indexes
Neo4j Tuning: Increase heap size for large knowledge bases

🤝 Contributing

Contributions are welcome! Please follow these steps:

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

Development Guidelines

Follow PEP 8 style guide
Add tests for new features
Update documentation
Keep commits atomic and well-described

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

Ollama - Local LLM inference
Neo4j - Graph database
ChromaDB - Vector database
Sentence Transformers - Embedding models
Rich - Beautiful terminal output

🗺️ Roadmap

📧 Support

For questions and support:

Open an issue on GitHub
Check existing issues for solutions
Review the troubleshooting section

Built with ❤️ using AI and Python

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

🧠 Second Brain - AI-Powered Knowledge Management System

✨ Features

🤖 Intelligent Processing Pipeline

🔍 Advanced Search & Retrieval

🕸️ Knowledge Graph

📊 AI Insights

🏗️ Architecture

🔧 Prerequisites

Required Software

System Requirements

📦 Installation

1. Clone the Repository

2. Set Up Python Environment

3. Install Neo4j

4. Install Ollama

5. Configure Environment Variables

6. Set Neo4j Password

🚀 Usage

Start the Processing Pipeline

Query Your Knowledge Base

View AI Insights

Reset/Clean Database

📁 Project Structure

🛠️ Development

Code Quality Standards

Adding New Tools

Running Tests

🐛 Troubleshooting

Common Issues

1. Neo4j Authentication Error

2. Ollama Model Not Found

3. ChromaDB Permission Error

4. PDF Extraction Fails

Logs

📊 Performance Tips

🤝 Contributing

Development Guidelines

📝 License

🙏 Acknowledgments

🗺️ Roadmap

📧 Support