Harness the power of Local LLMs + Intelligent Web Scraping — All on Your Machine

---

🌟 Project Overview

Nexus AI is a cutting-edge, privacy-first AI assistant that combines the power of locally-hosted Large Language Models (via Ollama) with intelligent web navigation using Playwright automation.

Unlike cloud-based AI assistants, Nexus AI keeps your data on your machine while still providing real-time web intelligence when needed.

🎯 Key Highlights

• 🧠 Local AI Power — Mistral LLM via Ollama
• 🌐 Smart Web Search — DuckDuckGo + Playwright
• 🛒 Product Scraper — Table & Gallery Views
• 🔒 Privacy First — 100% Local Processing
• ⚡ Lightning Fast — 3-5 second responses
• 🎨 Modern UI — ChatGPT-style interface

---

✨ Features

🧠 Local LLM Integration ✅ Ollama Mistral for instant offline answers ✅ 3-5 second response times ✅ Zero data leaves your machine ✅ Full conversation context support	🌐 Intelligent Web Search ✅ CAPTCHA-free DuckDuckGo integration ✅ Playwright-powered browser automation ✅ Smart result prioritization & scoring ✅ Real-time web data aggregation
🛒 Product Scraper Module ✅ Table View with sortable columns ✅ Gallery View with product images ✅ CSV export functionality ✅ Batch processing (configurable)	💡 Smart Prioritization Engine 📅 Exact Date: +3000 score boost 📰 Today/Recent: +1000 score boost 📚 Wikipedia Range: +5000 supreme priority 🧊 Old Content: Penalty system applied

---

🎨 Tech Stack

Technology	Purpose	Version
Python	Backend & AI Logic	3.10+
Flask	REST API Server	3.x
Playwright	Browser Automation	1.48
BeautifulSoup	HTML Parsing	4.12
Ollama	Local LLM Runtime	Latest
HTML5	Frontend UI	5
CSS3	Styling	3
JavaScript	Frontend Logic	ES6+

---

🏗️ Architecture

flowchart TB
    subgraph USER["👤 User Interface"]
        A[Web Browser]
    end

    subgraph FRONT["🎨 Frontend Layer"]
        B[index.html — ChatGPT-Style UI]
        C[script.js — Frontend Logic]
        D[styles.css — Modern Styling]
    end

    subgraph BACK["⚙️ Backend Layer"]
        E[Flask API Server — app.py]
        F[Agent Core — agent_step3.py]
        G[Web Scraper — web_scraper.py]
    end

    subgraph AI["🧠 AI Layer"]
        H[Ollama Runtime]
        I[Mistral LLM]
    end

    subgraph WEB["🌐 Web Layer"]
        J[Playwright Browser]
        K[DuckDuckGo Search]
        L[Website Scraping]
    end

    subgraph STORE["💾 Storage"]
        M[agent_state/]
        N[JSON Batch Files]
    end

    A --> B
    B --> C
    C --> E
    E --> F
    F --> H
    H --> I
    F --> G
    G --> J
    J --> K
    J --> L
    F --> M
    M --> N

    style USER fill:#001a1a,stroke:#00f5ff,color:#00f5ff
    style FRONT fill:#001a0d,stroke:#00ff88,color:#00ff88
    style BACK fill:#1a1500,stroke:#ffd700,color:#ffd700
    style AI fill:#0f0020,stroke:#a855f7,color:#a855f7
    style WEB fill:#001a1a,stroke:#00f5ff,color:#00f5ff
    style STORE fill:#1a0010,stroke:#ff006e,color:#ff006e

Loading

🔄 Request Flow

sequenceDiagram
    autonumber
    participant U as 👤 User
    participant F as 🎨 Frontend
    participant B as ⚙️ Backend
    participant L as 🧠 Local LLM
    participant W as 🌐 Web Search

    U->>F: Ask Question
    F->>B: POST /ask
    B->>L: Query Mistral

    alt Confident Answer
        L-->>B: Return Local Answer
    else Needs Web Data
        B->>W: Playwright Search
        W-->>B: Search Results
        B->>B: Apply Scoring Algorithm
    end

    B-->>F: JSON Response
    F-->>U: Render Answer + Sources

Loading

---

🚀 Quick Start

⚡ One-Click Start (Windows)

Script	Description
🟢 START_NEXUS_FIXED.bat	Full Setup — Starts Ollama + Backend + Opens Browser
🔵 START_BACKEND_FIXED.bat	Backend Only — Manual frontend opening
🟣 START_NEXUS_AI.ps1	PowerShell — Alternative starter

# Simply double-click any .bat file to start!

🛠️ Manual Installation

📋 Click to expand Step-by-Step Setup Guide

Prerequisites

# 1. Install Ollama — https://ollama.ai

# 2. Pull the Mistral model
ollama pull mistral

# 3. Clone the repository
git clone https://github.com/AshmitThakur23/Local-AI-Web-Navigator.git
cd Local-AI-Web-Navigator

# 4. Install Python dependencies
pip install -r requirements.txt

# 5. Install Playwright browsers
playwright install chrome

Running the Application

# Terminal 1: Start Ollama
ollama serve

# Terminal 2: Start Backend
python backend/app.py

# Terminal 3: Serve Frontend (Optional)
cd frontend
python -m http.server 8000
# Visit: http://localhost:8000

Health Check

curl http://localhost:5000/health
# Response: {"status":"ok","message":"Nexus AI Backend is running"}

---

📁 Project Structure

📦 Local-AI-Web-Navigator/
│
├── 🎨 frontend/
│   ├── index.html              # Main ChatGPT-style interface
│   ├── script.js               # Frontend logic & API calls
│   ├── styles.css              # Modern dark theme styling
│   ├── setup.html              # Setup wizard page
│   └── _redirects              # Netlify routing config
│
├── ⚙️ backend/
│   ├── app.py                  # Flask REST API server
│   ├── agent_step3.py          # Core AI agent & scoring logic
│   ├── web_scraper.py          # BeautifulSoup web scraper
│   ├── test_mistral_speed.py   # Performance testing
│   └── agent_state/            # Saved scrape batches
│
├── 🚀 Starters/
│   ├── START_NEXUS_FIXED.bat       # One-click full start
│   ├── START_BACKEND_FIXED.bat     # Backend only
│   ├── START_NEXUS_AI.ps1          # PowerShell starter
│   └── start_all_services.py       # Python launcher
│
├── 📖 Documentation/
│   ├── DEPLOYMENT_GUIDE.md
│   ├── USAGE_GUIDE.md
│   ├── PRODUCT_SCRAPER_GUIDE.md
│   ├── NVIDIA_GPU_CONFIG.md
│   └── ... (more guides)
│
├── requirements.txt            # Python dependencies
├── netlify.toml                # Deployment config
└── README.md                   # You are here! 📍

---

🛒 Product Scraper

🎯 Scrape Products Like a Pro!

The Product Scraper module allows you to extract structured product data from the web with support for multiple view modes.

📊 View Modes

📋 Table View http://localhost:5000/table_view.html ✅ Sortable columns ✅ CSV export ✅ Clean data layout

🖼️ Gallery View http://localhost:5000/table_view.html ✅ 4-5 images per product ✅ Visual browsing ✅ Quick overview

📦 JSON Cards http://localhost:5000/scraper.html ✅ Raw JSON data ✅ Developer friendly ✅ API integration

🔄 Scraper Flow

flowchart LR
    A[🔍 Search Query] --> B[Flask API]
    B --> C[DuckDuckGo]
    C --> D[Web Scraper]
    D --> E[Batch Processing]
    E --> F[JSON Files]
    F --> G{View Mode}
    G --> H[📋 Table]
    G --> I[🖼️ Gallery]
    G --> J[📦 Cards]

    style A fill:#001a0d,stroke:#00ff88,color:#00ff88
    style F fill:#1a1500,stroke:#ffd700,color:#ffd700
    style G fill:#0f0020,stroke:#a855f7,color:#a855f7
    style H fill:#001a1a,stroke:#00f5ff,color:#00f5ff
    style I fill:#001a1a,stroke:#00f5ff,color:#00f5ff
    style J fill:#001a1a,stroke:#00f5ff,color:#00f5ff

Loading

💡 Example Usage

// API Call
POST /scrape_products
{
    "query": "top 20 laptops under 60000",
    "limit": 20,
    "batch_size": 5
}

// Response
{
    "success": true,
    "total_items": 20,
    "total_batches": 4,
    "batches": [...]
}

---

🔌 API Reference

📡 Available Endpoints

Method	Endpoint	Description	Request Body
GET	/health	Health check	—
POST	/ask	Ask AI question	{ "question": "..." }
POST	/scrape_products	Scrape products	{ "query": "...", "limit": 20 }
POST	/shutdown	Graceful shutdown	—

📝 Response Format

{
    "answer": "Your detailed answer here...",
    "method": "local | web",
    "sources": [
        {
            "title": "Source Title",
            "url": "https://...",
            "score": 5000
        }
    ]
}

---

⚙️ Configuration

🔧 Click to expand Customization Options

Frontend Configuration

// frontend/script.js
const API_URL = 'http://localhost:5000';  // Change if using different port

Backend Configuration

# backend/agent_step3.py
OLLAMA_API = "http://127.0.0.1:11434/api/generate"
MODEL_NAME = "mistral"
channel = "chrome"      # or "msedge"
headless = False        # True for headless mode

Port Configuration

# backend/app.py
app.run(host="0.0.0.0", port=5001)  # Change port here

---

📈 Performance

Metric	Value
⚡ Local LLM Response	3-5 seconds
🌐 Web Search Response	10-20 seconds
💾 Memory Usage	~500MB (with browser)
🔄 Concurrent Requests	Supported
🖥️ GPU Acceleration	NVIDIA CUDA (optional)

xychart-beta
    title "Response Time by Query Type (seconds)"
    x-axis ["Local LLM", "Web Search", "Product Scraper", "Health Check"]
    y-axis "Seconds" 0 --> 25
    bar [4, 15, 20, 1]

Loading

---

🧠 Scoring Algorithm

The intelligent scoring system prioritizes relevant and recent information:

Priority	Score Boost	Trigger
⭐⭐⭐⭐⭐	+5000	Wikipedia + Historical range (e.g., "2003–2006")
⭐⭐⭐⭐	+3000	Exact date match (e.g., "Oct 10, 2025")
⭐⭐⭐	+1000	Today / Recent information
⭐⭐	+500	Wikipedia general articles
⭐	0	Standard results
🚫	-200 to -800	Old / outdated content penalty

graph LR
    Q([Query Input]) --> D{Date Signal?}
    D -->|Exact date found| E1[+3000 Boost]
    D -->|Today or recent| E2[+1000 Boost]
    D -->|Historical range| E3[+5000 Boost]
    D -->|Wikipedia match| E4[+500 Boost]
    D -->|Old content| E5[-200 to -800]
    E1 & E2 & E3 & E4 & E5 --> F[Final Score]
    F --> G([Top Ranked Results])

    style Q fill:#001a0d,stroke:#00ff88,color:#00ff88
    style D fill:#1a1500,stroke:#ffd700,color:#ffd700
    style E3 fill:#001a1a,stroke:#00f5ff,color:#00f5ff
    style E1 fill:#001a1a,stroke:#00f5ff,color:#00f5ff
    style E5 fill:#1a0000,stroke:#ff006e,color:#ff006e
    style G fill:#001a0d,stroke:#00ff88,color:#00ff88

Loading

---

🧪 Usage Examples

┌─────────────────────────────────────────────────────────────────┐
│  🌤️ Today's Information                                         │
│  ❓ "What's the weather today in Hyderabad?"                    │
│  ✅ Boosted by +1000 (recent/today priority)                    │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│  📅 Exact Date Query                                            │
│  ❓ "What's happening on October 10, 2025?"                     │
│  ✅ +3000 exact date match priority                             │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│  📚 Historical Range                                            │
│  ❓ "What happened between 2003 and 2006?"                      │
│  ✅ Wikipedia prioritized with +5000                            │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│  🧠 General Knowledge                                           │
│  ❓ "How does photosynthesis work?"                             │
│  ✅ Mistral answers locally — fast & private                    │
└─────────────────────────────────────────────────────────────────┘

---

🔧 Troubleshooting

🐛 Click to expand Common Issues and Solutions

Backend Won't Start

# Check if port 5000 is free
netstat -ano | findstr :5000

# Reinstall dependencies
pip install -r requirements.txt

# Start Ollama first
ollama serve

Frontend Can't Connect

// Verify API_URL in frontend/script.js
const API_URL = 'http://localhost:5000';
// Make sure backend window shows:
// "Running on http://localhost:5000"

Chrome Won't Open

# Install Playwright browsers
playwright install chrome
# Or use Edge — set channel="msedge" in agent_step3.py

System Check

python test_setup.py

---

🤝 Contributing

flowchart LR
    A[🍴 Fork Repo] --> B[🌿 Create Branch]
    B --> C[💻 Make Changes]
    C --> D[✅ Test]
    D --> E[📤 Pull Request]
    E --> F[🎉 Merged!]

    style A fill:#001a0d,stroke:#00ff88,color:#00ff88
    style C fill:#1a1500,stroke:#ffd700,color:#ffd700
    style E fill:#001a1a,stroke:#00f5ff,color:#00f5ff
    style F fill:#001a0d,stroke:#00ff88,color:#00ff88

Loading

1. Fork the repository
2. Create a feature branch — git checkout -b feature/amazing-feature
3. Commit your changes — git commit -m 'Add amazing feature'
4. Push to the branch — git push origin feature/amazing-feature
5. Open a Pull Request

---

📜 License & Notice

Copyright © 2024 Ashmit Thakur. All rights reserved.

This project — Nexus AI: Local AI Web Navigator — was created by:
  Ashmit Thakur
  GitHub : https://github.com/AshmitThakur23
  Repo   : https://github.com/AshmitThakur23/Local-AI-Web-Navigator

Original Creation Date: October 2024

Licensed under the MIT License — see LICENSE file for details.

If you fork, clone, or use any part of this code, you MUST:
  1. Give credit to the original author
  2. Include this NOTICE
  3. Include the LICENSE file
  4. Not claim this as your own original work

Violation of these terms may result in DMCA takedown requests.

---

👨‍💻 Authors

Ashmit Thakur

Abhinendra

Sriram

Prajitha Parani

Ansh Gupta

---

⭐ Show Your Support

If you found this project helpful, please give it a ⭐!

---

🚀 Built with ❤️ for Intelligent, Privacy-Focused AI Assistance