GitHub AI Repository Clustering

Unsupervised clustering of 2,853 GitHub AI/ML repositories — grouping them by popularity and activity using K-Means, Agglomerative Clustering, DBSCAN, and HDBSCAN.

Overview

GitHub hosts a huge, fast-moving landscape of AI/ML repositories with no built-in way to group similar projects by popularity or activity. This project treats that as an unsupervised learning problem: repositories are clustered on four numeric signals — stars, forks, open issues, and repository size — after a log transform and standardization to correct for heavy right-skew.

All four algorithms converge on the same two-cluster structure: a large group of typical, lower-popularity repositories, and a small group of high-popularity, larger projects (e.g. TensorFlow, AutoGPT, Ollama).

Results

Model	Clusters	Noise	Silhouette ↑	Davies-Bouldin ↓
DBSCAN (eps=0.7, min_samples=10)	2	52 (1.8%)	0.398	0.512
K-Means (k=2)	2	0	0.391	0.969
Agglomerative (Ward, k=2)	2	0	0.377	0.938
HDBSCAN (min_cluster_size=20)	2	1,183 (41.5%)	0.175	1.148

DBSCAN performs best — highest silhouette score, lowest Davies-Bouldin index, and only 1.8% of repositories left unclustered. K-Means is a strong, simpler runner-up. HDBSCAN underperforms here because the dataset's density changes gradually rather than in clearly separated regions.

Dataset

github_ai_repos_2026.csv — 2,853 AI-related repositories (≥50 stars) pulled from the GitHub REST API, snapshotted April 2026. 24 columns, including:

Column(s)	Notes
stars, forks, watchers, open_issues	Popularity & activity signals
size_kb	Repository size in KB
language, license, topics	Metadata
created_at, updated_at, pushed_at	Timestamps
contributors, commits_last_4w, commits_last_52w	Missing for 82–99% of rows — excluded from modeling

Clustering features: stars, forks, open_issues, size_kb — log1p-transformed, then standardized. watchers was dropped as it's near-perfectly correlated with stars.

Repo contents

File	Description
GitHub_AI_Repository.ipynb	Full analysis — EDA, preprocessing, all four clustering models, evaluation, PCA visualization. Runs top-to-bottom.
app.py	Interactive Streamlit app — upload the CSV and run/compare all four clustering algorithms in the browser.
github_ai_repos_2026.csv	The dataset.
docs/GitHub_AI_Repository_Report.docx	Full written report.
docs/GitHub_AI_Repository.pptx	Presentation slides.

Running it

pip install -r requirements.txt

Notebook

jupyter notebook GitHub_AI_Repository.ipynb

Streamlit app

streamlit run app.py

Then upload github_ai_repos_2026.csv when prompted in the browser.

Methodology

1. EDA — checked skew, correlation, and missingness across features.
2. Feature selection — stars, forks, open_issues, size_kb.
3. Transform — log1p (corrects right-skew) → StandardScaler.
4. Model selection — silhouette analysis over k = 2–10 to pick K for K-Means / Agglomerative.
5. Clustering — K-Means, Agglomerative (Ward linkage), DBSCAN, HDBSCAN.
6. Evaluation — Silhouette Score and Davies-Bouldin Index (no ground-truth labels exist, so only internal metrics apply).
7. Visualization — PCA to 2D for cluster plots (captures ~87.5% of variance).

Applications

• Repository discovery — compare any repo against its cluster peers on popularity and activity at a glance.
• Competitive / investment analysis — the high-popularity cluster surfaces the projects concentrating community attention; DBSCAN's noise points flag unusual outliers worth a closer look.
• Ecosystem monitoring — re-running the pipeline on a fresh snapshot (no code changes needed) shows repos migrating between clusters over time.
• Education — a complete, reproducible unsupervised-learning workflow on a real dataset, from data-quality checks to model comparison.

Limitations

• No ground-truth labels — evaluation relies on internal metrics only (silhouette, Davies-Bouldin), not accuracy/precision/recall.
• Only 4 numeric features are used; language, license, and topics are excluded despite being informative, since they're categorical.
• contributors and recent commit counts are missing for most rows and couldn't be used, so recent development velocity isn't reflected in the clusters.
• The two-cluster solution is coarse — finer distinctions (e.g. tutorials vs. production tools) within the "typical" cluster aren't captured.
• Snapshot-based and star-filtered (≥50 stars): results reflect the popular segment of the ecosystem as of April 2026, not the long tail or how it evolves.

Team

Yash Jain · Yash Kumar · Umang Ladha · Dikshit Suthar (Batch TECDS01 — originally an academic ML coursework project)