Add FastLOF algorithm#699
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d7a29af16e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| improvements = np.sum(new_kdist < old_kdist - epsilon) | ||
|
|
||
| # Early stopping if no improvements | ||
| if improvements == 0: |
There was a problem hiding this comment.
When the data is split into more than two chunks, offset is only a separation in the random permutation, not a distance ordering. If one diagonal (for example offset 1) happens not to improve any current k-distance, this break skips all later chunk pairs, even though those chunks may contain closer neighbors; training scores then depend on the random chunk order and can miss true nearest neighbors. Continue over all offsets or base convergence on a full pass instead.
Useful? React with 👍 / 👎.
| k_distances_new = dists[:, -1] # k-distance of each new point | ||
|
|
||
| rd_new_to_train = np.maximum( | ||
| k_distances_new[:, np.newaxis], # Broadcast k-distance to match distances shape | ||
| dists |
There was a problem hiding this comment.
For LOF novelty scoring, the reachability distance from a query to each training neighbor should be max(distance(query, neighbor), k_distance(neighbor)). This code instead broadcasts the query's own kth-neighbor distance to every neighbor, so decision_function/predict scores for new samples are biased whenever the neighbors have different k-distances, which is common near dense or sparse regions. Store the training k-distances from fit and index them with indices here.
Useful? React with 👍 / 👎.
| """ | ||
| # Lazy initialization: Initialize the neighbor searcher only if needed | ||
| if not hasattr(self, 'nbrs_'): | ||
| self.nbrs_ = NearestNeighbors(n_neighbors=self.n_neighbors, |
There was a problem hiding this comment.
fit intentionally clamps n_neighbors_ to n_samples - 1 when the requested value is too large, but the lazy NearestNeighbors is created with the original self.n_neighbors. In the documented/tested case such as FastLOF(n_neighbors=300).fit(X) on 200 samples, the first decision_function/predict call asks scikit-learn for more neighbors than were fitted and fails instead of using the adjusted count.
Useful? React with 👍 / 👎.
|
|
||
| # Symmetric update for chunk j (if different chunks) | ||
| if i != j: | ||
| _update_neighbors(neighbors, neighbor_dists, tpos, qpos_active, M.T, k) |
There was a problem hiding this comment.
After pruning starts, active_mask is applied to chunk i, and this symmetric update then offers only qpos_active as candidate neighbors for every point in chunk j. With the default threshold, normal dense points can be marked inactive and then excluded as possible nearest neighbors for later active/outlier points in other chunks, corrupting their kNN sets and LOF scores; pruning should avoid querying inactive points, not remove them from other points' candidate neighborhoods.
Useful? React with 👍 / 👎.
| n_samples = X.shape[0] | ||
|
|
||
| # Validate and adjust n_neighbors (match scikit-learn behavior) | ||
| if self.n_neighbors >= n_samples: |
There was a problem hiding this comment.
This adjustment only handles values that are too large; if a caller passes n_neighbors=0 (or fits a single-sample dataset, which makes n_neighbors_ become 0), the neighbor arrays have zero columns and _update_neighbors later indexes heap_dists[0], causing a low-level runtime failure. Validate that the effective neighbor count is at least 1 before entering the FastLOF computation.
Useful? React with 👍 / 👎.
|
Thanks for adding FastLOF, @Powerscore. A faster LOF variant (Goldstein, ICPR 2012) is a welcome addition, and the core LOF math checks out: a single-chunk run reproduces scikit-learn's LOF at correlation 1.0. A few changes are needed before this can merge. 1. clf = FastLOF(n_neighbors=300).fit(X_train) # 200 rows: fit() succeeds
clf.decision_function(X_test) # ValueError: Expected n_neighbors <= n_samples_fitFix: use the clamped 2. Undefined citation breaks the docs build (blocker). 3. Strengthen the tests. Minor (non-blocking): the module and example headers credit "Markus Goldstein" as Author; the PyOD convention lists the implementer (Goldstein is the paper author). The file is also missing a trailing newline. Happy to re-review once these are addressed. Thanks again for the contribution. |
Release v3.6.1: pythresh v1, EmbeddingOD air-gapped encoder, DataFrame fix
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 341d970783
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| LOF scores for the new data points. | ||
| """ | ||
| # Lazy initialization: Initialize the neighbor searcher only if needed | ||
| if not hasattr(self, 'nbrs_'): |
There was a problem hiding this comment.
When the same estimator is refit after a prior decision_function/predict, the old nbrs_ attribute remains fitted on the previous X_train_, so this guard skips rebuilding it. Subsequent scores then combine neighbor indices from the old training set with lrd_ from the new fit (or raise if the indices no longer fit); clear nbrs_ in fit or rebuild it whenever the training data changes.
Useful? React with 👍 / 👎.
| if metric_type >= 0: | ||
| return _compute_distances_numba(X, Y, metric_type, self.p, is_symmetric) |
There was a problem hiding this comment.
When callers supply metric_params for one of the optimized metrics, e.g. metric='minkowski' with weights in metric_params, this branch returns Numba distances before those parameters are applied. Training scores are then computed with unweighted distances while decision_function passes the same params to NearestNeighbors, so fit and query use different metrics; fall back to the sklearn/scipy path or implement the params here.
Useful? React with 👍 / 👎.
| X = np.ascontiguousarray(X, dtype=np.float32) | ||
| Y = np.ascontiguousarray(Y, dtype=np.float32) |
There was a problem hiding this comment.
For float64 inputs with large offsets or small separations, this unconditional float32 cast can change the nearest-neighbor ordering before LOF is computed; for example values around 1e8 that differ by 1 collapse to the same float32 value, producing zero distances and corrupted scores. Since the estimator accepts normal NumPy float64 data without documenting reduced precision, keep the input precision or make the approximation explicit/configurable.
Useful? React with 👍 / 👎.
| # Validate n_jobs | ||
| if self.n_jobs is not None and self.n_jobs < -1: | ||
| raise ValueError(f"n_jobs must be None, -1, or positive, got {self.n_jobs}") |
There was a problem hiding this comment.
This validation allows n_jobs=0, so fit succeeds even though the value is later forwarded to NearestNeighbors in decision_function/predict, where joblib/scikit-learn treat zero workers as invalid. Users who configure FastLOF(n_jobs=0) therefore get a delayed prediction-time failure instead of the intended constructor/fit-time validation; reject zero here as well.
Useful? React with 👍 / 👎.
| params = self.metric_params or {} | ||
| if self.metric == 'minkowski': | ||
| params['p'] = self.p | ||
| return cdist(X, Y, metric=self.metric, **params) |
There was a problem hiding this comment.
For metric='cosine', SciPy's cdist returns NaNs whenever either vector is all zeros, and this call succeeds so the safer sklearn fallback is never reached. Zero vectors are common in sparse/text-like feature spaces that use cosine distance, and the NaNs prevent those candidates from entering the neighbor heaps, leaving corrupted LOF scores; use pairwise_distances for cosine or explicitly handle zero norms.
Useful? React with 👍 / 👎.
| params = self.metric_params or {} | ||
| if self.metric == 'minkowski': | ||
| params['p'] = self.p | ||
| return cdist(X, Y, metric=self.metric, **params) |
There was a problem hiding this comment.
For fallback metrics such as seuclidean or mahalanobis without explicit metric_params, SciPy estimates the variance/covariance from only the two arrays passed to each cdist call. Because FastLOF calls this once per chunk pair, distances are scaled differently for different chunk pairs, so the kNN ordering and LOF scores depend on the random chunking; precompute global V/VI or require callers to supply them.
Useful? React with 👍 / 👎.
Address reviewer blockers and minor notes on the FastLOF addition: - Fix decision_function crash when n_neighbors >= n_samples by using the clamped n_neighbors_ (not the raw n_neighbors) when building the NearestNeighbors searcher in _compute_lof_scores. - Add goldstein2012fastlof entry to docs/zreferences.bib and cite it in place of the undefined goldstein2016comparative key, which broke the Sphinx build. - Strengthen tests: restore the test-set roc_auc_score >= roc_floor assertion in test_prediction_scores, and add test_decision_function_large_n_neighbors to lock in the n_neighbors fix. - Credit the implementer in the module and example Author headers; add trailing newline to fastlof.py.
Powerscore
force-pushed
the
feature/fastlof
branch
from
341d970 to
3710aee
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3710aee145
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| # Find k nearest neighbors in training data | ||
| # dists: Distances to the k nearest neighbors | ||
| # indices: Indices of the k nearest neighbors in X_train | ||
| dists, indices = self.nbrs_.kneighbors(X) |
There was a problem hiding this comment.
When callers score the fitted training set, e.g. through predict(X_train) or predict_proba(X_train), this call supplies X to kneighbors, so each row is returned as its own nearest neighbor instead of being excluded as it was during fit via the infinite diagonal. With small n_neighbors (especially n_neighbors=1), the zero self-distance drives the query LRD to an artificial value and can make training predictions inconsistent with the already computed labels_; handle the fitted-data case by requesting one extra neighbor and dropping self matches.
Useful? React with 👍 / 👎.
| params = self.metric_params or {} | ||
| if self.metric == 'minkowski': | ||
| params['p'] = self.p | ||
| return pairwise_distances(X, Y, metric=self.metric, **params) |
There was a problem hiding this comment.
The class advertises metrics supported by pairwise_distances, which includes metric='precomputed', but this fallback passes row subsets of the full distance matrix as X and Y. Once chunking creates a target chunk smaller than the full training set, sklearn expects the precomputed block to have shape (len(q_indices), len(t_indices)), not (len(q_indices), n_samples), so fit fails instead of using the supplied distances; either reject precomputed up front or slice the block with the query and target indices before this call.
Useful? React with 👍 / 👎.
| return distances | ||
|
|
||
|
|
||
| class FastLOF(BaseDetector): |
There was a problem hiding this comment.
Adding the class here without a matching pyod/utils/knowledge/algorithms.json entry leaves it invisible to ADEngine/MCP/pyod info, because ADEngine.list_detectors() enumerates engine.kb.algorithms rather than scanning modules. I checked repo-wide rg FastLOF fastlof, and the only hits are this new module/test/example/reference, so users cannot select or build the shipped FastLOF detector through the catalog/agent APIs; add the catalog metadata and routing/docs entries as appropriate.
Useful? React with 👍 / 👎.
@yzhao062 Thank you very much for your review and guidance, I really appreciate it. I've addressed all the changes you requested (the n_neighbors_ fix in _compute_lof_scores, the goldstein2012fastlof citation, the strengthened tests including a decision_function regression test for the clamped-neighbor case, and the author-header/trailing-newline cleanups), and brought the branch up to date with the latest upstream for a cleaner git history. I also noticed the Codex bot raised some additional suggestions. Would you like me to address those in this PR as well, or would you prefer to keep this one scoped and handle them as follow-ups? Happy to do whichever you prefer. |
2 participants
Summary
This PR implements FastLOF, a high-performance approximation of the Local Outlier Factor algorithm proposed by Markus Goldstein (2012) in "FastLOF: An Expectation-Maximization based Local Outlier Detection Algorithm".
By utilizing an Expectation-Maximization (EM) strategy combined with Numba-optimized JIT compilation, FastLOF addresses the computational bottlenecks of standard LOF. It achieves order-of-magnitude speedups on large ($N > 100k$ ) and high-dimensional ($D > 1000$ ) datasets while maintaining high detection fidelity.
Motivation
Standard LOF implementations face significant scalability challenges:
Original Research Insight:
As identified by Goldstein (2012), there is a key asymmetry in outlier detection: precise nearest neighbor identification is essential for outliers but less critical for normal instances. FastLOF exploits this by iteratively estimating density and "pruning" obvious inliers from the calculation, allowing it to skip the vast majority of distance computations.
Technical Details
Algorithm: Expectation-Maximization Strategy
FastLOF operates on the heuristic proposed in the original paper:
Implementation Stack
@njit(parallel=True, fastmath=True).Benchmarks & Performance
We evaluated FastLOF against the PyOD baseline (Brute Force, KD-Tree, Ball Tree) across 15 datasets.
1. Speedup on Large Datasets ($N > 100k$ )
FastLOF consistently outperforms standard implementations on large datasets where$O(N^2)$ is prohibitive.
2. High-Dimensional Efficiency
On datasets where tree-based methods fail due to the curse of dimensionality, FastLOF retains its advantage.
3. Detection Quality (ROC-AUC)
Changes Made
pyod/models/fastlof.py- Core implementation using Numba.examples/fastlof_example.py- Standard usage script.pyod/test/test_fastlof.py- Comprehensive unit tests covering parameters, shapes, reproducibility, and prediction consistency.Usage Example
Research Foundation
Reference: Goldstein, Markus (2012). "FastLOF: An Expectation-Maximization based Local Outlier Detection Algorithm." In Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), pp. 2282-2285. IEEE.
BibTeX:
Checklist
All Submissions Basics:
All Submissions Cores: