feat: Add CLEAR MOT tracking evaluation metrics (MOTA, MOTP) with combined TrackingMetrics class#2193
Draft
Preksha2 wants to merge 2 commits intoroboflow:developfrom
Draft
feat: Add CLEAR MOT tracking evaluation metrics (MOTA, MOTP) with combined TrackingMetrics class#2193Preksha2 wants to merge 2 commits intoroboflow:developfrom
Preksha2 wants to merge 2 commits intoroboflow:developfrom
Conversation
e50d010 to
424c9a9
Compare
Member
|
@Preksha2 thank you for your proposal. However, I think supervision will deprecate the tracking section since we created it as a separate repo - https://github.com/roboflow/trackers Am I right @SkalskiP" 🦝 |
Author
|
Thanks for the feedback @Borda! That makes sense — I wasn't aware tracking was being separated into its own repo. I'd be happy to move this contribution to https://github.com/roboflow/trackers instead if that's the better home for tracking metrics. The implementation (MOTA, MOTP, TrackingMetrics) follows the same update()/compute() pattern and should integrate cleanly there. Alternatively, if there's another area in supervision where metrics contributions are welcome, I'm happy to adapt. Let me know how you'd like me to proceed! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before submitting
Description
Adds CLEAR MOT tracking evaluation metrics to supervision's metrics module, implementing the standard defined in Bernardin & Stiefelhagen, "Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics" (JIVP, 2008). Includes MOTA (tracking accuracy), MOTP (tracking precision), and a combined TrackingMetrics class that computes both in a single pass.
Type of Change
Motivation and Context
Supervision has detection metrics (mAP, F1) but no tracking metrics. MOTA and MOTP are the two most widely used multi-object tracking evaluation metrics, standard in benchmarks like MOTChallenge. Tracking metrics were mentioned as welcome contributions in the 0.25.0 release notes. This PR fills that gap.
Changes Made
MultiObjectTrackingAccuracyclass (MOTA) measuring overall tracking quality via false positives, false negatives, and identity switchesMultiObjectTrackingPrecisionclass (MOTP) measuring average localization precision of matched detectionsTrackingMetricsclass computing both MOTA and MOTP in a single pass (single IoU matrix computation and matching per frame)MOTAResult,MOTPResult,MOTMetricsResultdataclasses with full component breakdowntracker_idupdate()/compute()interface consistent with the metrics APITesting
30 tests covering: perfect tracking, false positives, false negatives, identity switches, IoU thresholds, partial overlap localization, empty frames, reset behavior, input validation, and string representations.
MOTA = 1 - (FN + FP + IDSW) / GT
MOTP = sum(IoU of matched pairs) / num_matches
Usage:
Additional Notes
Happy to add documentation entries for autogeneration and adjust the API style if the maintainers have preferences. This is my first contribution to supervision.