Mirage is an evaluation framework for vision language models. It measures object hallucination with POPE, ROPE, CHAIR and THRONE, each modified so it still works on datasets with sparse labels such as Visual Genome. Results, methods and guides are at mirage-eval.com.
pip install -r requirements.txt
python -m nltk.downloader punkt_tab averaged_perceptron_tagger_eng stopwords wordnet
python run.py --dataset coco --metric pope --model gemma4-e4b --samples 10
python repro.py --records tests/data/records.json.gz --check # reproduce the results tableOpen source under the Apache License 2.0, including commercial use. Keep the NOTICE file with any copy, and if you use Mirage, please acknowledge it and cite it with CITATION.cff. Test images and annotations in tests/data/wiring keep their own licenses (ATTRIBUTION.md).