Skip to content

About

Zoom In, Scale Out: a label-free digital-zoom test showing that metric depth models mistake digital zoom for moving closer, and how to fix it.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

Zoom In, Scale Out

Metric Depth Models Mistake Digital Zoom for Moving Closer

A KITTI image at 1x, 2x and 3x digital zoom: Depth Anything V2 moves the scene closer

Digital zoom (crop the centre, resize back) changes the camera, not the scene. The focal length grows by the zoom factor s and the true depth stays the same. We use this as a free, exact test for metric depth models and find that most of them read a zoom as the camera moving closer.

We measure two numbers per image, with no depth labels needed:

  • α, depth exponent: depth ∝ s−α (ideal 0)
  • β, focal exponent: focal ∝ sβ (ideal 1, only for models that predict their camera)

Highlights

  • Depth Anything V2 has α = 0.95: almost all of the zoom is read as moving closer.
  • Models that predict their own camera (Depth Pro, MoGe-2, UniDepthV2) do better (α ≈ 0.31–0.36), but their focal grows too little.
  • Given the true camera, the same networks keep their depth (α ≈ 0), so the error is in the focal head.
  • FocalUnshrink, a fix with no labels and no training, brings Depth Pro to α = 0.04.
  • EFT+B, fine-tuning with a zoom self-consistency loss, brings DA-V2-S to α = 0.05 (KITTI) and 0.07 (NYUv2).

Results

Zooms 1–3×, 800 images (200 each from KITTI, NYUv2, DIODE indoor and outdoor), median over images with a 95% interval from a scene-level bootstrap (as in Table 1 of the paper). Per-model CSV summary: outputs/tables/table1_pooled.csv.

Model Camera α (↓, ideal 0) β (ideal 1)
DA-V2-S pixels only 0.95 [0.83, 1.12] –
DA-V2-B pixels only 0.85 [0.76, 0.94] –
ZoeDepth pixels only 0.61 [0.53, 0.75] –
MoGe-2-S predicts 0.36 [0.30, 0.40] 0.56
UniDepthV2-S predicts 0.34 [0.23, 0.51] 0.45
Depth Pro predicts 0.31 [0.20, 0.41] 0.65
Metric3D v2-S takes the true camera −0.07 [−0.11, −0.05] –
Metric3D v2-S takes the old (un-zoomed) camera 0.93 [0.89, 0.95] –

Installation

git clone <this repository> && cd zoom-equivariance
python -m venv .venv && . .venv/bin/activate
pip install torch --index-url https://download.pytorch.org/whl/cpu   # or a CUDA build
pip install -r requirements.txt

# two models are installed from their own repositories
git clone --depth 1 https://github.com/lpiccinelli-eth/UniDepth.git third_party/UniDepth
pip install --no-deps -e third_party/UniDepth
git clone --depth 1 https://github.com/YvanYin/Metric3D.git third_party/Metric3D

requirements-lock.txt has the exact versions behind every result.

Data

bash scripts/download.sh     # KITTI depth selection (val), NYUv2 labelled set, DIODE val -> data/raw/

Sources, licences and the exact image ids of every sample are in DATA.md and outputs/samples/. SR-RAW (real optical zoom) is described in DATA.md too.

Quick start

Zoom one image and compare the model's depth:

import sys; sys.path.insert(0, "src")
import numpy as np
from PIL import Image
import models, zoom

model = models.load("depthpro")                 # any name from src/models.py REGISTRY
img = np.array(Image.open("my_photo.jpg").convert("RGB"))   # uint8 H x W x 3

h, w = img.shape[:2]
box = zoom.zoom_box(h, w, s=2.0)                # centred crop for a 2x zoom
img2 = zoom.zoom_image(img, box)                # crop + bicubic resize back to H x W

d1 = model.predict(img)["depth"]
d2 = model.predict(img2)["depth"]
d1_crop = zoom.resize_depth(zoom.crop(d1, box), (h, w))
print("depth ratio, ideal 1:", np.median(d2 / d1_crop))   # about 1/2 means "moved closer"

Evaluation

# audit: alpha, beta and errors against ground truth for one model on several sets
python scripts/run_audit.py depthpro kitti nyu diode-in diode-out --n 200

# controls used in the paper
python scripts/run_audit.py depthpro kitti --n 200 --off-centre     # zoom into the four quadrants
python scripts/run_audit.py depthpro kitti --n 200 --cues           # context loss vs magnification
python scripts/run_audit.py depthpro kitti --n 200 --jpeg-zoom      # JPEG before / after the zoom
python scripts/run_audit.py metric3d-s kitti --n 200 --stale-k      # give the old camera after zooming

python scripts/run_regions.py dav2-s kitti nyu --n 200        # alpha on objects vs walls / floors / roads
python scripts/run_srraw.py dav2-s --root data/raw/srraw/test  # real optical zoom (SR-RAW) vs digital zoom

Each run writes a per-image CSV to outputs/results/. Model names: dav2-s, dav2-b, dav2-l, zoedepth, depthpro, moge2-s/b/l, unidepth2-s/b/l, unik3d-l, metric3d-s/l, da3-nested, da3-metric-l, geocalib, and the -k variants (given the true camera).

Training (EFT)

# EFT / EFT+B: fine-tune DA-V2-S with the zoom loss (+ blur term, --w-blur) on unlabeled images
python scripts/train_eft.py outdoor --images "data/raw/kitti_raw/*/*/image_02/data/*.png"
python scripts/train_eft.py indoor  --nyu-train

# EFT-S: frozen network, one learned log-scale per image (cache the frozen outputs, then fit)
python scripts/eft_scale.py cache outdoor --images "data/raw/kitti_raw/*/*/image_02/data/*.png"
python scripts/eft_scale.py fit   outdoor

# EFT-F: tune only Depth Pro's focal head
python scripts/train_eft_focal.py --kitti "data/raw/kitti_raw/*/*/image_02/data/*.png" --nyu-train

Only RGB is read; depth labels of the training images are never used. Training settings and every checkpoint are listed in CHECKPOINTS.md. Checkpoints are not in git.

Reproducing the paper's numbers

All result CSVs are already in outputs/results/. To rebuild every table and number:

python scripts/make_paper_numbers.py

No number in the paper is typed by hand; all of them come from this script. The large runs were done on Kaggle: scripts/kaggle_build.py packs src/ and the job list into a kernel (credentials only through the KAGGLE_USERNAME / KAGGLE_KEY environment variables).

Repository layout

src/            zoom geometry, datasets, model wrappers, audit, analysis, EFT trainers
scripts/        command-line entry points (audit, regions, SR-RAW, training, paper numbers)
kaggle/         kernel templates for the large runs
notebooks/      analysis notebooks
tests/          unit tests (no model weights needed)
outputs/        result CSVs, tables, figures, training logs

Run the tests with pytest.

Acknowledgements

This work builds on the released code and weights of Depth Anything V2, Depth Anything 3, ZoeDepth, Depth Pro, MoGe-2, UniDepthV2, UniK3D, Metric3D v2, VGGT, GeoCalib and SegFormer, and on the KITTI, NYUv2, DIODE and SR-RAW datasets. We thank their authors.

Citation

A BibTeX entry will be added after the review.

License

Code: MIT (see LICENSE). Datasets and model weights keep their own licences.

About

Zoom In, Scale Out: a label-free digital-zoom test showing that metric depth models mistake digital zoom for moving closer, and how to fix it.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages