DDMS: Discriminative Distillation of Multi-view Foundational Features into Single-view Models

NeurIPS 2026

Jeong-gi Kwak1, Sho Kagami2, Yuki Ono3, Kwang Moo Yi1
1University of British Columbia 2Sony Semiconductor Solutions Corporation 3Sony Corporation
Paper Code (coming soon)
Overview of the method: a multi-view teacher built from a 2D foundation model and a multi-view geometry model, refined with a ranking loss and an anchor, and distilled into a single-view student.
A multi-view teacher, built from a 2D foundation model and a frozen multi-view geometry model and refined with a ranking loss and a feature-space anchor, is distilled into a single-view student, the only part used at inference.
TL;DR: We distill multi-view geometry models into a single-view encoder, obtaining 3D-aware dense features that are consistent across views and locally distinctive, while preserving semantic transferability.
Left: PCA of DINOv2 and DDMS features, cross-view matches and feature-splat renderings. Right: radar chart comparing DINOv2, Fit3D, MEF, SnD and DDMS over nine benchmarks.
Left: PCA of the features, nearest-neighbor matches across two views, and features rendered after splatting onto 3D Gaussians. Right: scores on nine benchmarks, normalized per axis (minimum set to 50%).

Abstract

Foundational visual features such as DINO have played a critical role across modern computer vision, and have recently become key components in multi-view feed-forward geometry estimators. In this work, we demonstrate that by re-distilling these multi-view models—their internal knowledge of 3D geometry—into a single-view estimator, we can obtain enhanced 3D consistent foundational features. Our key idea is to construct a multi-view teacher by fusing pretrained 2D foundation features with multi-view geometric features, and refining the fused representation with a discriminative ranking objective. Through our discriminative distillation framework, we enforce the learned features to be both 3D consistent and locally distinctive, while keeping them aligned with the feature space of the original foundation model to preserve the semantic structure of the pretrained representation. Consistency and local discriminability are critical for 3D computer vision problems such as forming semantic and geometric correspondences across images. To demonstrate the effectiveness of our method, we perform comprehensive experiments spanning multiple angles: direct feature analysis, dense prediction transfer, and explicit 3D lifting and rendering. Across these evaluations, our method consistently produces stronger 3D-aware foundation features that improve multi-view consistency and local discriminability while preserving the semantic transferability of the original representation.

Results

We compare against other 3D-aware fine-tunings of DINOv2 (Fit3D, MEF, SnD) on direct feature quality, dense prediction by linear probing, and after lifting the features to 3D. SnD is reported with both of its released checkpoints, without SAM-mask blending (SnD) and with it (SnD (+blending)). In the tables, the best result in each column is bold and the second best is underlined, within each backbone group.

Correspondence

On ScanNet, nearest-neighbor matching accuracy goes from 21.7 (DINOv2) to 66.0 PCK@10px, and the advantage holds as the viewpoint change between the two images grows.

Nearest-neighbor matches of DINOv2, Fit3D, MEF, SnD and ours on a ScanNet pair (top) and a NAVI-Wild pair (bottom).
Nearest-neighbor matches on ScanNet (top) and NAVI-Wild (bottom). Green lines are correct matches and red lines are incorrect ones.
PCK@10px and PCK@20px on ScanNet as a function of the relative rotation between the two views.
ScanNet PCK@10px and PCK@20px by relative rotation between the two views (ViT-B).
SeparationGeometric correspondenceSemantic matching
MethodScanNetScanNetNavi-WDAVISSPair-71k
Margin↑@10px↑@20px↑@0.05↑@0.1↑J&F↑@0.05↑@0.1↑
ViT-B
DINOv2 (baseline) 0.4108 21.72 36.11 57.97 75.94 67.82 52.98 68.56
Fit3D (ECCV'24) 0.3494 22.89 39.50 31.26 53.81 66.70 30.25 44.86
MEF (ICLR'25) 0.4288 39.17 55.06 67.71 80.87 66.82 56.78 72.07
SnD (ICLR'26) 0.4520 28.65 45.67 59.53 76.86 68.05 53.68 69.65
SnD (+blending) 0.3526 24.37 40.79 56.33 75.11 69.11 52.25 68.14
Ours 0.4632 66.01 78.48 65.66 79.40 68.95 61.52 75.46
ViT-S
DINOv2 (baseline) 0.3967 23.00 38.16 53.72 73.03 65.78 51.11 67.16
Fit3D (ECCV'24) 0.3001 22.27 39.29 26.60 47.87 66.01 21.12 34.21
MEF (ICLR'25) 0.4170 38.54 54.30 61.67 76.31 66.16 51.64 67.32
SnD (ICLR'26) 0.3819 28.65 46.24 54.13 73.30 67.10 51.13 67.83
SnD (+blending) 0.3147 23.26 39.39 50.22 70.42 68.20 49.43 66.17
Ours 0.4314 61.35 74.24 59.57 74.55 67.68 54.09 68.70

Direct feature evaluation. Separation: gap between the cosine similarity of corresponding points and of non-corresponding points. Correspondence: PCK of nearest-neighbor matches (ScanNet in pixels, NAVI-Wild and SPair-71k in normalized units); DAVIS: mask propagation.

Dense prediction

With a linear probe on frozen features, semantic segmentation is on par with SnD (+blending), the strongest baseline here, and depth is competitive, while the correspondence is far ahead of both SnD checkpoints.

Semantic segmentationDepth estimation
ScanNetScanNet++ScanNetNYUv2
MethodmIoU↑mAcc↑aAcc↑mIoU↑mAcc↑aAcc↑AbsRel↓RMSE↓δ1↑AbsRel↓RMSE↓δ1↑
ViT-B
DINOv2 53.69 66.29 77.63 31.69 41.41 81.47 0.1553 0.3718 78.12 0.1705 0.6566 74.90
Fit3D 51.35 63.37 76.96 30.58 38.31 84.06 0.2045 0.4599 66.70 0.1975 0.7426 68.03
MEF 52.49 65.17 76.93 29.40 38.62 81.43 0.1677 0.3937 75.06 0.1661 0.6222 75.98
SnD 56.36 69.40 79.43 31.95 41.67 82.62 0.1449 0.3446 80.89 0.1537 0.6018 78.56
SnD (+blending) 57.64 70.35 80.52 33.37 42.73 84.13 0.1454 0.3459 80.54 0.1464 0.5777 80.70
Ours 57.00 70.51 80.61 33.53 43.00 84.34 0.1450 0.3453 80.48 0.1490 0.5835 79.78
ViT-S
DINOv2 50.18 62.81 75.06 27.80 37.06 79.52 0.1709 0.3950 74.70 0.1699 0.6465 75.26
Fit3D 44.62 56.41 72.56 26.19 33.49 81.97 0.2171 0.4821 64.44 0.2120 0.7691 65.20
MEF 47.22 59.70 73.29 24.53 32.96 78.69 0.1925 0.4319 69.72 0.1792 0.6541 73.39
SnD 51.38 64.11 76.10 27.90 36.90 81.00 0.1612 0.3780 76.45 0.1536 0.5882 79.32
SnD (+blending) 52.61 65.23 77.07 29.18 38.14 82.40 0.1631 0.3777 76.22 0.1531 0.5842 79.46
Ours 52.23 64.96 77.12 29.29 38.53 82.17 0.1628 0.3786 76.25 0.1582 0.6036 78.13

Linear probes on frozen features. Segmentation: ScanNet and ScanNet++. Depth: ScanNet and NYUv2.

3D lifting

We lift image features to 3D in two ways. For ScanNet point clouds, features are averaged over the views in which a point is visible, and a linear classifier is trained on the averaged features. Since averaging can hide disagreement between views, the same classifier is also applied to the feature of each view separately. For Mip-NeRF 360 scenes, features are assigned to pretrained 3D Gaussians and rendered from held-out views.

MethodmIoU↑mAcc↑aAcc↑Per-view mIoU↑Multi-view agree.↑
DINOv2 62.30 74.30 82.69 54.49 77.34
Fit3D 59.54 71.04 81.45 54.21 80.73
MEF 61.54 73.73 82.24 54.07 77.71
SnD 62.79 74.57 82.99 56.51 80.28
SnD (+blending) 63.26 75.18 83.25 57.14 80.80
Ours 63.32 74.85 83.36 57.79 82.49

3D point segmentation on ScanNet. Per-view mIoU applies the probe to each view’s feature without averaging; multi-view agreement is the rate at which two views of the same point predict the same class.

PCA of 2D features and of features rendered from 3D Gaussians for DINOv2, Fit3D, MEF, SnD and ours on held-out views.
PCA of the 2D features and of the features rendered from the 3D Gaussians, for held-out views.
Nearest-neighbor matching accuracy between rendered and image features: render to 2D, 2D to render, render to render.
Nearest-neighbor matching accuracy (PCK@10px) between rendered features and the 2D features of the same view.

Usage

python
import torch
from ddms import DDMS

model = DDMS.from_checkpoint("ddms_vitb14.pt").eval().cuda()
x = ...  # (B, 3, H, W) RGB, ImageNet mean/std normalised
with torch.no_grad():
    feats = model(x)                       # (B, 768, H/14, W/14)
    dense = model(x, out_hw=x.shape[-2:])  # bilinearly upsampled to the input size
# features are not L2-normalised; use cosine similarity for matching

BibTeX

@article{kwak2026ddms,
  title   = {DDMS: Discriminative Distillation of Multi-view Foundational Features
             into Single-view Models},
  author  = {Kwak, Jeong-gi and Kagami, Sho and Ono, Yuki and Yi, Kwang Moo},
  journal = {arXiv preprint arXiv:2608.23850},
  year    = {2026}
}