Cal-DETR: Calibrated Detection Transformer

Posted Feb 6, 2024 Updated Feb 7, 2024

By Geonu-Lee 3 min read

Cal-DETR: Calibrated Detection Transformer

NeurIPS 2023, 2024-02-06 기준 0회 인용

Task

Object Detection
DETR
Calibration

Contribution

Safety-critical applications 에 Deep neural networks (DNN) 를 적용 및 활용하는데 있어 제한을 갖고 있다
a deployed model could be exposed to constantly changing distributions which could be radically different from its training distribution.
Proposing a mechanism for calibrated detection transformers (Cal-DETR)
Uncertainty-guided logit modulation mechanism 제안
Logit mixing approach 제안

Proposed Method

Uncalibrated - cat을 dog로 꽤 높은 confidence score를 기반으로 잘못 예측 Calibrated - cat으로 정확하게 예측

Measuring Calibration

Expected Calibration Error (ECE) - For Classification
Datasets 을 $B$ 개수 만큼 나누어서 진행 - bin
(Average accuracy - Average confidence) 에 각 bin 마다의 weights를 곱하고 sum
-> miscalibration 를 meausre

Detection Expected Calibration Error (D-ECE) - For Detection
ECE 와 거의 동일한 형식이지만 Detection 에서는 Precision을 사용
mean Average Precision (mAP) 을 사용하지 않는 이유는 Recall 부분 때문

Cal-DETR: Uncertainty-Guided Logit Modulation & Mixing

Uncertainty-Guided Logit Modulation

$O_D \in \mathbb{R}^{L \times M \times Q \times C}$
$L$ 개의 decoder layer 에 대한 결과
$M$ 은 mini-batch
$C$ 는 Class number

$O_D^L \in \mathbb{R}^{M \times Q \times C}$
마지막 decoder layer 결과

$O_D \in \mathbb{R}^{L \times M \times Q \times C}$ 의 variance 를 계산
$u \in \mathbb{R}^{M \times Q \times C}$ -> uncertainty
$1-\text{tanh}(u)$ -> certainty

$\tilde{O}_D^L = O_D^L \otimes (1-\text{tanh}(u))$

$L$ 개의 decoder layer 결과로부터 구해진 uncertainty 를 활용해서 certainty 를 구하고 마지막 decoder layer 결과에 곱한다

Logit Mixing for Confidence Calibration

$Q_p = {Q_1, Q_2, Q_3 } \subset Q$ (total number of quries)

$\tilde{Q}$ 는 positive quries 들에 대한 평균
$\alpha$ 는 0.9 를 사용

이렇게 Logit Mixing 된 결과는 regularization loss 로 활용
label 은 $\alpha$ 로 smoothing (0.9)

$L_{cls} + \lambda L_R$ 으로 학습

The proposed regularizer loss tends to capture the information from the vicinity of object instances in the logit space by mixing logits and constructing a non-zero entropy target distribution.

Experimental Results

Deformable-DETR 에 적용했을 때 D-ECE 가 감소

CorCOCO

The CorCOCO evaluation set is constructed by incorporating random severity and random corruption levels from [13].

[13] - Hendrycks, D. and T. Dietterich (2019). Benchmarking neural network robustness to common corruptions and perturbations. International Conference on Learning Representations (ICLR).