A Generative Perspective on
MRFs in Low-Level Vision
Uwe Schmidt
Qi Gao
Stefan Roth
Department of Computer Science
TU Darmstadt
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010
Low-Level Vision
SuperResolution
Discriminative
! performance
Generative
" versatility
! versatility
" learning
Image
Restoration
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 2
Stereo
Optical Flow
Low-Level Vision
SuperResolution
Generative
MRF
Stereo
Priors
Image
Restoration
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 2
Optical Flow
Common MRF Evaluation
MRF prior
p(x)
Application likelihood
p(y|x)
y
generate/
measure
Posterior
p(x|y)
MAP estimation
Indirect
model
evaluation
(Gradient methods,
Graph cuts, ...)
^
x
Compare
# Measure PSNR, ...
Ground truth
MAP estimate
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 3
Desirable MRF Evaluation
■ Purpose of MRF priors
• Model statistical properties of
Difficult!
natural images and scenes
$ Evaluate generative properties
[Zhu & Mumford ’97]
MRF prior
Draw samples
(MCMC)
• e.g. derivative statistics of the model
• neglected ever since
Compare
statistical properties
Data
MRF samples
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 4
Agenda
1. Evaluate generative properties of common image priors
• Pairwise & high-order MRFs
• Based on a flexible MRF framework with an efficient sampler
2. Learn improved generative models
3. Find that in the context of MAP estimation our models
do not perform as well as expected for image denoising
4. Address this problem (and others) by changing the
estimator
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 5
Flexible MRF Model
■ Fields-of-Experts (FoE) framework [Roth & Black ’05, ’09]
• Subsumes popular pairwise & high-order MRFs
N
YY
2
1
⇤x⇤ /2
p(x;
)=
Z( )
Image
e
Expert function
T
Ji x(c) ; ↵i
c⇥C i=1
Linear filter
Parameters
e.g.
= {Ji , ↵i }
i = 1, . . . , N
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 6
Vector of nodes
in clique c
Flexible MRF Model
■ Fields-of-Experts (FoE) framework [Roth & Black ’05, ’09]
• Subsumes popular pairwise & high-order MRFs
N
YY
2
1
⇤x⇤ /2
p(x;
−1
10
)=
Z( )
e
Mixture Weights
T
Ji x(c) ; ↵i
c⇥C i=1
GSM example
Gaussian Scale Mixture
(GSM) Distribution
−3
10
[Wainwright & Simoncelli ’99,
Weiss & Freeman ’07]
−5
10
−200
−100
⇤(JT
i x(c) ; ↵i ) =
XJ
0
j=1
100
ij
200
2
· N (JT
x
;
0,
⇥
i (c)
i /sj )
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 7
Sampling from the MRF
■ Obtain joint distribution:
Mixture Weights
• Product of GSMs = GSM
• Augment MRF with auxiliary variables z
for the mixture components and
do not marginalize them out
X
z
Gaussian
p(z) p(x|z)
| {z }
p(x, z)
■ Gibbs sampling from the joint distribution p(x, z; )
[Geman & Yang ’95; Welling et al. ’02]
p(x|z) ) and p(z|x;
p(z|x) )
• Alternate block sampling from p(x|z;
• The z can be discarded in the end
• Least-squares method for sampling p(x|z; ) [Weiss ’05, Levi ’09]
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 8
MRF Sampling – Example
Pairwise MRF
High-order MRF
with 3 3 cliques
Subsequent iterations
of the Gibbs sampler
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 9
Generative Properties of Pairwise MRFs
30
25
20
15
■ Consider simplest
pairwise MRFs
−90
−95
MRF
−100
−105
−1
10
Potential function
Fit to the
Generalized
marginals
Laplacian
−3
−1
10
Derivative marginals
KLD=1.57
Natural
KLD=1.37
images
Marginal
KL-divergence
−3
10
10
−5
−5
10
10
−200 −100
0
100
200
−200 −100
0
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 10
100
200
Generative Properties of High-order MRFs
[Roth & Black ’09]
24 5 5 filters
Student-t experts
■ Common FoE models
• Evaluate filter statistics
of model filters Ji
■ Apparent
contradiction:
" Poor generative
properties
−1
10
Natural
Images
−3
10
−5
10
KLD=2.10
−1
Why?
KLD=5.26
10
! Good application
performance
[Weiss & Freeman ’07]
25 15 15 filters
fixed GSM experts
MRF
Samples
−3
10
−5
10
−150 −75
0
75
150
−150 −75
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 11
0
75
150
Learning Better Generative MRFs
■ Learn shapes of flexible GSM experts
and linear filters Jii
Ji (for high-order model)
• Use efficient sampler
• Otherwise training similar to [Roth & Black ’09]
■ Learned models:
1. Pairwise MRF with single GSM potential
(fixed first-derivative filters)
2. FoE with 3 3 cliques and
8 GSM experts (including filters)
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 12
Generative Properties of Our Pairwise MRF
■ Our pairwise MRF
compared to
previously shown
200
−100
250
100
−200
200
0
−300
150
30
25
20
15
−400
−90
−95
MRF
−100
−105
−1
10
Potential function
Our learned
GSM
−1
10
−3
Derivative marginals
KLD=0.006
Natural
images
−3
10
10
−5
−5
10
10
−200 −100
0
100
200
−200 −100
0
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 13
100
200
15
10
50
0
−5
Our Learned FoE in Comparison
Our learned 3 3 FoE
GSM
−150 −75
0
75
150
[Roth & Black ’09]
[Weiss & Freeman ’07]
Student-t
GSM
−150 −75
0
75
150
−150 −75
Learned linear filters
Much more
peaked!
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 14
0
75
150
Generative Properties of our FoE
■ Filter statistics of our learned 3 3 FoE
• Much better than previous models
• Room for improvement
−1
10
Natural
Images
KLD=0.08
MRF
Samples
−3
10
−5
10
−150 −75
0
75
150
−150 −75
0
75
150
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 15
Image Denoising
■ Image denoising assuming i.i.d. Gaussian noise with
known standard deviation σ
Gaussian Likelihood
p(x|y;
)
N y; x,
λ=1
2
Our 3 3 FoE
I · p(x;
)
Regularization
weight
optimal λ
[Roth & Black ’09]
MAP, optimal λ
PSNR=29.18dB
PSNR=30.06dB
MAP
PSNR=22.18dB
PSNR=26.64dB
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 16
Image Denoising – MAP
■ Recent works point to deficiencies of MAP
[Nikolova ’07, Woodford et al. ’09]
■ We find only modest correlation between:
• Image quality of the MAP estimate
• Generative quality of the MRF
Better generative properties
!
Better application performance
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 17
Image Denoising – MMSE
■ We propose to use Bayesian minimum
mean squared error estimation (MMSE)
Z
2
ˆ = arg min ||˜
x
x x|| p(x|y; ) dx = E[x|y]
˜
x
Samples
average
■ [Levi ’09] extended sampler to the posterior
• Only used a single sample in applications
■ We approximate the MMSE estimate
• Average samples from the posterior
■ We find high correlation between:
• Image quality of the MMSE estimate
• Generative quality of the MRF
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 18
MMSE
Image Denoising – Results
■ Compared the MMSE estimate for our learned models
with other popular methods
Average PSNR (dB) for 68 test images (σ = 25)
5 5 FoE [Roth & Black ’09] – MAP w/λ 27.44
Non-local means [Buades et al. ’05]
pairwise (ours) – MMSE
27.50
27.54
27.86
5 5 FoE [Samuel & Tappen ’09] – MAP
27.95
3 3 FoE (ours) – MMSE
28.02
BLS-GSM [Portilla et al. ’03] – (MMSE)
26.5
27.0
27.5
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 19
28.0
Advantages of the MMSE
■ Denoising performance highly correlated
with the generative quality of the model
■ No regularization weight λ required to perform well
■ Denoised image does not exhibit incorrect statistics
MAP
MMSE
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 20
Advantages of the MMSE
■ Denoising performance highly correlated
with the generative quality of the model
■ No regularization weight λ required to perform well
■ Denoised image does not exhibit incorrect statistics
Derivative marginals
• No piecewise constant regions
−1
10
MAP
MMSE
−3
10
−100
0
100
Original images (noise-free)
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 20
Advantages of the MMSE
■ Denoising performance highly correlated
with the generative quality of the model
■ No regularization weight λ required to perform well
■ Denoised image does not exhibit incorrect statistics
Derivative marginals
• No piecewise constant regions
MAP
KLD=1.64
−1
10
MAP
MMSE
−3
10
−100
0
100
[Woodford et al. ’09]
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 20
Advantages of the MMSE
■ Denoising performance highly correlated
with the generative quality of the model
■ No regularization weight λ required to perform well
■ Denoised image does not exhibit incorrect statistics
• No piecewise constant regions
• Works with standard MRFs
MAP
MMSE
Derivative marginals
−1
10
MAP
MMSE
KLD=1.64
KLD=0.03
−3
10
−100
0
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 20
100
Summary
■ Evaluated MRFs through their generative properties
• Based on a flexible framework with an efficient sampler
■ Common image priors are surprisingly poor generative
models
■ Learned better generative MRFs (pairwise & high-order)
• Potentials more peaked
■ Sampling makes MMSE estimation practical
• Several advantages over MAP
• Excellent results from generative, application-neutral models
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 21
Thanks!
Acknowledgments:
Yair Weiss, Arjan Kuijper, Michael Goesele,
Kegan Samuel, Marshall Tappen
Please come to our poster!
Code and models available soon at http://bit.ly/mmse-mrf
Questions?
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 22
More Generative Properties
KLD
Random filters 10
−1
9 9
7 7
5 5
3 3
0.13
0.09
−3
10
0
−5
0
75
150
Natural images
1
−150 −75
0
75
150
Our pairwise MRF
−150 −75
0
75
10
150
Our 3 3 FoE
KLD
−1
KLD
0.09
0.04
0
−3
10
2
0
10
−150 −75
Multiscale
derivative
filters
KLD
0
−5
10
4
−150 −75
0
75
150
Natural images
−150 −75
0
75
150
Our pairwise MRF
−150 −75
Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 26
0
75
150
Our 3 3 FoE