A Generative Perspective on MRFs in Low-Level Vision Uwe Schmidt Qi Gao Stefan Roth Department of Computer Science TU Darmstadt Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 Low-Level Vision SuperResolution Discriminative ! performance Generative " versatility ! versatility " learning Image Restoration Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 2 Stereo Optical Flow Low-Level Vision SuperResolution Generative MRF Stereo Priors Image Restoration Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 2 Optical Flow Common MRF Evaluation MRF prior p(x) Application likelihood p(y|x) y generate/ measure Posterior p(x|y) MAP estimation Indirect model evaluation (Gradient methods, Graph cuts, ...) ^ x Compare # Measure PSNR, ... Ground truth MAP estimate Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 3 Desirable MRF Evaluation ■ Purpose of MRF priors • Model statistical properties of Difficult! natural images and scenes $ Evaluate generative properties [Zhu & Mumford ’97] MRF prior Draw samples (MCMC) • e.g. derivative statistics of the model • neglected ever since Compare statistical properties Data MRF samples Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 4 Agenda 1. Evaluate generative properties of common image priors • Pairwise & high-order MRFs • Based on a flexible MRF framework with an efficient sampler 2. Learn improved generative models 3. Find that in the context of MAP estimation our models do not perform as well as expected for image denoising 4. Address this problem (and others) by changing the estimator Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 5 Flexible MRF Model ■ Fields-of-Experts (FoE) framework [Roth & Black ’05, ’09] • Subsumes popular pairwise & high-order MRFs N YY 2 1 ⇤x⇤ /2 p(x; )= Z( ) Image e Expert function T Ji x(c) ; ↵i c⇥C i=1 Linear filter Parameters e.g. = {Ji , ↵i } i = 1, . . . , N Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 6 Vector of nodes in clique c Flexible MRF Model ■ Fields-of-Experts (FoE) framework [Roth & Black ’05, ’09] • Subsumes popular pairwise & high-order MRFs N YY 2 1 ⇤x⇤ /2 p(x; −1 10 )= Z( ) e Mixture Weights T Ji x(c) ; ↵i c⇥C i=1 GSM example Gaussian Scale Mixture (GSM) Distribution −3 10 [Wainwright & Simoncelli ’99, Weiss & Freeman ’07] −5 10 −200 −100 ⇤(JT i x(c) ; ↵i ) = XJ 0 j=1 100 ij 200 2 · N (JT x ; 0, ⇥ i (c) i /sj ) Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 7 Sampling from the MRF ■ Obtain joint distribution: Mixture Weights • Product of GSMs = GSM • Augment MRF with auxiliary variables z for the mixture components and do not marginalize them out X z Gaussian p(z) p(x|z) | {z } p(x, z) ■ Gibbs sampling from the joint distribution p(x, z; ) [Geman & Yang ’95; Welling et al. ’02] p(x|z) ) and p(z|x; p(z|x) ) • Alternate block sampling from p(x|z; • The z can be discarded in the end • Least-squares method for sampling p(x|z; ) [Weiss ’05, Levi ’09] Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 8 MRF Sampling – Example Pairwise MRF High-order MRF with 3 3 cliques Subsequent iterations of the Gibbs sampler Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 9 Generative Properties of Pairwise MRFs 30 25 20 15 ■ Consider simplest pairwise MRFs −90 −95 MRF −100 −105 −1 10 Potential function Fit to the Generalized marginals Laplacian −3 −1 10 Derivative marginals KLD=1.57 Natural KLD=1.37 images Marginal KL-divergence −3 10 10 −5 −5 10 10 −200 −100 0 100 200 −200 −100 0 Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 10 100 200 Generative Properties of High-order MRFs [Roth & Black ’09] 24 5 5 filters Student-t experts ■ Common FoE models • Evaluate filter statistics of model filters Ji ■ Apparent contradiction: " Poor generative properties −1 10 Natural Images −3 10 −5 10 KLD=2.10 −1 Why? KLD=5.26 10 ! Good application performance [Weiss & Freeman ’07] 25 15 15 filters fixed GSM experts MRF Samples −3 10 −5 10 −150 −75 0 75 150 −150 −75 Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 11 0 75 150 Learning Better Generative MRFs ■ Learn shapes of flexible GSM experts and linear filters Jii Ji (for high-order model) • Use efficient sampler • Otherwise training similar to [Roth & Black ’09] ■ Learned models: 1. Pairwise MRF with single GSM potential (fixed first-derivative filters) 2. FoE with 3 3 cliques and 8 GSM experts (including filters) Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 12 Generative Properties of Our Pairwise MRF ■ Our pairwise MRF compared to previously shown 200 −100 250 100 −200 200 0 −300 150 30 25 20 15 −400 −90 −95 MRF −100 −105 −1 10 Potential function Our learned GSM −1 10 −3 Derivative marginals KLD=0.006 Natural images −3 10 10 −5 −5 10 10 −200 −100 0 100 200 −200 −100 0 Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 13 100 200 15 10 50 0 −5 Our Learned FoE in Comparison Our learned 3 3 FoE GSM −150 −75 0 75 150 [Roth & Black ’09] [Weiss & Freeman ’07] Student-t GSM −150 −75 0 75 150 −150 −75 Learned linear filters Much more peaked! Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 14 0 75 150 Generative Properties of our FoE ■ Filter statistics of our learned 3 3 FoE • Much better than previous models • Room for improvement −1 10 Natural Images KLD=0.08 MRF Samples −3 10 −5 10 −150 −75 0 75 150 −150 −75 0 75 150 Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 15 Image Denoising ■ Image denoising assuming i.i.d. Gaussian noise with known standard deviation σ Gaussian Likelihood p(x|y; ) N y; x, λ=1 2 Our 3 3 FoE I · p(x; ) Regularization weight optimal λ [Roth & Black ’09] MAP, optimal λ PSNR=29.18dB PSNR=30.06dB MAP PSNR=22.18dB PSNR=26.64dB Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 16 Image Denoising – MAP ■ Recent works point to deficiencies of MAP [Nikolova ’07, Woodford et al. ’09] ■ We find only modest correlation between: • Image quality of the MAP estimate • Generative quality of the MRF Better generative properties ! Better application performance Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 17 Image Denoising – MMSE ■ We propose to use Bayesian minimum mean squared error estimation (MMSE) Z 2 ˆ = arg min ||˜ x x x|| p(x|y; ) dx = E[x|y] ˜ x Samples average ■ [Levi ’09] extended sampler to the posterior • Only used a single sample in applications ■ We approximate the MMSE estimate • Average samples from the posterior ■ We find high correlation between: • Image quality of the MMSE estimate • Generative quality of the MRF Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 18 MMSE Image Denoising – Results ■ Compared the MMSE estimate for our learned models with other popular methods Average PSNR (dB) for 68 test images (σ = 25) 5 5 FoE [Roth & Black ’09] – MAP w/λ 27.44 Non-local means [Buades et al. ’05] pairwise (ours) – MMSE 27.50 27.54 27.86 5 5 FoE [Samuel & Tappen ’09] – MAP 27.95 3 3 FoE (ours) – MMSE 28.02 BLS-GSM [Portilla et al. ’03] – (MMSE) 26.5 27.0 27.5 Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 19 28.0 Advantages of the MMSE ■ Denoising performance highly correlated with the generative quality of the model ■ No regularization weight λ required to perform well ■ Denoised image does not exhibit incorrect statistics MAP MMSE Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 20 Advantages of the MMSE ■ Denoising performance highly correlated with the generative quality of the model ■ No regularization weight λ required to perform well ■ Denoised image does not exhibit incorrect statistics Derivative marginals • No piecewise constant regions −1 10 MAP MMSE −3 10 −100 0 100 Original images (noise-free) Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 20 Advantages of the MMSE ■ Denoising performance highly correlated with the generative quality of the model ■ No regularization weight λ required to perform well ■ Denoised image does not exhibit incorrect statistics Derivative marginals • No piecewise constant regions MAP KLD=1.64 −1 10 MAP MMSE −3 10 −100 0 100 [Woodford et al. ’09] Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 20 Advantages of the MMSE ■ Denoising performance highly correlated with the generative quality of the model ■ No regularization weight λ required to perform well ■ Denoised image does not exhibit incorrect statistics • No piecewise constant regions • Works with standard MRFs MAP MMSE Derivative marginals −1 10 MAP MMSE KLD=1.64 KLD=0.03 −3 10 −100 0 Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 20 100 Summary ■ Evaluated MRFs through their generative properties • Based on a flexible framework with an efficient sampler ■ Common image priors are surprisingly poor generative models ■ Learned better generative MRFs (pairwise & high-order) • Potentials more peaked ■ Sampling makes MMSE estimation practical • Several advantages over MAP • Excellent results from generative, application-neutral models Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 21 Thanks! Acknowledgments: Yair Weiss, Arjan Kuijper, Michael Goesele, Kegan Samuel, Marshall Tappen Please come to our poster! Code and models available soon at http://bit.ly/mmse-mrf Questions? Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 22 More Generative Properties KLD Random filters 10 −1 9 9 7 7 5 5 3 3 0.13 0.09 −3 10 0 −5 0 75 150 Natural images 1 −150 −75 0 75 150 Our pairwise MRF −150 −75 0 75 10 150 Our 3 3 FoE KLD −1 KLD 0.09 0.04 0 −3 10 2 0 10 −150 −75 Multiscale derivative filters KLD 0 −5 10 4 −150 −75 0 75 150 Natural images −150 −75 0 75 150 Our pairwise MRF −150 −75 Uwe Schmidt, Qi Gao, Stefan Roth: A Generative Perspective on MRFs in Low-Level Vision | CVPR 2010 | 26 0 75 150 Our 3 3 FoE
© Copyright 2024