Image Filtering
Slides, adapted from
Steve Seitz and Rick Szeliski, U.Washington
The power of blur
“All is Vanity” by
Charles Allen Gillbert
(1873-1929)
Harmon LD & Julesz B (1973)
The recognition of faces.
Scientific American 229(5):71–82
The power of blur
“All is Vanity” by
Charles Allen Gillbert
(1873-1929)
Harmon LD & Julesz B (1973)
The recognition of faces.
Scientific American 229(5):71–82
Image filtering applications
Today:
1. Noise removal
2. Blur
3. Image gradient
4. Edge detection
5. Image pyramids
Images as functions
• We can think of an image as a function, f: R2 -> R
• f( x, y ) = intensity at position/pixel ( x, y )
• Realistically, we expect the image only to be defined over
a rectangle, with a finite range:
• f: [a,b]x[c,d] -> [0,1]
• A color image is just three functions pasted together.
• We will focus mostly on intensity images.
Images as functions
What is a digital image
We usually work with digital (discrete) images:
• Sample the 2D space on a regular grid
• Quantize each sample (round to nearest integer)
What is a digital image
We usually work with digital (discrete) images:
• Sample the 2D space on a regular grid
• Quantize each sample (round to nearest integer)
The image can now be represented as a matrix of
integer values
Filtering noise
Image
Filtering noise
Image
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Filtering noise
Image
Mean image
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
10
20
30
30
30
20
10
0
0
0
90
90
90
90
90
0
0
0
20
40
60
60
60
40
20
0
0
0
90
90
90
90
90
0
0
0
30
60
90
90
90
60
30
0
0
0
90
90
90
90
90
0
0
0
30
50
80
80
90
60
30
0
0
0
90
0
90
90
90
0
0
0
30
50
80
80
90
60
30
0
0
0
90
90
90
90
90
0
0
0
20
30
50
50
60
40
20
0
0
0
0
0
0
0
0
0
0
10
20
30
30
30
30
20
10
0
0
90
0
0
0
0
0
0
0
10
10
10
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Filtering noise
Image
Mean image
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
10
20
30
30
30
20
10
0
0
0
90
90
90
90
90
0
0
0
20
40
60
60
60
40
20
0
0
0
90
90
90
90
90
0
0
0
30
60
90
90
90
60
30
0
0
0
90
90
90
90
90
0
0
0
30
50
80
80
90
60
30
0
0
0
90
0
90
90
90
0
0
0
30
50
80
80
90
60
30
0
0
0
90
90
90
90
90
0
0
0
20
30
50
50
60
40
20
0
0
0
0
0
0
0
0
0
0
10
20
30
30
30
30
20
10
0
0
90
0
0
0
0
0
0
0
10
10
10
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
What happens at boundaries?
Cross-correlation filtering
Let’s write this down as an equation. Assume the
averaging window is (2k+1)x(2k+1):
We can generalize this idea by allowing different weights for
different neighboring pixels:
Cross-correlation filtering
This is called a cross-correlation operation
H is called the “filter,” “kernel,” or “mask.”
Note: The above allows negative filter indices. When you implement need to use:
H[u+k,v+k] instead of H[u,v]
Mean kernel
What’s the kernel for a 3x3 mean filter?
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Mean kernel
What’s the kernel for a 3x3 mean filter?
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
9
1
1
1
1
1
1
1
1
1
Mean vs. Gaussian filtering
=
Blur!!!
Gaussian kernel
A Gaussian kernel gives less weight to pixels
further from the center of the window
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
2
1
2
4
2
1
2
1
Gaussian filtering
This kernel is an approximation of a Gaussian function:
What happens if you increase σ?
Mean vs. Gaussian filtering
Image filtering applications
Today:
1. Noise removal
2. Blur
3. Image gradient
4. Edge detection
5. Image pyramids
Image gradient as cross-correlation
How can we differentiate a digital image F[x,y] ?
– Option 1: reconstruct a continuous image, image, f,
then take gradient
– Option 2: take discrete derivative (finite difference)
Image gradient as cross-correlation
How can we differentiate a digital image F[x,y] ?
– Option 1: reconstruct a continuous image, image, f,
then take gradient
– Option 2: take discrete derivative (finite difference)
How would you implement this as a cross-correlation?
Image gradient as cross-correlation
How can we differentiate a digital image F[x,y] ?
– Option 1: reconstruct a continuous image, image, f,
then take gradient
– Option 2: take discrete derivative (finite difference)
How would you implement this as a cross-correlation?
Image gradient as cross-correlation
How can we differentiate a digital image F[x,y] ?
– Option 1: reconstruct a continuous image, image, f,
then take gradient
– Option 2: take discrete derivative (finite difference)
How would you implement this as a cross-correlation?
Forward
derivative
Image gradient as cross-correlation
How can we differentiate a digital image F[x,y] ?
– Option 1: reconstruct a continuous image, image, f,
then take gradient
– Option 2: take discrete derivative (finite difference)
How would you implement this as a cross-correlation?
Central
derivative
Image gradient
Image gradient
The gradient direction is given by:
how does this relate to the direction of the edge?
Image gradient
The gradient direction is given by:
how does this relate to the direction of the edge?
The edge strength is given by the gradient magnitude
Convolution
A convolution operation is a cross-correlation where the
filter is flipped both horizontally and vertically before being
applied to the image:
It is written:
Suppose H is a Gaussian or mean kernel. How does convolution
differ from cross-correlation?
Convolution
A convolution operation is a cross-correlation where the
filter is flipped both horizontally and vertically before being
applied to the image:
It is written:
Suppose H is a Gaussian or mean kernel. How does convolution
differ from cross-correlation?
Image filtering applications
Today:
1. Noise removal
2. Blur
3. Image gradient
4. Edge detection
5. Image pyramids
Effects of noise
Consider a single row or column of the image
• Plotting intensity as a function of position gives a signal
Where is the edge?
Solution: smooth first
Where is the edge? Look for peaks in
Derivative theorem of convolution
This saves us one operation:
Where is the edge?
Maxima point
Laplacian of Gaussian
Consider
Laplacian of Gaussian
operator
Where is the edge?
Zero-crossings of bottom graph
2D edge detection filters
Gaussian
derivative of Gaussian
is the Laplacian operator
Laplacian of Gaussian
Laplacian filter
Gaussian
delta function
Laplacian of Gaussian
(2nd derivative operator)
Laplacian image
smoothed (5x5 Gaussian)
delta function
Laplacian of Gaussian
(2nd derivative operator)
Laplacian image
smoothed (5x5 Gaussian)
original
Laplacian of Gaussian
(2nd derivative operator)
Laplacian image
smoothed (5x5 Gaussian)
original
Laplacian image
smoothed – original
(scaled by 4, offset +128)
Laplacian image
original
smoothed (5x5 Gaussian)
Laplacian image
What happens as we change σ ?
smoothed – original
(scaled by 4, offset +128)
Scale space (Witkin 83)
larger
Gaussian filtered signal
Scale space: insights
As the scale is increased
– edge position can change
– edges can disappear
– new edges are not created
Bottom line
– need to consider information at different scales
– (or else know what scale you care about)
Image filtering applications
Today:
1. Noise removal
2. Blur
3. Image gradient
4. Edge detection
5. Image pyramids
Image Pyramids
Image sub-sampling
1/8
1/4
Throw away every other row and column
to create a ½ size image
Image sub-sampling
1/2
1/4 (2x zoom)
Why does this look so bad?
1/8 (4x zoom)
Even worse for synthetic images
Really bad in video
Gaussian pre-filtering
G 1/8
G 1/4
Gaussian 1/2
Solution: filter the image, image, then subsample
In video: filter also temporally
Subsampling with Gaussian pre-filtering
Gaussian 1/2
Solution: filter the image,
G 1/4
G 1/8
Compare with...
1/2
1/4 (2x zoom)
1/8 (4x zoom)
Pyramid Creation
Octaves in the Spatial Domain
Lowpass Images
Bandpass Images
Pyramids
Advantages of pyramids
• Faster than Fourier transform
• Avoids “ringing” artifacts
Many applications
•
•
•
•
small images faster to process
good for multiresolution processing
compression
progressive transmission
Example: pyramid blending
Laplacian
level
4
Laplacian
level
2
Laplacian
level
0
left pyramid
right pyramid
blended pyramid
Pyramid Blending
Video Pyramids
Same idea in 3 dimensions x,y,t
Space-time pyramid:
Temporal pyramid:
Example: action recognition
Recognizing Actions
Use dense
information
Gradient orientation <--> space-time shape information
(behavior)
Gradient magnitude <--> photometric information
(clothing, lighting, contrast, etc)
Behavior-Based Distance Measure
Temporal pyramid:
Distance measure = the χ2 distance between
corresponding empirical distributions
Action recognition results
Next week:Features and descriptors