1. No category

Color Image Processing
Yu-Wing Tai
Preview
• Why use color in image processing?
– Color is a powerful descriptor
• Object identification and extraction
• eg. Face detection using skin colors
– Humans can distinguish thousands of color shades and
intensities
• c.f. Human distinguish only two dozen shades of grays
Preview (cont.)
• Two category of color image processing
– Full color processing
• Images are acquired from full-color sensor or
equipments
• E.g. camera, scanner
– Pseudo-color processing
• Color sensors and processing hardware are not
available
• Colors are assigned to a range of monochrome
intensities for visualization purpose
• E.g. MRI
Outline
• Color fundamentals
• Color models
• Pseudo-color image processing
Color fundamentals
• 1666, Isaac Newton
Visible light
• Chromatic light span the electromagnetic
spectrum (EM) from 400 to 700 nm
How human eyes sense light?
• 6~7M Cones are the sensors in the eye
• 3 principal sensing categories in eyes
• Most wavelengths will not stimulate only one type of cone cell,
because the spectral sensitivity curves of the three types of cone cells
overlap
Metamerism
• Two light sources made up of different
mixtures of various wavelengths. Such light
sources may appear to be the same color. This
effect is called metamerism. Such light sources
have the same apparent color to an observer
when they produce the same tristimulus
values, no matter what are the spectral power
distributions of the sources
Primary and secondary colors
• In 1931, CIE(International Commission on
Illumination) defines specific wavelength values
to the primary colors
– B = 435.8 nm, G = 546.1 nm, R = 700 nm
– Secondary colors: G+B=Cyan, R+G=Yellow,
R+B=Magenta
Primary colors of light v.s.
primary colors of pigments
• Primary color of pigments
– Color that subtracts or absorbs a primary color of
light and reflects or transmits the other two
Color of light:
Color of pigments:
R
absorb R
Cyan
G
absorb G
Magenta
B
absorb B
Yellow
Application of additive nature of light
colors
• Color TV
CIE XYZ model
The CIE XYZ color space encompasses all color
sensations that an average person can
experience. It serves as a standard reference
against which many other color spaces are
defined. A set of color-matching functions, like
the spectral sensitivity curves of the LMS space
but not restricted to be nonnegative
sensitivities, associates physically produced light
spectra with specific tristimulus values.
Tristimulus Values
• Once we select and fix any 3 primary SPD’s P1, P2 and P3, we
can express a color in terms of 3 tristimulus values
corresponding to the amount of the 3 primaries needed to
create the color
(i.e. color perceived
Color  (1 ,  2 ,  3 )
from physically adding

Color  1 P1   2 P2   3 P3 primary lights in
different amounts)
• Grassman’s Law
– human perception of color mixing is linear
– if ColorA  (1 A ,  2 A ,  3 A ), ColorB  (1B ,  2 B ,  3 B )
and ColorC  ColorA  ColorB
=>
ColorC  (1 A  1B ) P1  ( 2 A   2 B ) P2  ( 3 A   3 B ) P3
CIE XYZ model
• The concept of color can be divided into two
parts: brightness and chromaticity.
• Normalized tristimulus values
X
x
X Y  Z
Y
y
X Y  Z
Z
z
X Y  Z
=> x+y+z=1. Thus, x, y (chromaticity coordinate) is
enough to describe all colors
CIE xy Chromaticity Diagram
• The CIE xy chromaticity
diagram may be used to
show how all different
perceivable colors can
specified
• Note: this diagram is
reproduced using RGB
(with its gamut limitation),
so the colors you see are
only approximate and do
not correspond to actual
colors
By additivity of colors:
Any color inside the
triangle can be produced
by combinations of the
three initial colors
RGB gamut of
monitors
Color gamut of
printers
Note that the colors your screen displays in this image are specified using sRGB, so the
colors outside the sRGB gamut are not displayed properly.
Outline
•
•
•
•
•
•
Color fundamentals
Color models
Pseudo-color image processing
Basics of full-color image processing
Color transformations
Smoothing and sharpening
Color models
• Color model, color space, color system
– Specify colors in a standard way
– A coordinate system that each color is
represented by a single point
•
•
•
•
RGB model
CYM model
CYMK model
HSI model
Suitable for hardware or
applications
- match the human description
The RGB Color Base
• Express a color as a 3D point in RGB color space
– Color = (R, G, B), each R, G, B either in [0, 1] or [0, 255]
– Because there is a limit to the maximum amount of R, G
and B that can be added, we can also visualize this RGB
color space as an RGB color cube
Pixel depth
• Pixel depth: the number of bits used to
represent each pixel in RGB space
• Full-color image: 24-bit RGB color image
– (R, G, B) = (8 bits, 8 bits, 8 bits)
Luminance
• Luminance is a quantity defining
(approximately) the brightness by which
humans perceive different colors
– e.g. for RGB color base, a common way of computing
luminance Y is
Y  ( R  G  B) / 3
– However, human visual experiments show that a blue
light is perceived as much more dark than a red light,
and green light is the brightest (even if they all have
the same radiance)
– Based on experimental data, a more accurate
computation of luminance for phosphor RGB is
Y  0.2125 R  0.7154 G  0.0721 B
Chromaticity
• Tristimulus values encode color in terms of
3 numbers
– colors can have different levels of luminance
• However, we often want to compare color
attributes that are independent of
luminance
– These attributes are called chromaticity
– Colors with the same ratio of tristimulus values have
the same chromaticity
• e.g. Ca=(w1a, w2a, w3a) and Cb=(w1b, w2b, w3b) have the same
chromaticity if w1a:w2a:w3a = w1b:w2b:w3b
– Actual chromaticity coordinates are however defined
differently for different color systems
• since luminance is 1 variable, we can nevertheless expect that
there are 2 chromaticity variables
CMY model (+Black = CMYK)
• CMY: secondary colors of light, or primary
colors of pigments
• Used to generate hardcopy output
 C  1  R 
 M   1  G 
    
 Y  1  B 
600
700 nm
400
500
600
700 nm
400
500
600
700 nm
blue
magenta
500
green
400
yellow
red
cyan
Color names for cartoon spectra
400
500
600
700 nm
400
500
600
700 nm
400
500
600
700 nm
HSI color model
• Will you describe a color using its R, G, B
components?
• Human describe a color by its hue, saturation,
and brightness
– Hue : color attribute
– Saturation: purity of color
• no color-> 0, very colorful->1
– Brightness: achromatic notion of intensity
HSI color model (cont.)
• RGB -> HSI model
Colors on this triangle
Have the same hue
Intensity
line
saturation
HSI model
HSI component images
R,G,B
saturation
Hue
intensity
The HSI Color Models
 Converting colors from RGB to HSI

H 
360  
if B  G
if B  G
1


[(
R

G
)

(
R

B
)]

1 
2
  cos 
2
1/ 2 
[(
R

G
)

(
R

B
)(
G

B
)]




3
S  1
[min( R, G, B)]
( R  G  B)
1
I  ( R  G  B)
3
H.R. Pourreza
HSV: MATLAB Command
• RGB  HSV
– MATLAB: rgb2hsv(Red, Green, Blue);
• HSVRGB
– MATLAB: hsv2rgb(Hue, Saturation, Value);
31
Comments on HSI Color Base
• The HSI color base is very often used for image
editing
– Hue, saturation and intensity are intuitive
attributes that users want to manipulate
independently
• However HSI is not perceptually accurate
– intensity is computed as I = (R + G + B ) / 3 and
does not reflect accurate luminance
Color Enhancement via HSI Manipulation
• We can manipulate each component of hue,
saturation and intensity for all pixels
simultaneously
• Normally for image-editing applications:
1. Convert colors in RGB representation to HSI
2. Manipulate HSI components, typically by
•
•
Hue transformation involves adding a user-specified constant to
the hues of all pixels (equivalent to rotating chromaticity plane
about intensity axis)
Saturation and intensity transformation involves scaling these
values by a constant factor
3. Convert colors in HSI representation back to RGB
HSI Manipulation Examples
Hue
Saturation
Intensity
YIQ Color Space
• Y : luminance, brightness
• I, Q: chrominance (color information)
Y  0.299
 I   0.596
  
Q   0.211
 R  1.000
G   1.000
  
 B  1.000
0.114   R 
 0.274  0.322 G 
 0.523 0.312   B 
0.587
0.621  Y 
 0.272  0.647  I 
 1.106 1.703  Q 
0.956
35
YIQ: MATLAB Command
• RGB  YIQ
– MATLAB: rgb2ntsc(Red, Green, Blue);
• YIQRGB
– MATLAB: ntsc2rgb(Y, I, Q);
36
RGB Image VS YIQ Image
RGB Image
Y Image
I Image
Q Image
37
http://en.wikipedia.org/wiki/YIQ
Other Color Spaces
YUV/YIQ/YCbCr
• YUV
– PAL TV signal color space decomposition from RGB to Yluminance and UV-chromacity
YIQ
• YIQ
– NTSC (North America) TV signal color space decomposition
from RGB to Y-luminance and IQ-chromacity
– Note the Y projection of RGB is the same as YUV
YCbCr
• YCrCb is a color space used in digital video, including digital
encoding for standard TV, HDTV, and digital images (JPEG).
The transforms can be slightly different depending on the
signal and intended output device. The following is the one
example used by JPEG encoding of RGB.
0.587
0.114   R 
 Y   0   0.299
Cb  128   0.168736  0.331264
 G 
0
.
5
    
 
Cr  128  0.5
 0.418688  0.081312  B 
Adjustment translates Cb/Cr so the value is between 0-255
(assume RGB is 0-255)
Comparison
YUV
YIQ
YCbCr
Matlab
Color space conversions – see:
hsv2rgb
- Convert HSV values to RGB color space
ntsc2rgb
- Convert NTSC values to RGB color space.
rgb2hsv
- Convert RGB values to HSV color space
rgb2ntsc
- Convert RGB values to NTSC color space.
rgb2ycbcr
- Convert RGB values to YCBCR color space.
ycbcr2rgb
- Convert YCBCR values to RGB color space.
Which is the best color space?
• In research, it is often not clear which is the best colorspace to choose.
– If you need to work with grayscale, it is important you use the proper Y
conversion (not R+G+B/3)
• Sometimes it is nice to separate the luminance from the chrominance
– For example, histogram equalization. Often convert to YUV, equalize Y, then
convert back to RGB
• For some tasks (like segmentation), you often will not get different results
Example when writing a research paper:
“Our main algorithm X, works in RGB color space. For our experiments we
tried other color spaces, including HVS and YCbCr and obtained similar
results.”
Outline
•
•
•
•
•
•
Color fundamentals
Color models
Pseudo-color image processing
Basics of full-color image processing
Color transformations
Smoothing and sharpening
Pseudo-color image processing
• Assign colors to gray values based on a
specified criterion
• For human visualization and interpretation of
gray-scale events
• Intensity slicing
• Gray level to color transformations
Intensity slicing
• 3-D view of intensity image
Color 1
Color 2
Image plane
Intensity slicing (cont.)
• Alternative representation of intensity slicing
2D Illustration
Application 1
Radiation test pattern
8 color regions
* See the gradual gray-level changes
Application 2
X-ray image of a weld
Application 3
Rainfall statistics
Application 4
Washington D.C.
R
G
Near
Infrared
(sensitive
to biomass)
B
R+G+B
near-infrared+G+B
Useful Functions in Matlab
• colorbar
– http://www.mathworks.co.kr/kr/help/matlab/ref/
colorbar.html
• colormap
– http://www.mathworks.co.kr/kr/help/matlab/ref/
colormap.html
Color Transfer
• Make one image look like another
• For both images:
– Transfer to new color space
– Compute mean and standard deviation
along each color axis
• Shift and scale target image to have
same statistics as the source image
Lab Color Space
Convert RGB triplets to
LMS cone space
Take logarithm
Rotate axes
Why not use RGB space?
Input images
Output images
RGB
Lab
Why decorrelation is better?
• Apply different operations in different color
channels that without undesirable
crosschannel artifacts.
• The data in lab space is more compressed
(log space)
• RGB:correlations between the different
channel’s values. (0,0,0) - (255,255,255)
Steps
• Convert the color space to Lab
• Subtract the mean from the data points
• Scale the data points comprising the synthetic
image by factors determined by the respective
standard deviation
Color constancy correction
• add the averages computed for the
photagraph.
+ <lt>
+<at>
+<bt>
• Last step: convert the result back to RGB
Color Transfer Example
Color Transfer Example
Color Transfer Example
Color Harmonization
• Idea
Harmonization is the result of choosing colors that
are pleasing to humans
- Can we provide a way to do this for images?
- Retarget colors to be harmonized?
What is color harmony?
• Harmonic colors are pleasing to the eye.
• They engage the human observer and give a
sense of order and balance in the visual
experience.
[slides from Cohen-Or’s SIGGRAPH talk]
Formal definition of color harmony?
• Mathematical formulation has been developing
together with color theory
– Newton, Goethe, Young, Maxwell
• Itten [1960]: harmony means relationships on the
hue wheel:
2-color harmony:
complementary
colors
3-color harmony:
equilateral triangle
N-color harmony:
equilateral N-gon
Formal definition of color harmony?
• Matsuda [1995]: extensive empirical studies, derived
8 hue templates
• Tokumaru et al. [2002] developed a fuzzy system to
evaluate the harmony of color schemes
i type
V type
L type
I type
T type
Y type
X type
N type
Harmonic scheme
• The templates can be arbitrarily rotated
• Harmonic scheme is template type Tm +
specific orientation α
i type
V type
L type
I type
T type
Y type
X type
N type
Type “N” is not considered, this is grayscale images.
Harmony score
• To evaluate the harmony of an input image X
we analyze its hue histogram:
– Every pixel p contributes its saturation S(p) to the
bin of the hue H(p)
Harmony function
• The harmony of image X w. r. t. harmonic
scheme (Tm , α) :
F  X ,(Tm , a )    H ( p)  ETm(a ) ( p)  S ( p)
pX
H(p)
ETm(α)(p)
This term is
the closest edge
of the template
(oriented at angle
alpha)
Best template
• We compute α that minimizes F(X, (Tm , α))
for each template Tm using Brent’s algorithm
• The best-fitting harmonic scheme:
(Tm0 , α0 )  arg min F  X , (Tm , α) 
( m ,α)
So, given an image, they compute the best fit template from
the different types at the best orientation alpha.
Harmonization
• Given (Tm , α) we shift the hues so that the
hue histogram is contained in (Tm , α)
Color shifting
• The hue of pixel p is shifted to its associated sector ETm(α)(p)
• The amount of squeezing is controlled by a Gaussian fall-off
function
Color Coherency Problem
The problem, no way
to force neighboring pixels
to similar colors.
Here, similar colors (blue) move to
two different regions (green, purple).
Color coherency
• Another example
Problem!
This would be better!
Graph-cut optimization
• To make the coloring more coherent we assign
ETm(α)(p) by optimizing the labeling V
E (V )   E1 (V )  E2 (V )
Favors short
distance to the
template sector
[ We will talk about MRF formulation later. ]
Favors coherent
labeling of
neighboring pixels
Graph-cut optimization
• To make the coloring more coherent we assign
ETm(α)(p) by optimizing the labeling V
E (V )   E1 (V )  E2 (V )
E1 (V )   H ( p)  H (V ( p))  S ( p)
p
E2 (V ) 

{ p , q}N
 V ( p),V (q)   H ( p)  H (q)
1
 Smax ( p, q)
Results
Results – choosing colors
Results – cut and paste
• The background is harmonized according to the best-fitting
harmonic template of the pasted foreground
original
harmonized
harmonized
Results
• Text over a poster
Results
Images harmonized
to different “flags”
colors.
Find the flags template,
force the image to this.
Discussion
• Nature is already harmonic
original
bestfitting
template
poorlyfitting
template
Discussion
• Cannot improve good artwork!
Wassily Kandinsky, Composition VII, 1913
Discussion
• Grayish colors will remain such
Harmony Summary
• Provides a method to enhances the harmony
of colors in a given image
• Operates by fitting the image hues into a given
harmonic distribution
– Several different harmonic chooses are predefined (based on color theory)
• Especially useful for ‘artificial’ colors, cut-andpaste settings and collages that combine
imagery from different sources