Download Report

Digital Communications III (ECE 154C)
Introduction to Coding and Information Theory
Tara Javidi
These lecture notes were originally developed by late Prof. J. K. Wolf.
UC San Diego
Spring 2014
1 / 26
Lossy Source Coding
• Lossy Coding
• Distortion
A/D & D/A
Scalar Quantization
Vector Quantization
Audio Compression
Coding With Distortion
2 / 26
Coding with Distortion
Lossy Source Coding
• Lossy Coding
• Distortion
A/D & D/A
Scalar Quantization
Vector Quantization
Audio Compression
Placeholder Figure
1
2
ǫ = lim
T →∞ T
Z
T /2
−T /2
A
E(x(t) − x
ˆ(t))2 dt = M.S.E.
3 / 26
Discrete-Time Signals
Lossy Source Coding
• Lossy Coding
• Distortion
A/D & D/A
If signals are bandlimited, one can sample at nyquist rate and
convert continuous-time problem to discrete-time problem. This
sampling is part of the A/D converter.
Scalar Quantization
Vector Quantization
Audio Compression
Placeholder Figure
A
m
1 X
2
E(xi − x
ˆ i )2
ǫ = lim
m→∞ m
i=1
4 / 26
Lossy Source Coding
A/D & D/A
• A/D
• D/A
• D/A
• D/A
Scalar Quantization
Vector Quantization
Audio Compression
A/D Conversion and D/A
Conversion
5 / 26
A/D Conversion
Lossy Source Coding
A/D & D/A
• A/D
• D/A
• D/A
• D/A
Scalar Quantization
Vector Quantization
Audio Compression
• Assume a random variable X which falls into the range
(Xmin , Xmax ).
• The goal is for X to be converted into k binary digits. Let
M = 2k .
• The usual A/D converter first subdivides the interval
(Xmin , Xmax ) into M equal sub-intervals.
• Subintervals are of width ∆ = (Xmax − Xmin )/M .
• Shown below for the case of k = 3 and M = 8.
Placeholder Figure
A
• We call the ith sub-interval, ℜi .
6 / 26
D/A Conversion
Lossy Source Coding
A/D & D/A
• A/D
• D/A
• D/A
• D/A
• Assume that if X falls in the region ℜi , i.e. x ∈ ℜi
ˆ = Y which
• D/A converter uses as an estimate of X , the value X
is the center of the ith region.
ˆ is
• The mean-squared error between X and X
Scalar Quantization
Vector Quantization
Audio Compression
ˆ 2] =
ǫ2 = E[(X − X)
Z
Xmax
Xmin
ˆ 2 fx (x)dx
(X − X)
where fx (x) is the probability density function of the random
variable X .
• Let fX|ℜi (x) be the conditional density function of X given that
X falls in the region ℜi . Then
ǫ2 =
M
X
i=1
P [x ∈ ℜi ]
Z
x∈ℜi
(x − yi )2 fx|ℜi (x)dx
7 / 26
D/A Conversion
Lossy Source Coding
A/D & D/A
• A/D
• D/A
• D/A
• D/A
Scalar Quantization
Vector Quantization
Audio Compression
• Note that for i = 1, 2, ..., M
M
X
i=1
P [x ∈ ℜi ] = ∆
and
Z
x∈ℜi
fX|ℜi (x)dx = 1.
• Make the further assumption that k is large enough so that
fX|ℜi (x) is a constant over the region ℜi .
1
• Then fX|ℜi (x) = ∆
for all i, and
Z
1
(x − yi ) fX|ℜi (x)dx =
∆
x∈ℜi
Z
1
=
∆
Z
2
b
a
b−a 2
)) dx
(x − (
2
∆
2
−∆
2
1 2
· ·
=
∆ 3
(x − 0)2 dx
∆
2
3
∆2
=
12
8 / 26
D/A Conversion
Lossy Source Coding
A/D & D/A
• A/D
• D/A
• D/A
• D/A
Scalar Quantization
Vector Quantization
Audio Compression
2
• In other words, ǫ =
M
X
i=1
∆2
∆2
P [x ∈ ℜi ] ·
=
12
12
• If X has variance σx2 , the signal-to-noise
ratio of the A to the D
∆2
2
σx / 12
(& D to A) converter is often defined as
• If Xmin is equal to −∞ and/or Xmax = +∞, then the last and
first intervals can be infinite in extent.
• However fx (x) is usually small enough in those intervals so that
the result is still approximately the same.
Placeholder Figure
A
9 / 26
Lossy Source Coding
A/D & D/A
Scalar Quantization
• Quantization
• Quantizer
• Optimal Quantizer
• Optimal Quantizater
• Optimal Quantizater
• Iterative Solution
• Comments
• Comments
SCALAR QUANTIZATION of
GAUSSIAN SAMPLES
Vector Quantization
Audio Compression
10 / 26
Scalar Quantization
Lossy Source Coding
A/D & D/A
Scalar Quantization
• Quantization
• Quantizer
• Optimal Quantizer
• Optimal Quantizater
• Optimal Quantizater
• Iterative Solution
• Comments
• Comments
Vector Quantization
Placeholder Figure
• ENCODER:
Audio Compression
• DECODER:
x ≤ −3b
−3b < x ≤ −2b
−2b ≤ x ≤ −b
−b ≤ x ≤ 0
000
001
010
011
A
000
001
010
011
−3.5b
−2.5b
−1.5b
−.5b
0<x<b
b < x ≤ 2b
2b < x ≤ 3b
3b < x
100
101
110
111
100
101
110
111
+.5b
+1.5b
+2.5b
+3.5b
11 / 26
Optimum Scalar Quantizer
Lossy Source Coding
A/D & D/A
Scalar Quantization
• Quantization
• Quantizer
• Optimal Quantizer
• Optimal Quantizater
• Optimal Quantizater
• Iterative Solution
• Comments
• Comments
• Let us construct boundaries bi , (b0 = −∞, bM = +∞ and
quantization symbols ai such that
bi−1 ≤ x < bi −→ x
ˆ = ai
i = 1, 2, ..., M
Vector Quantization
Placeholder Figure
A
Audio Compression
• The question is how to optimize {bi } and {ai } to minimize
distortion ǫ2
2
ǫ =
M Z
X
i=1
bi
bi−1
(x − ai )2 fx (x)dx
12 / 26
Optimum Scalar Quantizer
Lossy Source Coding
A/D & D/A
Scalar Quantization
• Quantization
• Quantizer
• Optimal Quantizer
• Optimal Quantizater
• Optimal Quantizater
• Iterative Solution
• Comments
• Comments
Vector Quantization
Audio Compression
•
To optimize ǫ2
=
PM R bi
i=1 bi−1 (x
− ai )2 fx (x)dx we take
derivatives and putting them equal to zero:
δǫ2
=0
δaj
δǫ2
=0
δbj
• And use Leibnitz’s Rule:
δ
δt
Z
b(t)
a(t)
δb(t)
f (x, t)dx =f (b(t), t)
δt
δa(t)
− f (a(t), t)
δt
Z b(t)
δ
+
f (x, t)dt
a(t) δt
13 / 26
Optimum Scalar Quantizer
Lossy Source Coding
δ
δbj
A/D & D/A
Scalar Quantization
• Quantization
• Quantizer
• Optimal Quantizer
• Optimal Quantizater
• Optimal Quantizater
• Iterative Solution
• Comments
• Comments
δ
δbj
Z
bj
M Z bi
X
i=1
bi−1
(x − ai )2 fx (x)dx
δ
(x − aj )2 fx (x)dx +
δbj
bj−1
Z
bj+1
bj
!
=
(x − aj+1 )2 fx (x)dx
= (bj − aj )2 fx (x)|x=bj − (bj − aj+1 )2 fx (x)|x=bj = 0
Vector Quantization
Audio Compression
b2j − 2aj bj + a2j = b2j − 2bj aj+1 + a2j+1
2bj (aj+1 − aj ) = a2j+1 − a2j
bj =
aj+1 +aj
2
(I)
14 / 26
Optimum Scalar Quantizer
Lossy Source Coding
A/D & D/A
Scalar Quantization
• Quantization
• Quantizer
• Optimal Quantizer
• Optimal Quantizater
• Optimal Quantizater
• Iterative Solution
• Comments
• Comments
δ
δaj
M Z
X
i=1
bi−1
bi (x − ai )2 fx (x)dx
aj
Z
bj
fx (x)dx =
bj−1
Vector Quantization
Audio Compression
aj =
R bj
!
Z
= −2
Z
bj
(x−aj )fx (x)dx =
bj−1
bj
xfx (x)dx
bj−1
bj−1 xfx (x)dx
R bj
bj−1 fx (x)dx
(II)
15 / 26
Optimum Scalar Quantizer
Lossy Source Coding
A/D & D/A
Scalar Quantization
• Quantization
• Quantizer
• Optimal Quantizer
• Optimal Quantizater
• Optimal Quantizater
• Iterative Solution
• Comments
• Comments
Vector Quantization
Audio Compression
• Note that the {bi } can be found from (I) once the {ai } is known.
◦ In fact, the {bi } are the midpoints of the {ai }.
• The {ai } can also be solved from (II) once the {bi } are known.
◦ The {ai } are centroids of the corresponding regions.
• Thus one can use a computer to iteratively solve for the {ai } and
the {bi }
1.
2.
3.
4.
One starts with an initial guess for the {bi }.
One uses (II) to solve for the {ai }.
One uses (I) to solve for the {bi }.
One repeats steps 2 and 3 until the {ai } and the {bi } ”stop
changing”.
16 / 26
Comments on Optimum Scalar Quantizer
Lossy Source Coding
A/D & D/A
1.
2.
Scalar Quantization
• Quantization
• Quantizer
• Optimal Quantizer
• Optimal Quantizater
• Optimal Quantizater
• Iterative Solution
• Comments
• Comments
Vector Quantization
This works for any fx (x)
If fx (x) only has a finite support one adjusts b0 &bM to be the
limits of the support.
Placeholder Figure
3.
One needs to know
β
X
A
fx (x)dx and
α
Audio Compression
4.
β
X
xfx (x)dx (true for
α
any fx (x))
For a Gaussian, we can integrate by parts or let y = x2
Z
β
α
1 2
1
√ e− 2 x dx = Q(β) − Q(α)
2π
Z
β
α
1 2
1
x √ e− 2 x dx = ...
2π
17 / 26
Comments on Optimum Scalar Quantizer
Lossy Source Coding
5.
A/D & D/A
Scalar Quantization
• Quantization
• Quantizer
• Optimal Quantizer
• Optimal Quantizater
• Optimal Quantizater
• Iterative Solution
• Comments
• Comments
6.
If M = 2a one could use a binary digits to represent the
quantized value. However since the quantized values are not
necessarily equally likely, one could use a H UFFMAN C ODE to
use fewer binary digits(on the average).
After the {ai } and {bi } are known, one computes ǫ2 from
2
ǫ =
M Z
X
i=1
Vector Quantization
Audio Compression
7.
For M = 2 and fx (x)
bi
bi−1
=√
(x − ai )2 fx (x)dx
1
2πσ 2
− 12
e
x2
σ2
we have
b0 = −∞, b1 = 0, b2 = +∞, and a2 = −a1 =
8.
Also easy to show that ǫ2 = (1 − π2 )σ 2 = .3634σ 2 .
r
2σ 2
π
18 / 26
Lossy Source Coding
A/D & D/A
Scalar Quantization
Vector Quantization
• Quantization
• Gaussain DMS
• Gaussain DMS
Audio Compression
Vector Quantization
19 / 26
Vector Quantization
Lossy Source Coding
A/D & D/A
Scalar Quantization
Vector Quantization
• Quantization
• Gaussain DMS
• Gaussain DMS
• One can achieve a smaller ǫ2 by quantizing several samples at a
time.
• We would then use regions in an m-dimensional space
Audio Compression
Placeholder Figure
A
• Shannon characterized this in terms of ”rate-distortion formula”
which tells us how small ǫ2 can be (m → ∞).
• For a Gaussian source with one binary digit per sample,
2
ǫ ≥
σ2
4
= 0.25σ 2
◦ This follows from the result on the next page.
◦ Contrast this with scalar case: ǫ2s = (1 − π2 )σ 2 = .3634σ 2 .
20 / 26
VQ: Discrete Memoryless Gaussian Source
Lossy Source Coding
A/D & D/A
Scalar Quantization
Vector Quantization
• Quantization
• Gaussain DMS
• Gaussain DMS
• Let source produce i.i.d. Gaussian samples X1 , X2 , ... where
1
e
fX (x) = √
2πσ 2
−1 x2
2 σ2
Audio Compression
• Let the source encoder produce a sequence of binary digits at a
rate of R binary digits/source symbol.
◦ In our previous terminology R = log M
ˆ1, X
ˆ 2 , ..., X
ˆ i , ...
• Let the source decoder produce the sequence X
ˆ i } is ǫ2 .
such that the mean-squared error between {Xi } and {X
n
1X
2
ˆ i )2 }
E{(Xi − X
ǫ =
n
i=1
21 / 26
VQ: Discrete Memoryless Gaussian Source
Lossy Source Coding
A/D & D/A
Scalar Quantization
Vector Quantization
• Quantization
• Gaussain DMS
• Gaussain DMS
Audio Compression
• Then one can prove that for any such system
1
σ2
R ≥ log2 ( 2 )
2
ǫ
for
ǫ2 ≤ σ 2
◦ Note that R = 0 for ǫ2 ≥ σ 2 . What does this mean?
◦ Note that for R = log M = 1,
σ2
σ2
1
1 ≥ log2 ( 2 ) ⇒ 2 ≥ log2 ( 2 )
2
ǫ
ǫ
σ2
⇒ 4 ≥ 2 ⇒ ǫ2 ≥ (1/4)σ 2
ǫ
• This is an example of “Rate-Distortion Theory.”
22 / 26
Lossy Source Coding
A/D & D/A
Scalar Quantization
Vector Quantization
Audio Compression
• MP3
• CD
• MPEG-1 Layer 3
Reduced Fidelity Audio
Compression
23 / 26
Reduced Fidelity Audio Compression
Lossy Source Coding
A/D & D/A
Scalar Quantization
Vector Quantization
• MP3 players use a form of audio compression called MPEG-1
Audio Layer 3.
Audio Compression
• MP3
• CD
• MPEG-1 Layer 3
• It takes advantage of a psycho-acoustic phenomena whereby
◦ a loud tone at one frequency “masks” the presence of softer
◦
tones at neighboring frequencies;
hence, these softer neighbouring tones need not be stored(or
transmitted).
• Compression efficiency of an audio compression scheme is
usually described by the encoded bit rate (prior to the
introduction of coding bits.)
24 / 26
Reduced Fidelity Audio Compression
Lossy Source Coding
A/D & D/A
Scalar Quantization
Vector Quantization
Audio Compression
• MP3
• CD
• MPEG-1 Layer 3
• The CD has a bit rate of (44.1 × 103 × 2 × 16) = 1.41 × 106
bits/second.
◦ The term 44.1 × 103 is the sampling rate which is
◦
◦
◦
approximately the Nyquist frequency of the audio to be
compressed.
The term 2 comes from the fact that there are two channels in
a stereo audio system.
The term 16 comes from the 16-bit (or 216 = 65536 level) A
to D converter.
Note that a slightly higher sampling rate 48 × 103
samples/second is used for a DAT recorder.
25 / 26
Reduced Fidelity Audio Compression
Lossy Source Coding
A/D & D/A
Scalar Quantization
Vector Quantization
Audio Compression
• MP3
• CD
• MPEG-1 Layer 3
• Different standards are used in MP3 players.
• Several bit rates are specified in the MPEG-1, Layer 3 standard.
◦ These are 32, 40, 48, 56, 64, 80, 96, 112, 128, 144, 160,
192, 224, 256 and 320 kilobits/sec.
◦ The sampling rates allowed are 32, 44.1 and 48 kiloHz but
the sampling rate of 44.1 × 103 Hz is almost always used.
• The basic idea behind the scheme is as follows.
◦ A block of 576 time domain samples are converted into 576
◦
◦
◦
frequency domain samples using a DFT.
The coefficients then modified using psycho-acoustic
principles.
The processed coefficients are then converted into a bit
stream using various schemes including Huffman Encoding.
The process is reversed at the receiver: bits −→ frequency
domain coefficients −→ time domain samples.
26 / 26