1. No category

L
U
(NASA C R OR TMX O R A D H U M D E R )
(CATEi50RYI
JET PROPULSION LABORATORY
C A L I F O R N IIA
N S T I T U TO
EF T E C H N O L O G Y
PASADENA, CALIFORNIA
External Publication No. 206
"HE DESlCH OF OF'TIW LlNGAB SYSTRS
Eberhardt Rechtin
JEX PBOPULSION UB;)MTOBT
California Institute of Technolorn
Pasadem, miforrda'
April 1953
Sootion
I
IV
Su.bject
Introduction
2
Critcria for Optimization
4
Elomectary Calculus cf Variattons
9
Constrain+,?
15
Curva F i 5 t i n g and I'.!eig:?ting Functions
18
A.
B.
C.
D.
2.
Classical Sttitsmcnt of tho Cwvs F i t t i n 2 Problem
Citissicsl Solution ct thc Curve F i t t i n g F'roblem
Ektcnsion o f the Classical Solution
Xenn Squcrc E r r o r of t h c Z s - t i m t c d 2 e r a x t e r s
Solution by the Calculus o f Variations
Fourier and
'LF?.?~-?OC
Snectral E e i s t t p
IX
hg.
Traxsf o m s
18
20
21
23
24
29
44
s
57
59
XI1
63
Table of Contents, Cont'd.
XI11
M V
ZT
AT1
XVII
mn
4XJX
Zd
;.;M
A Physical Interpretatfon of the Wiener Optimum
&tensions
of Wiener l .s Solution
Abtched Filters
The Finite-time, Finite-order F i l t e r
Quasi-distortionless Notuozh
The Saturation Constraint
Transient E r r o r b!inirnization
The Dcsi,p of S e ~ o ~ c h ~ i i s ~
Re1 ntod Topics
Blbliopaphy
EP 204
-17
I
Criteria f o r
.
I
? h t c h e d Filters
n
TrtnsixS i3rrod
+
:;. 7 5 - 7 5 xetion
I
I
1
r--l-
1
I
I
.-_--
I
I
-2-
IN THODU
CTI ON
-----It is. t h e purpose o f this t e x t t o i n s t r u c t e n g i n e e r s in the design of
opttmm l i n e a r system.
dn optimum l i n e a r system is defined as an x % 5 a a e d s
l i n e a r system which performs a desired linear operation oriththe minimum mbm
square e r r o r -
The basio cause for the systom e r r o r is usually an external
disturbance of a s i q l a s t a t i s t i c a l nature,
by
a disturbance, .however;
System e r r o r need not be caused
the design technique is a l s o a p p l i c a b l e to tran-
s i e n t design of s y e t e m with l i m i t e d power c a p a b i l i t i e s .
The p r e s e n t a t i o n of t h e technique is d i r e c t e d toward t h e design and analysis of mobile-vehicle c o n t r o l systems, systems in which t h e c o n t r o l unit and
the element c o n t r o l l e d (the E h i c l e ) are independent e x c e p t f o r r a d i o or apt i c a l links,
Noise, d i s t u r b a n c e s o f m y kind; and d e l i b e r a t e enemy jamming
may e n t e r the system e a s i l y ,
The engineer's job is first,the d e s i @ of ths
best control system under the circumstances, and second, the analysis of p r formance data t o determine the degree of success achieved,
In a very real
sense, this ? a r t i c u l a r problem is t h e sum of many problems i n c l u d i n g f i l t e r i n g
s i g n a l s from noise, dssigninf: s t a b l e c o n t r o l systems, preventing internal
s a t u r a t i o n of a servomechanism, and analyzing n o i s y records,
Figure 1 in&-
c s t e s the o r d e r and r e l a t i o n s h i p of the t o p i c s which d e s c r i h the new
t ochique,
The technique and d e r i v a t i o n s a r e described in t e r m of Laplace t r a n s f o r m .
The a u t h o r has assumed t h e reader t o have a working knowledge of Laplace trans-
forms roughly e q u i v a l e n t t o that a t t a i n e d by reading Gardner and Barnes,
n
T r a n s i e n t s i n Linear Svstem."
No knowledge o f s t a t i s t i c s is assumed,
s t a t i s t i c s is p-esented i n t h i s development which cannot be learned in a
few ninutes.
30
-3-
Certain portion8 o f t h i s t e x t must be understood thoroughly i n order
that t h e engineer be able t o do more than plug In and g r i n d out t h e hmdamental equationr
The signifioanoe of the equation w i l l be r e a d i l y a p p r e c i a t e d
the develop=
ment proceeds, b u t i t is n o t always simple to formulate the problem at hand
in such a way a s t o make t h e equation upplicable.
the r e a d e r nust understand SectEans I
Regardless of background,
- X before reading f u r t h e r sections.
S e c t i o n s xf, XII, and XI11 present the minimum theory n e c e s s a r y t o solve
problems,
Section 1uv o u t l i n e s the remainder of the t e x t i n order t h a t the
r e a d e r may decide whioh f u r t h e r s e c t i o n s a r e d i r e c t l y r e l a t e d t o his h m d i -
ato problem.
The elrlstence of the technique is d i r e c t l y a t t r i b u t a b l e t o Norbert
Wiener, Professor of Xathematics a t M.I.T.
A,
8, Xo'Lmogoroff derived some-
w h a t s i m i l a r r e s u l t s b u t i n a less useful form.
N i t h few exceptions, all
extonsions of Wiener's work were done i n many places by c a y different people,
all a t about the same time.
The d e r i v a t i o n s i n this text a r e o r i g i n a l in the
sense of being independently derived a t J.P.L.
been w r i t t c n by s e v e r a l o t h e r engineer, at J.P.L.
and R, J, Parks i n i t i a t e d and guided t h e work.
T h i s t e x t could well have
I", H. Pickering, F. E- Lehaa,
C, W. B e r m , R . E, C o n ,
E.
Rcchtin, W, F, Sampson, R. 11. Stewart, and Do C, Youla continued the develop-
ment,
This t e x t i s one r e s u l t ,
-4-
Some amazingly v i o l s n t a r y m e n t e can ensue on t h e s u b j e o t of optimisation.
Because of the great v a r i e t y of p o s s i b l e systems, all performing
t h e same t a s k , it i a d i f f i c u l t t o get agreement even on the r e l a t i v e valuer
of cost, r e l i a b i l i t y , s i m p l i c i t y , and accuracy.
T h i s d i s o u s s i o n will be
l i m i t e d t o c o n e i d e r a t i o n s of accuracy for an extremely inportant reason
-
when i n t e r f e r e n c e is p r e s e n t , it is not possf-ble t o make a system as a c c u r a t e
as we please,
'Re ( p o s s i b l y ) oan i n c r e a s e c a s t u l t h o u t limit, and can n a b
a system as r e l i a b l e as desired, b u t we aro d e f i n i t e l y l i m i t e d i n the att a i n a b l e accuracy for t h e system.
The l i m i t i n g aocuracy of t h e system
depends p r i m a r i l y upon the c h a r a c t e r i s t i c s of t h e i n p u t f u n c t i o n s
speaking, upon the c h a r a c t e r i s t i c s of s i g n a l and noise
only by t h e optimum system.
--
loosely
-- and i s r e a l i z e d
If t h i s l i m i t e d accuracy i s i n s u f f i c i e n t for
our purposes, we must e i t h e r change t h e i n p u t c h a r a c t e r i s t i c s o r abandon
t h e attempt,
No m o u n t of c o s t , c o m p l e x i t y , or i n g e n u i t y can yield any
impr ovememt,
The e x i s t e n c e of an optimally-accurate system and a knowledge of i t s
a s s o c i a t e d " i r r e d u c i b l e error" is useful aven though such a system may
c a s i o n a l l y be too complicated, c o s t l y , and/or u n r e l i a b l e t o bLild.
00-
Such a
system provides a good standard f o r evaluation of proposed s u b s t i t u t e s .
It
i e o f t e n p o s s i b l e t o design s u b s t i t u t e s with a c c u r a c i e s within a f e w p e r c e n t
oi' t h e l i m i t i n g accuracy.
Let us define an optimum system as optimum i n
t h e r e i s much room for d i f f e r e n c e s of opinion.
M
accuracy sense.
Again,
For example, i f the problem
is t o t r a n s m i t information i n the prezsnce of noise, systems of d i f f e r e n t
r e l a t i v e acouraciea may be designed depending on t h e coding
dom) and upon t h e channel noiso level.
(AI:,
FIL, PCEL, ran-
If t h e information nay be quantized,the
number o f p r a o t i o a l oodes may be s t i l l f u r t h e r increased.
However, for eauh
t r a n s m i t t e d code and asaunod i n t o r f e r e n c e , there s t i l l e x i s t s an optimum re. oeiver design.
The problem becomes even more i n t o r e s t i n g if t h e i n t e r f e r e u o e
l a assumed i n t e l l i g e n t , i.e.,
i f t h e i n t e r f e r e n c o is always of t h e worst
p o s s i b l e type with r e s p e c t t o the seloctcd code,
This t e x t w i l l n o t c o n s i d e r
such problems in the t h e o r y of games, b u t w i l l be r e s t r i c t e d t o optimum systems
with r e s p e c t t o s p o c i f i e d inputs.
If a l l inputs t o a system
a r e c o i p l e t e l y s p e c i f i e d functions
c e p t for c e r t a i n p a r m e t e r s such as absolute m g n i t s d e or e x a c t
of time ex-
tiche
of occur-
ronce, e l i m i n a t i o n of t h e e f f e c t s of any of t h e i n d i v i d u a l i n p u t s may u s u a l l y
be accomplished i d e n t i s a l l y ,
i n t e r f e r e n c e elimination,
60 cycle "hum-bucking" is one example Qf complete
Output hum c o n s i s t s of a pure 60 cycle tone which is
e l i m i n a t e d by a d d i t i o n of an equal and opposite 60 c y c l e tone t o t h e output,
As
another i l l u s t r a t i o n , i f t h e b p u t s a r e e x a c t l y known except f o r t h r e e c o n s t a n t .
parameters, t h r e e operations are usually z u f f i c i e n t t o y i e l d e r r o r - f r o e performance,
If tho i n p u t s a r c s t a t i s t i c a l i n n a t u r e , however, t h e r e a r e so mny
u n h o v r n s t h a t exact scparations ai-o impossible,
me can only hope for good
perf orxnance averaged over many triese
It is i n p o r t a n t t h a t we l z o n bcforchar.d Ln what mi;. t h e v a r i o u s i n p u t s
d i f f e r fro= each othzr,
If a l l ir,puts evcer the system a t t h e same point c1
.12
a l l arc alike, separation is impossible,
f e r e n c e s , the b e t t e r t h o separation,
Conversely, the ' g r e a t e r t h o dif-
I n p u t s my be dcscribed by t h e i r e x a c t
time dependence, by t h e i r e q e c t o d time dependence, by t h e i r complete probab i l i t y functions, or by t h e i r c o r r c l a t t o n f u n c t i o n s (cr s p e c t r a l d e n s i t i c s ) .
Fs 204
S t a t i s t i c a l d e s c r i p t i o n s a r e comparatively weak, b u t they ray he t h e only des c r i p t i o n s available.
To quote Yiiener, " S t a t i s t i c a l p r e d i o t i o n i s e s s e n t i a l l y
a method of' r e f i n i n g a p r e d i c t i o n which would be p e r f e c t by i t s e l f ' i n an idea-
l i z e d c a m but whioh i: aorrupted by s t a t i s t i c a l errors, e i t h e r i n t h e observed
q u a n t i t y i t s e l f ' or i n t h e observation.
Geometrioal f a c t s must be p r e d i o t e d geo-
m e t r i c a l l y and a n a l y t i c a l f a c t s a n a l y t i c a l l y , 3eeving only s t a t i s t i c a l f a c t s t o be
p r e d i c t e d statistically.''
No complete the s p e c i f i c a t i o n of a n optimum system by agreeing on a nathem a t i c a l d e s c r i p t i o n of system accuracy.
If no r e s t r i c t i o n s a r e placed upon
e i t h e r t h e p r o b a b i l i t y f u n c t i o n s of t h e inputs, or on the type of operations perf o r n e d , ' i t is p o s s i b l e t o s p e c i f y t h e accuracy of t h e system i n many d i f f e r e n t
ways,
For example, u s i n g t h s "maximum likelihood" c r i t e r i o n , one a t t e m p t s t o
form t h e observed input by t h e a d d i t i o n o f samples s e l e c t s d f r o m h o r n d i s t r i b u t i o n s i n such n way t h a t t h e 3oint p r o b a b i l i t y of occurrence of the smples
is maximized,
The s e l e c t i o n process may w e l l be non-linear.
A l i n e a r operation
is describable by an equation of t h e ' f i r s t degree i n t h e dependent v a r i a b l e .
Such equations may bo time-variant,
i n t e g r o - d i f f e r e n t i a l o r d i f f e r e n c e , b u t may
n o t involve operations on ot'ner than t h e first. power of t h e dependent v a r i a b l e ,
Accuracy s p e c i f i c a t i o n s involving p r o b a b i l i t y f u n c t i o n s generally l e a d n o t to
e x p l i c i t s p e c i f i c a t i o n of the operation t o b6 performed, b u t only to c e r t a i n
c r i t e r 5 a on such operetio:is,
The f i e l d of non-linear mathecstics is n o t yet
in a condition t o 'be ext.ensively exploited by ongineers,
problem i s approached independently.
t h i s date.
Each non-linear
No g e n e r a l d i s c i p l i n e is a v a i l a b l e a t
If
we l i m i t ourselves t o linear systems,+ n o t only do we e n t e r a vrell-
developed field, b u t we can a l s o define system accuracy in a sinrple way.
Let
x(t) = d e s i r e d output at t i m s t
x o ( t ) = actual output a t t i m e t
* -
Lot
UO
(t) = x ( t )
- x,(t)
= error at timo t
r e p e a t . t h e experiment nnny times and observe the performance a t t i m t.
The performance w i l l be different for Gach experiment s i n c e the input i n t e r f e r e n c e s w i l l never be e x a c t l y a l i k e ,
Not knowing beforehand the exact time
dapendeuce of the i n t e r f e r e n c e , we can never guarantee p e r f e c t porfomance
(zero e r r o r ) for any one experiment.
After observing many sxperinlents, hmmr,
we c e r t a i n l y nmt to n o t i c e good o v e r a l l performance.
averaga.
F;e d e s i r e no e r r o r on the
L e t t i n g a bar s i g n i f y experimental (ensemble) overage, we require. the
condition:
Systems s a t i s f y i n g this condition are c a l l e d "unbiased."
LC
The condition i6 not
good measure of tompartitive performance of v a r i o u s systems,
tern will y i e l d
-
c(t)= 0
Almost any sys-
inasmuch as the average i n t e r f e r e n c e is u s u a l l y zero,
Let u s keep this zondition as
ti
desired f e e t u r e and i n v e s t i g a t e s e v e r a l addi-
t i o n a l masuros of s y s i c n accurecy.
Inasmuch as negative errors are ?robably
as s e r i o u s as p o s i t i v e e r r o r s , t h e measure should so i n d i c a t e .
*The r e s t r i c t i o n t o l i n e a r systems i s not a s s t r i n g e n t as i t might soom. Disturbances projuced by most j n t e r f e r c n c e phenomena are c h a r a c t e r i z e d by Gaussian
d i s t r i b u t i o n funciions. Any l i n e a r oparation on such disturbknces w i l i y i e l d
an output which a l s o has a Gaussian d i s t r i b u t i o n function. I f we r e s t r i c t our
d e s i r e d operations t c l i n s a r operations ( i , e . , d o not look f o r t h e square of
it h a s lxcn shown t h a t for Gaussian typo i n p u t n m c t i o n s the
t h e i n ~ u t ,ttc.),
optimum l i n s i r system i s a l s o t h e o p t . i m i of a l l systems, In a d d i t i o n , a l l
standard d e s c r i p t i o n s of accuracy reduce t o t h e one presented here.
Large errors are probably more serious t h m small errors.+
Two measures of
accuracy are of' particular i n t e r e s t :
c
Of t h e m , €2 ( t h e m a n squtlro or variance of [- ), is in wide
extensively developed, and leads t o linoar systems,
use, h a
The seoond,\Ei
boon
, is
ex-
tremeiy d i f f i c u l t rrsthematically, probably y i e l d s performance similar t o
and yet probably requires non-linear operations.
,
Thus our d e f i n i t i o n of an
optimum system:: AN OPTIMJII LICFM SYSTDJ IS DEFINZD AS AN UNBIASB), LINEAR
*It might be agreed, howwer, t h a t system design should ignore a fmtostlceKy
l a i g e error on t h a basis t h t if such M e r r o r does occur, t h a t particular e=parirrsnt i s worthless anyway, and should not be allowed t o influence our d_ec.:sion
-m an optinum systom. Such arAarpnent serves t o r e j e c t c r i t e r i a such as:"
? e
etco A system opern?ing w i t h an
c r i t e r i o n would tend to avcrage n o i s e
x a k s and disregard errors close 50 zero.
e
-9-
EF 204
I1 ELA?JENTARY CALCULUS OF VARIATIONS
The optimum 8ysmm has b-n
defined as t h a t system which minimizes a par-
t i o u l a r q u a n t i t y , the mean square e r r o r .
If our system is completely specified
except f o r one parameter suoh as t h e value of a p a r t i c u l a r r e s i s t o r or t h e gain
of a p a r t i c u l a r s e n 0 loop, the solution f o r an optimum system is q u i t e simple.
CT
We express t h o error
2
a8 a function of t h e v a r i a b l e parameter (r) and 1180
the u s u a l c a l c u l u s technique
t o find t h e minimum.
This equation s p o c i f i e s a s t a t i o n a r y ? o i n t only (maximum,
minimum, o r i n f l e c t i o n point) and t h e r e f o r e a check is u s u a l l y made of the sign
of
d"v2/ d k2
A p o s i t i v e sign i n d i c a t e s a t r u e minimum.
If the sys-
tern i s s p e c i f i e d except f o r p o s s i b l y f o u r o r f i v e parameters, the g e n e r a l prooedure is t h e same.
T r u e minimum can be r e a l i z e d proTiding t h e f o u r o r five
. simultaneous equations
can be solved properly.
It I s not g e n e r a l l y true, hcwever, that by making
more and more parametors v a r i a b l e we corn n e a r e r and n e a r e r t o t h e optimum
system.
The vory a c t of w r i t i n g the. error as a f u n c t i o n of c e r t a i n parameters
-
a l s o d e f i n e s t h o form of t h e system.
For example, c a l c u l a t i o n of t h e e f f e c t
of vorying the value of a given r e s i s t o r depends upon knowledge of t h e r e s t oftho c i r c u i t ,
I n p a r t i c u l a r , there i s no way of Icnoring which of t h e c i r c u i t s
below w i l l y i e l d t h e bsst p e r f o r m c e without l a b o r i o u s c a l c u l a t i o n .
?Lore
impartant, t h e r e i s no way o f knowing whether t h e b e s t c i r c u i t is even included
-19-
EP 204
i n t h i a set.
FIGURE 2
Exanples of Circuits w i t h an Adjustable Element
Even making a l l the elements * a r i a b l e will serFe no u s e f u l purpose:
There l a
no g u a r a d e e that the correct type element is l o c a t e d i n eaah branch.
I t i s probably n o t s u r p r i s i n g that a s o l u t i o a exists which not only giver
t h e form of t h e best c i r c u i t , but a l s o t h e v a l u e s o f all t h e elements in it.
It i s a l m o s t the e n g i n e e r ' s creed that nothing I s impossible,
It may be sur-
p r i s i n g (1) t h a t t h e concepts and techniques are q u i t e simple and ( 2 ) that most
of there coccepts and techniques a r e 250 years old.
Ths magic mathenatics is t h e c a l c u l u s of v a r i a t i o n s ,
I t s prime u s e s it
seem, have been t o prove t h a t t h e s h o r t e s t distance between two p o i n t s is a
s t r a i g h t line, o r t o derive equations of motion ~f a system i n p e c u l i a r
o r d i n a t e SyEtMm,
pliceted,
00-
For our purpozes both such problems a r e u n n e c e s s a r i l y com-
The b a s i c problem i n t h e c a l c u l u s of v a r i a t i o n s is t h e minimization
of an i n t e g r a l whose integrand i s some f u n c t i o n of t h e system,
As we s h a l l see
l a t e r , t h e mean square e r r o r of a system is e x p r e s s i b l e a s an integral whose
'integrand do;mds on thc s y s t ~ mtransfor funotion.
$'or the present, the exaot
way in whioh we obtain somo integral8 at3 are presented below i s n o t important.
what should H ( s ) be an e function of
Of 1rmdiot.s importance i s t h e questions
E
t o minimize tho value or th9 Integral?*
Tlm basic .logic o f ' t h e calculus of variations i s as follows:
1,
Assunme that the correct a m w e r i s %(e).
2.
Pick out any other function of
3,
Add t'((s) to
variable
E
H , ( V , but
. Th
only variable,
-
8,
%(s).
control the addition by a numerical
variable f, i s not a function of
however, since both ) J c ( S ) hnd
8.
4s ( 5 )
It i s the
are
-12-
presumed known.
4.
EP 204
Then w r i t e H(s) near Ho(s) ae:
Inveetigate t h e e f f e c t of ohanging
-
E on the integral of 5 n t e r e h
Since we have assumed that m know %(a)
a t least, p l o t the integral
and
l ( 8 )
we oould, forrrally,
(I) as.a funotion O f f
0
&
FIGURE 3
Value of t h e LIinimized I n t e g r a l as a Function of t h e V a r i a t i o n &
By hypothesis I for
E=O is
equal to the Tight answer
5.
Pick o u t many
t h e m i n i m I since H(s) for &SO
is
fb(6).
2") a n d p l o t t h e
r e s u l t i n g 1'8.
i
&
FIGUHE 4
V a r i a t i o n of t h e Idinimized Integral a s a W c t i o n of E,
andq!(S)
3.
IQalre t h i s oqimtinn
4.
If ncccssaq-, chcc!: tho si;n
trin
r e g a r d l e s s of
of
t!i3 sccnnd d e r i v a t i v e
&.ample:
Iio(s)
=
)2 (s)
+
I
8
2
d I/ d E
2
EP 204
-14-
The p o s i t i v e s i p y i e l d s a minimum v a l u e for t h e i n t e g r a l .
Problem 1 a:
-
Find t h e funotion V(s) which minimizes I
Problem 1 b:
-
Find t h e function H(s) which minimizes
3
in the text.
B
I
I 11 [dt:')]2]
=
+
A
Problem 2:
d s
I t ' i s a l s o p o s s i b l e t o solve normal c a l c u l u s minimization problems by t h e v a r i a t i o n a l method. The v a r i a t i o n a l method used i n
this way is somewhat like u s i n g a cannon t o k i l l a moth, but the
The only d i f f e r e n c e i n l o g i o i s that
use is titill legitimate.
12 is an a r b i t r a r y value r a t h o r than os a r b i t r a r y function.
Solve for a minimum of
y
= x2
- 2x
using tho v a r i a t i o n a l technique.
For t h e m a t h e m t i c a l philosophera among t h e r e a d e r s , t h e i n t e g r a l i s n o t
the only mathematical operation t o which t h i s technique i s applicable.
Any
o p e r a t i o n i s v a l i d which reduces tho performance of a system t o a single number.
B summation of d i s c r e t e values is thus l e g i t i m a t e , a s a r c such parameters
as j o i n t and marginal d i s t r i b u t i o n s i n p r o b a b i l i t y problems.
EP 204
-15-
IIJ COIJSTIWNTS
A coi13traint i s a r e s t r i c t i o n or condition imposod on the solution of a
problom.
Constraints i n optirni eation theory may e i t h e r reduce system per-
fomance or increaso it, dopending on the particular constraint.
Constraints
such as
.
1,
The available power i s limited a t oertain points within the system,
2.
The system m u s t be physically realizable, and
3.
Certain functions of t i m e (polynomiaIs or sine waves) mst pass
through t h c s y s t e m without d i s t o r t i o n
w i l l a l l chang
;sten performance from t h a t of t h e unconstrained system,
Constraints are er,countered on all l e v e l s of calculus.*
A typical
calculus p r o b l e m involvtng a constraint is:
T h a t i s t.he minimum value of
y =
q
1
- (x-1) 2
-
t2
uncier the condition
x + y + z
= 1 1
Of more direct interest t o us i s the analogous problem in i n t e g r a l form:
u d e r the constraint
1
*
S o k o h i k o f f , Advancod Calculus, page 327
etraint
1
-17-
EP 204
i n the same way that 11 was minimized alone.
The oonstants
I3 =
and
4
The r e s u l t of the minimization
are evaluated from the equations 12
= 2,o and
1,2.
Notice t h a t the minimization prooedure i s e y m e t r i o a l
13,
There i s no difference, therefore, between minimicing
T3 fixed, between minimizing I2 with Il and I fixed, and between minimizing
3
f
3 with 'I1 and I2 fixed.
This symmetry i s oocasionally useful in r e - s t a t i n g
a problem; sy-mm3t1-y will be used i n section XIX in p r e c i s e l y this way,
Problem 3
:
Find t h e H(s) which minimizes
c)
under t h e oonstraint
I2
Problem 4:
-
ds
0
ab)
Find the h ( t ) which minimizes
,"
cr2 --
-T
under the constraints
+T
h2W
-ti,
t
AIS
2 a-
I-
E;p
204
-18-
IV,
CURVE FITTING AND WEIGHTING FUNCTIONS
The idoa of optimizing system porfonnance is almost a s old a s the
One Qf t h e f i r s t problems
c a l c u l u s itself.
i n s t a t i s t i c s is t h a t of f i t t i n g
t h e b e s t a n a l y t i c a l curve t o a s e t of d a t a points.
This problem is worth a
detailed study because i t lends logically t o such concepts as weighting
f u n c t i o n s and matched f i l t e r s .
manner first.
Let us s o l v e the problem i n the c l a s s i c a l
Later we shall aolve t h e same problem using v a r i a t i o n a l
techniques under c o n s t r a i n t ,
Most of t h o b a s i c i d e a s of o p t i m i z a t i o n
theory will appear i n t h e course of t h e s e solutions.
A.
C l a s s i c a l Statement of t h e Curve F i t t i n e : Problem
Every optimization problem r e q u i r e s c e r t a i n a p r i o r i assumptions.
The assumptions in the c l a s s i c a l curve f i t t i n g problem are:
1.
A mechanism exists which i s producing a p e r f e c t , d i s t u r b a n c e - f r e e
function.
( A n o s c i l l a t o r is producing a p e r f e c t s i n e wqve,-
An o b j e c t
is moving under no external forces.)
2.
The form of this f u n c t i o n is
a r e unknown,
variable.
known, although c e r t a i n parameters
(Let t h e f u n c t i o n be x ( t ) where t i s the independent
Then some examples of f u n c t i o n a l form might be
x(t)
= a + b t
x(t)
= a sin ( t + b )
where the parameters a and b are t o be determined f r o m the data.)
3.
Our d i s c r e t e observations of x ( t ) are perturbed by a d i s t u r b a n c e
of S t a t i s t i c a l nature.
2
value and a 0 ,
I n p a r t i c u l a r t h e d i s t u r b a n c e has zero average
mean square value,
The s t a t i s t i c a l c h a r a c t e r i s t i c s - of
the d i s t u r b a n c e remain the same throughout t h e i n t e r v a l considered,
-39-
The c r i t e r i o n for optimization ia.the ninindzation of
4.
whero K i s the t o t a l number of observation points,
x(k)est.mate
i e t h e functional form with appropriately chosen parameters, and
represents t h e observed v81uea.
x
4
-
1
k
FIGURE 5
Typical Data P l o t
The standard presentation o f this curve f i t t i n g problem u s u a l l y mentions
a f i f t h assumption but seld >m adds an explanation.
The f i f t h assumption:
%he disturbance i s such t h a t a l l observations of it are independent.
This
assumption makes the overall f i e l d of d i s c r e t e s t a t i s t i c s considerrhly simpler,
but immediately precludes the extension t o t h e continuous case.
tinuous case, the samples are i n f i n i t e l y close t o g e t h e r .
Tn t h e con-
A statement t h a t
adjacent samples are s t i l l independent under t h e s e conditions i s n o t only
-20-
The most importmt
unbelievable, it also l e a d s to mathematical oxplosiona.
use of the independonce assumption i s in the proof t h a t t h e optimization
oriterion of assumptian 4 y i e l d s the b e s t possible discrete estimator i n the
ensemble mean square sense.- L e t us deliberately ignore t h e independenoe
assumption for the momnt and note the consequences lator.
The problem in c u m f i t t i n g can thus be stated as f o l l o n s .
xeatimate such that we minimito
four given assumptions, what i s the b e s t
B.
Under the
Classical Solution o f the Curve F i t t i n g Problem
For purposes of i l l u s t r a t i o n , l e t us assume that the d a t a points are
squally spaced and t h a t the assumed ilrnctional form i s
x(k)
= a + b k
rt
The estimated value of x(k) is given.by x
x(k)*
= a*
2
The mean square error
e2 =
+
b%
2L
is given by
,
+ b*k
a*
-
k = l
The mhim value o f which i s given by
a
and
62 = o
b bX
These operations y i e l d
K
o
=
7
dla*
+
k = l
b*k
X(k)data
3
EP 20&
-21-
K
and
t = i'
*
Solving f a r the parameters a* and b
C.
Fhtension of t h e C l a s s i c a l Solution
The above equations specifying t h e estimated values o f the parameters are
so complex i n appearance
Noting that
IC,
t h a t an extremely important concept is o f t e n missed.
2 k and z k 2
are all constants, we may r e - w r i t e the equations
as 8
K
k= 1
7)
k = l
*ere
Aa, Ba,*A,, and I& are constants dependent only on K,
The process of
e s t i m a t i n g the parameters thus consists i n weighting each d a t a p o i n t according
t o an appropriate weighting function L A
points.
-
B k]
and summing over a l l d a t a
I t i s i n t e r e s t i n g t o superimpose t h e weighting f u n c t i o n s on t h e d a t a ,
EP 204
- 22 -
Xdata
‘b
b+
0
f o r a*
ma
FIGURE 6
Weighting Functions
The s i m i l a r i t y in form between the weighting f u n c t i o n s and t h e f u n c t i o n a l
form of x(k) occurs whenever t h e assumed f o r n of x(k) i s
o. sum of f u n c t i o n s
with unknown c o e f f i c i e n t s (and tho n o i s e samples are independont).
Problem 5% Atsumo a form x ( k ) = a fljk) + ?I f Z ( k ) , where fl(k) and
fZ(k) a r e h o r n functions o f k, Show t h a t , as far a s t h i s
typo of optimization i s concerned, t h e weighting f’unctions f o r
a* and b* a r e of a s i m i l a r f u n c t i o n a l form.
Problem 6:
A t e x t i n s t a t i s t i c s c1cti.m t h a t t h o problem of e s t i m a t i n g a
and b .in t h e equation x ( k ) = a e-bk can be accomplished by
t a k i n g logarithms of both sides of t h c equation and t r e a t i n g
t h e system as l i n e a r . .Show t h a t t h i s process does n o t minimize
(Xestimate-x d a t a l 2 a s was required f o r t h e b e s t estimate.
Z
This s i m i l t i r i t y i n form of the weighting f u n c t i o n t o t h e assumed data f u n c t i o n
-a11 a;- -Aar f r e q u e n t l y i n l a t e r discussione,
The s i m i l a r i t y i s again e v i d e n t
EP 20k
i n matched f i l t e r networks, f i n i t o - o r d c r f i l t o r networks, and cross-correlat%on
de t o c ti on ne tvro r ke
D,
.
TJeon Square Error o f t h e E s t i m a t e d Paramtors
If tho d i s t u r b a n c e Froducing t h e d a t a s c a t t e r is s t r o n g ,
our ostimutes of nr.:.amotcrs
13 weak.
IVC
should expect
t o be much l e s s a c c u r a t e t h a n if t h e d i s t u r b a n c e
I n p a r t i c u l a r , if
58 is the
moan squnrs value of t h e data s c a t t e r ,
Wb is t h e weighting f u n c t i o n f o r t h e parameter b, and t h e d a t a s c a t t e r samples
are independent, then the w a n square e r r o r on b i s given by:
Proof:
Assume no s i g n a l x(k) present, only t h e noise d i s t u r b a n c e x
n
cstimate f o r x h e parameter b under t h e s o n o i s e c o n d i t i o n s is
The
If this experiment is pcrformcd many times, t h e average bn will bc zero
xn(lc) is zero. The mczn square v d n c of. b n is
k = 1
j = 1
Tho r i g h t hand s i d e o f t h - equation is d e l i b e r a t e l y written as t h e product
o f tvro sums i n d i f f e r e n t rnriablcs t o emphasize t h e o r d e r i n which t h e
The t r i c k is a common one and is u s e h l i n reoperations are performed.
w i t i n 6 t h e nean square value of bn as
2
b
n
-
9 [
d
k = l
mb(k)
zl
wb(j) x,(k)
x,(j)
3
Tho x ( k ) may be noved i n t o t h e j summation s i n c e it is a c o n s t a n t w i t h
r o s p e e t to j, The b r a c k e t is performed f i r s t , y i e l d i n g a f u n c t i o n of k
which i s m u l t i p l i e d by \Yb(k) and summed i n k. The averagixg q x r n t i o n
n f f e c t s only t h e xn's because t h e 11b ' 5 a r e t h e sane f o r ~ v c r yexpcrincnt.
(9)
EP 204
.
The avorngo product ~ ( k xn(
) j) is c a l l e d t h e a u t o - c o r r e l a t i o n function
of x
This c o r r e l a t i o n function v d l l be discussed i n d o t n i l i n l a t e r sections.
For ? h i s proof, it i s only necessary t o noto t h a t such an avcrngo
-product f o r
independent
x
1 s is zero wloss k = j
.
Providing
t
h
a
t
t
h
e
s
t
a
t
i
s t i o s do
--.n o t chi:ngo d u r f n p , t h e experimont o r between oxpcrimcnts, t h e mcan square d a t a
s c a t t e r w i l l be the same f o r a l l points k. I n o t h e r words,
2
constant independent
=
n = of choice of k.
",(SI
The m a n squaro e r r o r i n estimating t h e parrullzter b i n t h o presonce of indepord e n t noiue samples is t k . z givcn by
K
k = l
[
(kj2
Any equation of t h e form
b
=
k = l
Conversely, a l l l i n e a r operations may be w r i t t e n i n
2
t h e form of equation 10,
The c r r o r equation, 5 , t h u s a p p l i e s t o any
is l i n e a r in x(k),
b
l i n e a r o p e r a t i o 3 in t h e prcsonce of indepandcnt n o i s e samples of mean square
2
v a l u e CY
n
E.
.
Solution of t h e Curve F i t t i n g Problem
by t h e Calculus of V a r i a t i o n s
-
The c l a s s i c a l s o i u t i o n t o the curve f i t t i n g problem is an application
of the standard calculus.
Attempts a t extending t h e c l a s s i c a l technique,
however, u s u a l l y rosult i n such cumbersone m t h c m t i c s t h a t t h e b a s i c logic
is obscured.
s3.tuation.
The c a l c u l u s o f v a r i a t i o n s technique does much t o remedy the.
An understanding o f t h e c u r v e - f i t t i n g s o l u t i o n to f o l l o w w i l l
mean a n easy understanding of t h e TJiener d e r i v a t i o n and t h e various r e l a t e d
c o n s t r a i n t problem8 (f i n i t e - o r d e r f i l t e r s , matched f i l t e r s , povfcr level
con st ra in t s , and
3e rvomechani sm
o p t im i z8 ti on).
- 25 The assumptions i n t h i s v a r i a t i o n a l s o l u t i o n a r e q u i t e s i m i l a r t o the
c l a s s i c a l assumptions.
For v a r i e t y , l e t u s choose a a l i g h t l y more general
form for the assumed x(k).
1.
Assume x ( k )
x(k)
t o be of t h e form
=
a f&k)
+
b f2(k)
where estimate8 of a and b a r e desired.
2.
Assume t h e d i s t u r b a n c e t o be s t a t i e t i c a l i n n a t u r e with a mean
2 and
aquare value of
n
811
average of zero.
The s t a t i s t i c s of the
d i s t u r b a n c e a r e assumed t h e same throughout the experiment,
So
Assume t h a t the c r i t e r i o n f o r the b e s t e s t i m a t e i s the minimization
of t h e mean square e r r o r o f t h e estimated parameters,
A subtle differ-
ence e x i s t s between t h i s assumption and t h e e q u i v a l e n t c l a s s i c a l one,
The c l a s s i c a l c r i t e r i a n minimizes the mean square d i f f e r e n c e between
t h e d a t a arid t h e estimate over t h e i n t e r v a l ,
The c a l c u l u s of v a r i a t i o n s
c r i t e r i o n used h e r e minimizes t h e mean square d i f f e r e n c e between the
estimate and t h e c o r r e c t a n m e r over an en~sinble. AB w i l l be seen,
t h e s e two c r i t e r i a a r e equivalent only i f t h e n o i s e samples a r e independent,
I n this problem, since t h e form o f x(k) is l i n e a r i n a and b,
t h e weighting h c t i o n r e p r e s e n t a t i o n i s v a l i d
finear
-- i.8..
t h e system is
-- and hencer
a*
a
The expressions f o r mean square parameter e r r o r for independent n o i s e
samples are
-26-
If t h e noise samples are n o t independent, the expressions are
more oomplioated.
4,
(See equations 8 and 9)
We d e s i r e t h a t in t h e presenoe of no d i e t u r b a n c e . t h e
y i e l d a and t h e
wb , b.
wa
will
Thu8
not necessarily related3
.
fp3
fp
fpd
W
oonstraints
IV
constraints
f+)
These equations express mathematically t h e c o n s t r a i n t that a noise-
free system y i e l d t h e c o r r e c t answeram
The problem t o be solved is:
minimized under t h e
PD,
2
1, What i s ma(k) such that
constraintef
2.
What is Wb(k)
Ga 5s
such t h a t
6
minimized under thewb o o n s t r a i n t e ?
S o l u t i o n for Tia
t
and solve for,necessary X'S.
Va(k)
+ E
The minimization procedure:
Q (k) f o r W (k), d i f f e r e n t i a t e w i t h r e s p e c t ko
a
and make t h e r e s u l t zero independent of
z
substitute
& at & = 0
2
5s
EP 204
-27-
The ref ore
Wa(k)
=
[
1-
fl(k)
n
2G:
a r o evaluated by s u b s t i t u t i n g equation 18 i n t o thew,
The
The form of wa(k)
equation 15.
(18)
constraints i n
is again similar t o t h e assumed form of x(k)
and is e a s i l y shown t o be t h e same weighting f u n c t i o n d e r i v a b l e by t h e classic a l technique.
The s o l u t i o n for Vlb(k)is e x a c t l y t h e same except for sub-
scripts.
I t i s n o t s u r p r i s i n g t h a t t h e calculus of v a r i a t i m s technique y i e l d s
the same answer as t h e standard calculus technique under the same a s s u q t i o n s .
It should.
-
The v a r i a t i o n a l technique shows t h e approach t o t h e non-indepen-
d e n t problem, however.
I n s t e i d of t h e o r r o r expression
-2u-
7
Use of the bn
oxproosion in the o p t i m i z a t i o n process w i l l y i e l d a lTb(k)
which produces tho smallest mean square error.
solution y i e l d s an answer derivable f r o m
-
2
Inasmuch as the c l a s s i c a l
-z
6 n and not bn
result in the b e s t lib(k’for non-independent n o i s e samples.
,it can not
This f a & may
also be dnmonstraCed using more elegant s t a t i s t i c a l nrgumcnts.
‘It thus
hecones obligatory t o u s e t h e calculus of variations tochnique i n solving
tho continuous x ( t ) problem where t h e indopondcnco assumption i s n o t
ronlistic.
Problem 7: Vhot are tho values of A, and
f o r Tia(k) in equation 18?
Problem 0 :
2,
‘in tho dcrivo? o x ~ r c s s i o n
Vihnt average error and mean square error should bo
oxpcctcd in
f i n d i n e b where x(k) i s assumed of the form x(k)
a + h k and
whcro t e n o i s e consists of independent saryles of m a n squnrc:
9
value Gn m d average value zero?
EP 2oG
- 29 Y.
FOURIER AND LAPLACE TRANSFORNS
&
_
I
The two-sided F o u r i e r transform i s a u s e f u l t o o l i n o e r t a i n sections
optimization theory.
Of
This transform is c l o s e l y r e l a t e d t o , and more restrio-
t i v b than, t h e more f a m i l i a r one-sided Laplnce transform+ defined by
-
F(s)
=
m
J-
-st
e
f(t)
0
dt
+jF+
0
where f ( t ) is t h e time f u n c t i o n t o be transformed, s is the so-called canplex frequency, and c is a real number ( o f t e n zero) used to guarantee
i n t e g r a l convergence.
Using t h e s u b s t i t u t i o n s = jw i n t h e conventional
d e f i n i t i o n s of the two-sided F o u r i e r transform shows its s i m i l a r i t y t o the
Laplace t r a n s f o m .
-3The absenoe of t h e oonvergenoe number, c r e s t r i c t s the Fourier transform for
our purposes t o f u n c t i o n s which s a t i s e
*Gardner & Barnes, "Transients i n Linear Systems."
The transform is called
one-sided b'ecause only the p o s i t i v e time region is considered. A two-sided
transform considers behavior i n both p o s i t i v e and m g a t i v e time.
-jL,
2
dt
0
ktl
Thus f w o t i a n s suoh as e
and c o s @ * have no Fourier transform.
Pro-
viding the r e s t r i c t i o n s are met on f(t), however, t a b l e s of Fotlrier trans-
forms are easily created f r o m t a b l c s of Laplace transforms.
From t h e
defining intolYa1s. if 21{s) is t h c one-sided Laplkce transform of f(t)
from t = 0 t o t = +
and if F2(s) i s t h e one-sitlod Laplace transform o f
f(t) going f r o m t = 0 t o t =
-W
,then
0
ti-
Tim Functions
-.
Problem 9:
-
Problem 10:
Find the two-sided Fourier t r a n s f o r m o f
-dl tl
e
and e
-4tf
COS^^
N
Prove t h a t
& (s) =
F1(s) + Fz(-~) as s t a t e d in t h e t e x t .
The r e s t r i c t i o n on the Fourier transform provides an i n t e r e s t i n g and
- 31 u s o f u l mons of dettrrndning whether f ( t ) occurred i n +t o r -t.
This infor-
maticn js highly inportant i n d i s c u o s i ~ gthe r e a l i z a b i l i t y of networks.
For
exar;~;?lo:given
=
%SI
1
d.0
__I_-
& - a
it m i ~ h tappear that two alternatives are possible.
1-
f.W
I
0 for t
2.
fZ(t)
e
uct
eo,
-4 It1
for t
for t
e
<
0,
0 for t
>
0
>0
A- /
I
0
Time
Figure 8
Alternative Time Functions
Only the sccor,d eltsrnativc s e t i s f i e s t h e c o n d i t i o n for existence of the
Fourier transf o m .
For
,!IC!
f i r s t alternative
an8 henco the Fourier transform does not exist.
concept t o show t h a t
x-transforms of t h e type
V e might generalize t h i s
EP 204
qescribe time f u n c t i o n s in j o s i t i v - time ocly ax~d t h a t ~ - t r ~ c s f n r - nafs th.!
-
d c r;c r ibe no ga t j ve t irn- f1I n c t i 3r, Y.
- 1
-<..-
I s s a t i s f i e d , t h e P m r i e r t r a n s f o m . i s :.aid to be f a c t 2 r n b l e :
t h e fclcforablc
t r a n s f o r m can b.r written as
(-9)
As w i l l
can .be w r i t t e n by s u b s t i t u t i n g
(-s)
e v n r y x h e r e f>r(($s)i n
(s)..
b:? seexi i n l v t + r s e c t i ? r i s , t h e :;i?ner o g t i m u r r soiution .Ie,en~lsu:cxi bin(:
able t o factor a Fourier t r u n s f w n into its t-.o berts
BY
The f a c t o r i z a t i o n t h e e r e m also hss a p r a c t i c a l s i g i f i c a n c e w i t h r.-sir'.:t
netvjorks.
25.
. 3 e f i n s j in cjku9ticm
ti,
rnalizable
R e a l i z a b l e networks have an i m p l s i v - r e s s n s e h ( t ) of z e m fcr t L 0
and c o n s e q u e n t l y will h a v e a transform H ( s ) in p s i t i v e s only.
of the n e t w o r k is I l l ( s ) /
a n t i t s a t t e n u a t i o n log(H(s!/
2
.
Th*
8nliJitli-k
The cTit.eriorA ? h b s
s t a t e s t h a t no realizable netw,-l:-k can have i n f i n i t e a t t r n u c t i x .
LH(Y~)-
3 i
c
x-q,on.se
over any f i n i t c frequcncy band.
-d{t I
J
Fxanple:
Factor the
($8)
for
f(t) =
e
which f a c t o r s i n t o
Froblcm 11:
-
Factor t h e
x(S>
for
-d
f(t) = e
Itl
cos
p
t
Both t h e Laplace and t h e F o u r i e r t.ransforms arc nathcmatical a i d s t o
1
simplif'y the s o l u t i o n of time-invari a n t systems,
,
Such transforms are ,rela-
t i v e l y worthless, howevcr, i n t r e a t i n g systems which v a r y with t i m e , *
A
complete treatment o f thc o p t i m i z a t i o n of l i n e a r systems c e r t a i n l y should
include the time-variant case,
Rather than develoy? an a l l - i n c l u s i v e body
of mathematics capable of s o l v i n g both t h e v a r i a n t and non-variant cases,
There are good
however, vre shall d e v e l o p t h e non-variant s o l u t i o n only.
m a s o n s for t h i s apprcach.
The only r e a l d i f f e r e n c e between t h e a l l - i n c l u s i v e
t h e o r y and t h e time-variant theory is t h e l e v e l of mathematical labor.
b a s i c s o l u t i o n s f o r optimum systems a r e i d e n t i c a l i n form.
The
I t is possible
t o c a r r y t h e i n v a r i a n t case through thc d s s i g n s t a g e , however, while t h e
--__
* Typical
~
I
-
--
systems a r e thosc i n which th- values of R's,
t time.
h
L's,
and C ' s vary
EP 20k
-34v a r i a n t case I s blocked by a formidable I n t e g r a l equation which, i n general,
Inmsmch
813 aerobywnicfrta
and guidance engineers are often faced
w i t h time-variant aydtem, 8 r s p l d develoymant should occur of the necereory
mathensatlcal aids t o reduce t h e labor of t h - v a r i a n t eystern design.
h
r e f e r e n c e of v a l u e ' i s "hanshrma for Linear The-Varying Syetcms. by John
A. A s e l t i n e , a thesis submitted a t
Februa.ry 199,
U.C.L.A.,
Department of Engineerin& in
Tbe t h e e i s Is primarily a discussion of the use of other
types of transforms than Laplace and Fourier,
Bessel and Cauchy transforms
are discussed in d e t a i l ; methods of reducing v a r i a n t s y s t e m s t o Invariant
systems by s u b s t i t u t i o n are mentioned.
Problem 12: To show how time-variant equations may occasionally be reduced
t o i n v a r i a n t equations, reduce t h e following e q u a t i n to a
c o n s t a n t c o e f f i c i e n t equation by s u b s t i t u t i n g u e 0
e
au2&
du2
+
b u a
du
+ e x
-g(u)
In this equation x is t h e dependent v a r i a b l e ; u i s
t h e inde-
pendent variable; a, b, and c are c o n s t a n t s ; and g(u) i n a
d r i v i n g function.
E l c c t r i c n l onSinoors h a ~ : long b c m f a n i l i a r v d t h t h e use of t r a n s f e r
fiuictions f o r d e s c r i p t i o n of mtwork performance.
‘In a l i n c a r i n v a r i a n t
systam t h o t r a n s f e r f u n c t i o n r o l a t c s t h e output of t h e network t o its i n p u t by moans of anpropriate Fourier or Laplace transforms.
If F(s) ropro-
s o n t s t h e transform of t h e input, N ( s ) r c p r c s s n t s t h e t r a n s f e r function,
and C(s) reprc-scnts. t h e transform cf tho output, t h a n
C(s)
= H(s) F(s)
This zamc r o l a t i o n s h i p m y bc w r i t t e n i n t h e time domain* using the supcr-
.
poa i ti on i n t c era1
where g, h, and f a r e t h e Laplace transform of G, H, and F, r e s p e c t i v e l y ,
Thc relationships arc s i m i l a r using two-sided Fourier transforms:
*
Gardner and B a r n o s pages 228-236,
3ccomcnded reading,
- 36 +
"
,
Proof;
The t i m e domain representation is more fundomntal than a repreaontntion i n
t h e complex frequency donain.
Tina-variznt linear s y s t e m , for cxaxple,
are
described by making the function h time-variable.
The superposition i n t e g r a l i s also useful as t h e c l e a r e s t way of v i s u a l i z i n g
the cross-correlation operation cssd i n detection of signals of known form.
If t h e integral i s re-written as
a sum of d i s c r e t e sampling operations, the
r e s u l t i n g equation is recognizable as a weighting func'Aon equation similar
t o thosc given in t h e section on c'x-ve f i t t i n g .
x
c5q
=
d(K)
h(1:)
- k)
f(lC
/__I
k = l
b
--
(cquntion 10, s e c t i o n f V C)
Ylb(k) x(k)
k = l
Tlic physical significuiice of the superposition i n t a g r a l may be v i s u a l i z e d
is several ways.
If tlic i n p u t m v c is p l o t t e d on
d o m t h e record z i t t h e sailte rate.
a t i m c scale as it a r r i v e s ,
The o u t p u t is the i n t c g r n t F d product of
c?te trio ! ) i D t s a t e a c h i n s t a n t ,
prebent
i ns t a n t
ti=
I
-----
o u t p a t g( t)
.\ m
, .-.
.-
--\
_._
e
time
_---
/ @ \
/-
-I
+-
F'IGUHE 9
Graphical Ficture of S u p e r p a s i t i on I n t e g r a t i o n
Froin t h e s u p e r p o s i t i o n i n t e g r a l it is evident t h a t h ( t ) is the rcsponsc cf
t.he iietbvvork to an impulse.
Thus t h e s u p ~ r f . o s i t on
. j i n t e g r s l m i c l i t also be
v i s u a l i z e d as the(presont) sum of res;'onses
to t h s ( 1 ) a s t ) impulses d e f i n i r . g f ( t ) .
Q
t
1
204
picsent
instant
TIKE
FIGURE 10
Graphical P i c t u r o of S u p r p o s i t i o n Integration
These v i s u a l i z a t i o n s i l l u a t r n t c t h c fcct tht t h e ’ nctwork h ( T ) detsrini nes
the present relxtive i n p o r t a c c of t h = i n p t s i g n a l
t h i s sciisc, h(
r
Z
seconds b e f w c .
) i s a weizhtad ncniory.
The frc;cilom w i t h which w e m y choose h( Z ) is l i n i i t c d i n
cases.
I n a phjrsically-rcalizahle system, i t is
j R;?os:ii
t o hero a ~ . e m o r yTor cvcnts -i&,ich have nc;t y c t ha;>;.cncd.
c
..rtnin
b l r for t h n system
Thus, for
-
phy s i c a 1l y res 1i zab 1e networks :
T h i s r u s t r i c t i o n is not always a;r)licable,
d a m w i t h res?e-+, t o t h e i-itcrval
and fcture,
In
‘r!ei,htin;;
gf
I n d a t a rediiction work, a l l
intc;rc.,t iz a v a i l a l i l e , b o t h p a s t
fiuictiojla urtondin;; i n both 2 i r e c t i o n s i n time
ure t h e r e f a r e adnh s s i b l a .
T
0
Problwn 15:
-_.--I-
Sketch the aci ghtin,; f u n c t i o n s for t h e follovrimg:
3
.
An intcgrator
bo A pure tim--d313y
~ ( s =
)
l/(d + s)
c.
d.
~n im;)crfect i n t e g r a t o r
A dirf~rsntiatar
e-
~uie x t r c r d y iiurPotv-3uni f i l t e r b a n d - c e n t s r c d
Prob1ol.n 16: Dcinnns? r:&e
thc intcgrG1 equation
--t r a n s f o r m 3 w-lcsr;f(t< 0) = 0.
wh;r
a t foe
b e l o w cannot be Laplace
+r*J
0
Su;,,c3Cic'q:
Trx- it, u s i c b t h z samc? stc?,s 2 s given in the
k - ' . , r m s ? o r n p r o o f of cqi:;?tion 27, Keep c l o s e
t r t c k of t h e l i m i t s .
Yctr::
This particular intzgral cquxtion w i l l appear
i a t o r i n s c c t i o n X I 1 E w i t h t h c conditions
;1> t h e t f ( t c ' 0 ) # 0 and ( 2 ) the e q u a t i o n
holds f o r t> 0 only.
The s o l u t i o n of such m
integral c q x t i o n i s fairly tricky.
- 40 -
The conccpt of indep~~nrlent
phononsnn i s a u s i l y dcscribcd and i u d c r s t u d
THO d i f f c r o n t randoa 1101scgonorators arc undoubtcdly indopttiidenk.
The
s t a t i c i n r a d i o roception is probably independent of t h o broadcast 3 i J l n l .
Related phcnorrlcna (ire also c o s i l y conceived, a1thoul;h t h c p r t i c u l s r
way i n which relatedness
is mrsurcd micht 3ccm a r h l t r u r y ,
Lot f ( t ) bc
of rclatcdnsss is ohoscn t o bc t h e c o r r a l a t i o n f'unct;on.
c o q r r 2 d i n experiment after oxperimsnt, always at; a t i m e
-
exporiincn-t; s t a r t s , we -define t h c c o r r e l a t i o n f u n c t i o n
Eie bar silnifics t h e avcrai;a o v w
iilrtiiy
one
If t.!irs.-! f m c t i m s arc!
f'unction of timo and g ( t ) mother f u i c t i o i i of tine.
recult of averaging f(rJ g(to) o x r
Tbo w n s i w e
(p
f
to after thc:
(to) to
l:
hn
thz
exprirnontc.
If f(t) aiid ~ ( t )
!.IXI~e x 2 m i m n t s .
r e p c a t thcmsclves i d e n t i c a l l y i n each experiment, t h e i i v ( : n g i n & p r o ~ c s . 3
is unnecessary.
On the otlicr hand, i f f ( t ) tu12 g(t) a r c
;tztiStiCiil
in
n n t u r c from expcriimnt t o e x p d n c u t , t h e rrvcraLing process is o f p i n u
i q )ortan c c
.
The m n j o r rccson f o r maswing relatcdncsz by the c o r r ~ l l n t i o i ifiuiction
is t h a t %he c o r r c l a t i o n function ( o r i t s F o u r i c s t r u n s f ' o r x ) i s t!ic
w - :nn4
only necessary f m c t i o n i n dcrivirig the optimum system.
Instead o f comparing f ( t ) sild i;(t)b o t 3 at to,
a t to+ 2 with g ( t ) at t,.
a40
might coi:;isr.> f ( t )
The c o r r c l a t i o n function then hucorws
*The order of s u b s c r i p t s follows Wiener's convention.
Lab. 25 uses reverse order, gf.
Pkilliys i n hh.1.T.
EP 204
-4
1-
-
This p a r t i c u l a r c o r r e l a t i o n function is c a l l e d t h e ---cross-corrclntion
.
function of f on g
of f(t,
+ r )w i t h
Another comparison vm might imke is a comparison
f(t,). The r e s u l t is tho auto-correlntion function of f-
The c o r r e l a t i o n botween a f u n c t i o n and i t s value
7 seconds away might well
be expoctod t o p l a y an important p a r t i n system design.
inentioncd as h-ving weizhted meimri (3s.
Systems have been
A s i p n l auto-correlation f'unction
i n d i c a t e s a slowly oclr:Jing s i g n a l .
decreasing slowly with
Such s i p n l s
aro more e a s i l y c x t r a c t c d from noise than t h o s e w i t h auto-correlation
f u n c t i o n s d e c r e a s i n g r a p i d l y w i t h tine.*
the degree of r e l a t e d n e s s ,
The c o r r o l a t i o n f u n c t i o n measures
The a u t o - c o r r e l a t i o n f u n c t i o n measures t h e amount
t h a t a signal h%gs t o g e t h e r i n t i m e and hence s p e c i f i e s how mich o f its pabL
h i s t o r y i s worth remcmbering.
The behavior of most s t a t i s t i c a l phenomena is such t h a t t h e averaged
performance i s t h e same r e g a r d l e s s of t h e p a r t i c u l a r instant t
0
t h o observation,
The behavior i n t i 3 of such phenomena a r e c a l l e d
s t a t i o n a r y time series.
is stationary.
chosen for
---
Notice t h a t it is t h e overage performance t h a t
I n s t a t i o n a r y time s e r i e s , t h e c o r r e l a t i o n f u n c t i o n s do
n o t depend upon to.
'Pff(*C)
*
.
=
f(t +
Z)f(t)
-
Noise i s u s u a l l y characterized by a u t c - c o r r e l a t i o n h c t i ons decreasing
r n p i d l y with 7
The most d i f f i c u l t s i g n a l s t o e x t r a c t f r o m n o i s e a r e
t h o s e i n which t h e s i g n a l a u t o - c o r r e l a t i o n c l o s e l y r e s c i ~ ~ b l et h
s e noise
auto-correlation.
S t a t i o n a r y timc s e r i e s are cndowcd with another propcrty _by
- hypothosjs.
-- --A
The hE)othosis s t a t o s t h a t it nakea n o d i f f e r c n c e ~ r h c t h c rthc averaj;c is
c a r r i e d out a t to f o r inany o x p o r i m n t s , or whother t h o averago is c a r r i e d
o u t . b y using observations at d i f f o r c n t timcs in a s i n g l o experiincnt.
Tho
ergodic hypothesis can not be proved, i t can only be j u s t i f i e d by cxyeriIneni;al
results.. Some j u s t i f i c a t i o n f o r t h e hypothesis might be found, however, by
m a s o n i n g tbxit because the t i m s e r i e s is always prcse:it
(we just a r e n ' t
watchins it) and because t h e t i m e of s t a r t of t h o experiment is a r b i t r a r y ,
any observed v s l u o might riel1 have occurred at t h c chosen t
0
.
Using the
ergodic hypothesis and t h o i n t e g r a l d e f i n i t i o n o f t i m e average:
+T
'pfE ( Z )
=
limit
T--
1
2T
-T
(33 1
I n this d e v c l o p c n t o f optimization theory,
bo used p r i m a r i l y os a t o o l i n d e r i v a t i o n s ,
thc correlation f u n c t i o u i;ill
Wicner has shown t h a t the
F o u r i c r transform of t h e c o r r e l a t i o n f u n c t i o n is t h e s p u c t r a l d e n s i t y
--
a somewhat e a s i e r s t a r t i n g point f o r t h e design engineer's inngination,
The s p e c t r a l d e n s i t y d e s c r i b e s t h o power d i s t r i b u t i o n of the function i n
the frequency domain.
It is e a s i e r f o r the enginecr t o d e s c r i b e hum
i n t e r f e r e n c e by remarking on t h e amount of 60 cycle power p r e s e n t than by
s t a t i n g t h a t the c o r r e l a t i o n f u n c t i o n e x h i b i t s a marked p e r i o d i c i t y every
1/60 t h o f a sacond.
Problem 17:
Find t h e auto-corrclntion f u n c t i o n of f ( t ) =
a sin(
$
t + C )
There are two problems In which the correlation function i t s e l f
particularly useful.
i8
The firet problem i s the one posed by disturbance#
correlated with the signals.
The second 1, the problem of signal d e t e c t i o n
by comparison of the incoming signal and noise with a locally generated
signal.
syatems.
The l a t t e r problem l a encountered In crose-correlation d e t e c t i o n
When such eystema can be used (detection of a aine.wave In noise).
they are capable of out performing conventional filter systems.
%e
correlation function8 are a l a o used ir: reducing t h e labor required In
empirical determination of apectral denmities.
problem,
2.
80m
'f
Problem 18:
ff
For assistance In the80
of the propertles of correlation functions are:
(01
' L
= CT
= mean q u a r e value or i ( t )
Using t h e d e f i n i n g Integral (equation 33). demonstrate the
f i v e properties listed above, To demoimtrate the f i r e t property
consider th5 function
f(t)
f(t i%)
and its autocorrelation function properties,
+
- 44 -
Tho s 3 e c t r n l d m s i t y is t h o b a s i c d 3 s i ~ 1 1t o o l i n t h c optimization of
t i r e - i n v a r i a n t l i n e a r systcms,
The o p r z t i o n s of dckcnpi:-i??G t h c optiiiiin
1ir.l: bct;.csc s t 2 t i o n a - y tin3 scries, system t r a n s f e r functions, ;:rid t h e
s
-.,-r y
b i I.,
$.lL.*
S ~ Z C ,
tho dcsi,n
Tlic s - c s t r a l d e n s i t y ,
XL;~
of' t i n e - v a r i a n t systcms is I m t h t l i f rjc11lt rrn.4
$(&I
), or" a f i i c c t i o n
f ( t )i s
ti,
fjric? nr; +.hc
EP 204
- 45 cnougb sn t h a t t h e poxer d e m i t y does n o t change a p p r r c i a b l y within t h e
bandwidth.
Inasmuch ea t b a f i l t t r scans 2 frequencies a t t h e s a w t i m e
t h e s p c t r a l density at either a positive frqueney or its corresponding
nsgutive frequ-ncy I s given by one-half t h e yower o u t p u t of the f i l t i r
Figure 11 shows n t y p i c a l l i e n e r 8l’c:ctral
divided by t h e f i l t L r bmndwidth.
d e n s i t y and t h e a c t i o n of a narrow band scanning f i l t e r .
Suprimposed ’ f i l t e r c h a r a c t e r i s t i c
N e rrow-band scam in43
filter
centered a t z
bandwidth B.
Reading of
Power i n d i c a t o r
%
uo
B
-----_
.t
‘i-5
two>
i
FIGURE 11
Vienar S p ” c t r a 1 Density and Action of Scanning F i l t e r
If t h e density is I n t e g r a t e d over a l l frequencies, t h e n by d e f i n i t i o n the
EP 204
- 46 B.
--
Properties
Perhaps Vlienor ' 8 g r e a t e s t c o n t r i b u t i o n tvas t h e rigorous demonstration
t h a t , for s t a t i o n a r y t i m e s e r i e s , tho s p e c t r a l density and thc correlation
f u n c t i o n are
x - t r a n s f o r m s of each other.
This s i n g l e c o n t r i b u t i o n changed
the spectral d e n s i t y f r o m t h e l e v e l of empirical graphs t o t h o level of ann-
l y t i c a l mathematics.
t h e s p e c t r a l density,
P r o p e r t i e s .of F o u r i e r transforms becone p r o p e r t i o s of
Referring t o t h e s e c t i o n s on F o u r i e r transforms and
c o r r e l a t i o n f u n c t i o n s we can mite:
t-
-5"
Another concept is almost immcdiotcly introduced which might othenvi sc be
missed i n t u i t i v e l y :
t h e so-called c r o s s - s p e c t r a l depsity, a power densit;S.
p r e s e n t due t o r e l a t e d n e s s betvreen 4x0 t i m e functions.
pfp
Qo
z g(s)
f
=
-sr
e
dT
-9)
Based on t h c p r o p e r t i e s of t h e c o r r e l a t i o n function, we can now rrrite a few
p r o p e r t i e s of t h e s p e c t r a l density::
- 47 -
Problem
12:
Demonstrate the above properties,
Using the notation FT(s) tor the finite-time x t r a n s f o r m of f(t) and
GT(s) f o r the equivalent in g(t)
Those equations must be used w i t h extreme caution.
*wconvention,
combinstion.
The cross-s>eclrel d e n s i t y
t h e order of subscripts gf d e f i n e s t h e F(-s)G(ss) transform
Th2 order, fg, w o u l d r e f e r to F(SS)G(-S).
equation is p a r t i c u l a r l y treacherous.
The f i n i t e - t i m transforms given h e r e
must all be taken w i t h respect t o t h e same t i m e reference.
With proper cau-
t i o n , however, it l s possible t o d e r i v e t h e f o l l o w i n g propertiea of r p c t r a
passed through 8yste1lldl.
junction
i
(W)
h a o r a l l y s p a k i n g , the m o u n t of c r o s s - c o r r e l a t i o n is s m a l l between die-
turbances er between d i s t u r b a n c e s and s i g n a l s .
If the various time f u n c t i o n s
are independent of each other, the c r o s s - s p e c t r a l d e n s i t i e s are I d e n t i c a l l y
zero.
The converse is not n e c e s s a r i l y true.
Problem 20:
What is the output s p e c t r a l d e n s i t y of a realizable network whose
f u n c t i o n i s exp(- d% ) t o a s t a t i o n a r y t i r w s e r i e s of euto3
c o r r e l a t i o n furction e x p ( - t / % / )
cos
.
- 49 Problsm 21:
Find t h e output spectrum of t h e system given bclow by first
combining t h e t r a n s f e r functions.
Compare w i t h equation (41).
0
t hc system sivcn below. Coniparo
with htii cquction 41 and equation 4 0 ( a d d i t i o n a t innut),
0
Ar;>lyjnG t h e i-icnsr-Palcy c r i t e r i o n
3.r,
'
t o a Given s p e c t r a l c3cnsity d c t c r r . i n e s its. f t i c t o r a b i l i t y , i r e , , whcthor i t
:72,vor has shown t h a t when t h e c r i t e r i o n docs n o t hold, tho zt.at9ioiim-y t i
t,.
I ia2s
tlnit:r
cons1 d c r a t i o n is completely p r e d i c t a b l e ,
d p n s i t i c s are x ( s )
t.hc fnrm A 0
-te9
2
S
=e
and
n/i
polynmi
and
1
+r2,
in
e
- Is1
resFcctively,
,
wit11
71;
TTJOsuch s j w c t r a l
c o r r c l u t i on f r i x t - . i
PI'
EF 20&
-
50-
Such spectral d c n s i t y forms are f a c t o r a b l e i n t o
ivhcrc v(S) c o n t a i n s poles and zeros i n tho lsft half-plane only, and$(->)
contains t h c i r m i r r o r i m p s i n t h e r i g h t hand plane.
The c o r r e l a t i o n f u n c t i o n s corresponding to t h e s e spectral d e n s i t i e s
arc sums of exponential t i m functions w i t h complex cxponcnts.
The fami-
l i a r relationships of Fourier transforms t o each o t h c r hold as w e l l hi?-
twcen a functirn's
spectral d e n s i t y and its c o r r c l a t i o n function,
c o r r e l a t i o n behavior nqar
-c=O d e s c r i b e s
T r a n s f o r m s occur in " p a i r s , " i.c.,
spectral hchavj or a s
except for th?
crwsi.:ittl
'
TIM
St
Qo
*:a' ,1 r - , t ; s f ~ - ~ - m -
ing from t h c Z d o m i n t o the (ridomin is t h e saiie as traiisfui-i~!ingfrmi t h e
id
d o m i n t o t h e (-
T )domin.
For example:
-
C.
51
-
Comparison with tho Fourier "power spectrum"
I
-
The d e f i n i t i o n of s p e c t r a l d e n s i t y sounds much liko tho d o f i n i t i o n of
"power"-present i n a F o u r i e r transform.
The q u e s t i o n is o f t c n asked, if
f(t) has t h c t r a n s f o r m F ( s ) what is t h o r e l a t i o n s h i p of lF(s;)l
spectral density?
The answer t o the question as s t a t c d is:
2
to the
none.
From
equation 39:
= o
density,
Such f u n c t i o n s a l s o have zero mean square valuc,
To he trans-
fornnble f(t) obeys
J
-b
= A
thus
As w i l l be seen in the next s e c t i o n , t h e F o u r i e r power spcctrum can bc
t o p n c r a t e a s p c c t r a l density by introducing randomess.
transform, hotvever, is e v i d e n t l y not the mcsns by which tfic
of a h u l c t i o n is determined,
iised
Tho n n r m l Foiirif I
IW-ULT
;~IILI-C\
va11t~.
f u c t i m s a r ? known, the o2tirum syst.3m m y ba s l , ' ( : i i f i - d by a w:f
of i n p u t siectrtil d c n s l t i s s . . The nbatroction, * f l a t noise.,'
algebraic.
is o f t i n u s 5 f d
in t h i s cmnnction.
is a constant.
The correlation f u n c t i o n of such a sk?ctr.al 3eri5ity is
F l a t noisr requires infinite crverhg- power.
s t r a c t i o n is o f t e n u s e f u l .
A3
7;j:l
.
. (
:
), a d e l t a fr.cctlon
3t seen, howevzr, th? ab-
-53= 2RKT
where
R = r e s i s t a n c e in ohma
k = Boltzman's coristant
T = absolute t e m p r a t w e in degrees Kelvin
Ple u s u a l l y know
The second use i n f o r mathematical conventence only.
the approximate signal bandwidths.
T h e bandwidth of ~ R Yc i r c u i t we might de-
aSgn c e r t a i n l y w i l l not exceed t h e s i g n a l bandwidths by much more than an order
of magnitude,
Thus If t h e noise I s f l a t o u t t o a t l e a s t 10 signal bandwidths
its behavior o u t s i d e such l t m i t s w i l l not be important in s p e c i f y i n g t h e optl-
mum c i r c u i t ,
For example, suppose t h a t t h e eigne18 have a bandwidth (J,
centered a t z e r o frequency, and t h a t the noise s p e c t r a l d e n s i t y is given by
speaking, t h e r e s u l t i n g optimum system w i l l n o t be a f f c e t e d n o t i c a b l y by ignoring the ~ u ~ ~ o term.
u ~ 2For t h i s reason, f l s t noise is often used aa a
first approximation i n t h e d e s c r i p t i o n
Df
wide-bandwidth disturbances.
The t h i r d use of f l a t noise is o f t e n the most h e l p f u l .
We noticed t h a t
Fourier t r a n s f o r m of t i m e functions -+.ereu n s a t i s f a c t o r y i n d e s c r i b i n g s p e c t r a l
d e n s i t i e s or mean square values,
The behavior of systems t o s p e c t r a l densi-
t i e s , however, suggests an i n t e r e s t i n g way of g e n e r a t i n g s p e c t r a l d e n s i t i e s
f o r s i g n a l s (and d i s t u r b a n c e s ) whose approximate time c h a r a c t e r i s t i c s are
known,
Assume a fair knowledge of f(t) aad hmco F ( s ) ,
From t r a n s f e r function
EP 204
theory vie hiow t h a t f(t) can bo produced by insorting a unit impulse i n t o a
o i r o u i t of transfer function F(r.).
unit impulse
Let us substitute f l a t noise for t h e impulso.
Flat noise
The output spectral density
output
The output t i m e function o f t h i s system c o n s i s t s of many f(t)*s of random
amplitude and time of occurronco.
cancel.
The m u l t i p l e f ( t ) * s w i l l o v e r l a p , add, and
For e x a q l e , if f ( t ) had been asslimed a s t o p function, t h e noise-
driven c i r c u i t would produce *
or if f ( t ) = t, thc circuit w o u l d produco
-
* Using
the random l m p u l s e modo1 f o r f l a t n o i s o and assuming that t h e
l e n g t h of t h e experiment in time is not i n f i n i t e . Given i n f i n i t e time,
it can be shovm that both f t m c t i o n s above v r i l l d r i f t t o o i t h c r + or Q O .
-
The Laplace transforms of a s t o p function and a ramp h o t i o n a r e 1/a
and l/s
2
, respectivoly,
The equivalent c i r c u i t s F ( s ) for t h e i l l u s t r a t i v e
f ( t ) are t h e r e f o r e an i n t e g r a t o r and
(L
double i n t e g r a t o r ,
The sketchee
Other
above t h u s r e p r e s e n t i n t e g r a t e d and doubly-integrated f l a t noise.
f u n c t i o n s of time r e s u l t i n appropriate o i r c u i t s .
Occasionally c i r c u i t s
Bocnuse of t h e demonetrablq
a r e required with i n f i n i t e power c a p a b i l i t i e s .
p r o p e r t i e s of i n t e g r a t e d f l a t noise t o d r i f t t o plus or minus i n f i n i t y ,
b o t h t h e i n t e g r a t o r and double i n t e g r a t o r f a l l i n t o this class.
The issue
may be side-stepped by s t a t i n g :;mt t h e r e i s no such t h i n g as a p e r f e c t
i n t e g r a t o r and t h a t a l l i n t e g r a t o r s should be r e p r e s e n t e d by 1/s+a where
is i s some vory small n d e r .
If in doubt about the mathematical v a l i d i t y
o f c e r t a i n o p e r a t i o n s involving such c i r c u i t s , d e l i b e r a t e l y degrade t h e
c i r c u i t during t h e optimization process, removing the degradation at the
end of t h e process i f possible,
Several important p o i n t s a r i s e i n t h i s l a s t u s e of f l a t noise.
The
f i r s t p o i n t d e a l s with t h e s- and t-sylmnetry of t h e f i n a l s p e c t r a l density.
Inasmuch as the s p e c t r a l d e n s i t y is N2 F(s)F(-s),
f u n c t i o n may have been e i t h e r f ( t ) o r f ( - t ) ,
the g e n e r a t i n g t-
If f ( t ) were
for
example, t h e s p e c t r a l d e n s i t y describes not only a random s e r i e s of e-at
b u t e-a'-t'as
well.
I n addition, f l a t noise c o n s i s t s of o t h e r t i m e
h o t i o n 8 than a random Bet of impulses, as may r e a d i l y a s c e r t a i n e d by
viewing wide-band n o i s e on an oscilloscope,
The preceoding sketches, t h e r e -
f o r e , d e s o r i b e p o s s i b l e outputs only.
The design engineer oompletes t h e generation of s p e c t r a l d e n s i t i e s from
approximatoly knovin f ( t ) * s by specifying t h e expected mean square v a l u e of
2
t h e f ( t ) and i t s r e p e t i t i o n period, If the mean square v a l u e is gf
Ep
201,
- 56 and t h e mean r e p e t i t i o n r a t e A T f , t h e n t h e s p e c t r a l density for a ranaom
oeries of f(t)'s
Is given by
is f i r e d v e r t i c a l l y i n t o t h e etmsyhare, Vind is an
inporttint disturbance. Assuming t h a t t h e atmosphere is charact e r i z e d by l a y e r s o f random, constant-velocity r i n d s and t h a t
t h e rocket encounters new l a y e r s a t a roughly constant rate,
derive a s p c t r a l d e n s i t y f o r t h e wind v e l o c i t y , Assliming t h a t
t h e rocket ha3 a t r a n s f e r frinction 1/Ts2 from Tfnd v e l o c i t y to
missile position, what i s the s p e c t r a l d e n s i t y of rocket position?
Problem 22:
A rocket
Problem 23:
Derive t h e s p e c t r a l density of t h e v e r t i c a l e c c e l e r a t i m produced
by bump i n a road. Ths bump height y is givel; approximately by
ar2e-bx w h 4 r e x is b o r i z o n t a l d i s t a n c e , Assume constant v e l o c i t y
t r a v e l gown the bumpy rosd.
S p c i f i c a t i o n of t h e a p c t r a l j e n s i t j r of d i s t a i b s n c e s should be mdz at
t h e origin bf t h e disturbance.
S p c i f i c a t i o n of t h e t r a c k i n g noise of' a r a d a r ,
for example, is most a c c u r a t e k j obtained by l o c a t i n g t h e sourcya of t h e noise
within t h e t r a c k i n g loop, s p c i f y i n g the noise a p c t r a l d e n s i t i P s e t t h e
source, and then using t h e known t r a n s f e r c h a r a c t e r i s t i c s o f t h e t r a c k i n g
loop t o s l s c i f y t r a c k i n g nciue,
It is s u r p r i s i n g bow o f t e n a f l a t noise
assumption f o r t h e source noise w i l l yield a s p e c t r a l d e n s i t y which checks
e m p i r i c a l l y a t t h e point of i n t e r e s t i n a systtm.
Tracing the source of t h e
noise will o c c a s i o n a l l y demonstrate certain c r o s s - s g c c t r a l d e n s i t i e s a s well,
- 57 X
THE ERROR SPECTRfU DRlSIm
Now we begin t o hit pay d i r t ,
known.
The dosired operation of tho system is
By writing an expression f o r
The input spectral d e n s i t i e s are laiown.
the system error (actual perfonnanoe minus desired perf orinance) converting
t h i s expression i n t o i t s spectral d e n s i t y fom
grro,
( e ) , we may specify the
man siuare error of tho sy 8 tom,
3.
Given such an integral expression f o r moan square error, w e may ndnilnize it
by any technique applicable.
Consider the b a s i c system given i n figure 3.
Describing the output by a Laplace trnnsform equation:
E(.>= H ( s )
F(s)
-k
li(s) N ( s )
Assume t h a t the desired operation i s t o obtain the b c s t f(t) possible.*
Tho error o f the system i s thus
(4 - Fb)
E(.) =
b(s) 11
=
.-
F(s) + H(s) N(s)
The squared error i s given by
~~
*
Tho so-called s m o o t h i n g operation.
~-
The F o u r i e r Transforms F ( s ) and N(s) do n o t actually exist for s t a t i o n a r y time
series.
The error-squared expression of equation 49 is used only
method of finding the a p p r o p r i a t e s p e c t r a l d e n s i t i e s . .
Equation 49 could be
j u s t i f i e d by considering F ( s ) and N(s) a s f i n i t e - t i n o transfornis.
d e n s i t y form of equation 49 may therefore be w r j t t e n
as a shortout
The s p e c t r a l
by t a k i n g t h e necessary t i n e
averages ( s e e equation 39).
xfi
If t h e signal and noise are uncorrelated, b o t h z
-
and
&re zero. It is not
nf
l e g i t i m a t e t o find c r o s s - s p e c t r a l d e n s i t i e s by f a c t o r i n g t h e a i p a l and n o i s e
s p e c t r a l d e n s i t i e s into h c t i o n s of +s and -s rrhich a r e then conbined i n t o
apparent c r o s s - s p e c t r a l densi t' es.
Cross-spectral d e n s i t i e s must be j w t i f i e d
Independently of the form of t h e auto-spectral dr;nsities.
>
Problem 24: Assume a system i n which t h e s i g n a l f( t and n o i s e n ( t ) a r r i v e at
t h e output by t w o d i f f e r e n t r o u t e s s m h t h a t H1(s) a c t s on f ( t ) and
R2(s) on n ( t ) . The desired system o p e r a t i o n is H3(s) a c t i n g on f(t).
Assuming c o r r e l a t i o n between s i g n a l and n o i s e , what is tho e r r o r
s p e c t r a l density2
Problem 25:
1
-
What i s tho e r r o r s p e c t r a l d e n s i t y o f a 8m o t h i q ; system R ( s ) =
to a s i g n u l of s p e c t r a l d e n s i t y
s2)-'
cad a f l a t n o i s e
a3.n
s p e c t r a l d e n s i t y ? Express the answer 8s the quotient of two poly-
(a2 -
nomials.
EP 204
XI
PHILLIPS' OPTIh~lIZATION TECHNIQUE
*
Sf a complete ayotem is .at our d i s p o s a l it is worthwhile t o d e e i g n the
optimum p o s s i b l e system.
Often, however, t h e system is completely s p e c i f i e d
by o t h e r considerations and only a few parameters a r e a v a i l a b l e which can bo
v a r i e d to y i e l d best r e s u l t s ,
For example, t h e optimum system may r e q u i r e
i s o l a t i o n a m p l i f i e r s i n order to r e a l i z e t h e optimum trmsfer function,
Addition
of such a m p l i f i e r s may so i n c r e a s e t h e c o s t t h a t i n a competitive market the
r e s u l t is economic loss,
I n t h e previous s e c t i o n s we have shown how t o spocify t h e e r r o r s p e c t r a l
d e n s i t y for a system and have remarked that s p e c t r a l d e n s i t i e s u s u a l l y occur
as q u o t i e n t s of polyr~omiala, P h i l l i p s noted t h i s f a c t and s e t about evaluat i n g t h e i n t e g r a l below i n terms of t h e c o e f f i c i e n t s a and b,
jtw
whe re
%(SI
=
The r o o t s of hn(s) must all l i e i n t h e l e f t h a l f plane,
The f a c t o r i z a t i o n
of the denominator p u t s t h e i n t e g r a l i n t o i t s most u s e f u l form.
The nume-
r a t o r is n o t f a c t o r e d because such f a c t o r i z a t i o n , w h i l e c e r t a i n l y possible,
would e n t a i l a d d i t i o n a l work for the user.
of complexity of t h e p o l p o m i a l s .
*
The s u b s c r i p t n gives t h e o r d e r
.Is an i l l u s t r a t i o n l e t
From "Theory of Servomechanisms" V o l , 2 5 , N I T Radiation Laboratory Series
&(s)
2
= -(A + E) s + (A!
+ Bd2)
and thus
P h i l l i p s has evaluated such integrals for values of n f o r m 1 to 7 .
Servomechanisms page 369).
The first four integrals are
(The3r-y of
n b
0 1
The expression for
2
contains 21 terms.
2
contains 4 7 torms and
2
, ll;!
Vhat i s t h e mean square error given that
Problcm 26:
p,
=
102
1 - s
+
-ET7
v o l t s2/cps
The o p t i m i z a t i o n problem in Phillips technique is t h e problem of mimimizing
3
c.
the
d,
w i t h whatever variable parameters are available ,
w h c r e 4 i s variable then
2
0' = - A+
2 4
Ed
2 8
For example, let
.)
EP 204
f o r a mini-
dr2
ax-
= o
and consoquently
Problem 27:
What i s the best RC t i m constant i n the network below t o
mize tho mean square error when smoothing a s i p a l
3
,
in t h e prosonce of (independent)noise
f+
3
2.c
INPUT
R
c
1_
mipi-
-=-a
g.-s
OUTPUT
I
This optimization technique followed
- Piiencr 1s technique by roughly a
year. even though, mathcmatically, this technique is far more l i m i t e d in
-
63
-
Ep 204
Baaic Logic
--A. -_----The Phillips technique minimizes t h e
square error o f a system in
m9m
which a l i m i t e d number of parameters are a d j u s t a b l e ,
Tho V i e n e r optimum 5 s
Howover, the d e r i v a t i o n of
achieved v i a the calculus of v a r i a t i o n s route.
tho o p t i h , a s Y'lisnor p r e s e n t s it i n his "Stationary T i m Series,"
58
80
mathematical i n Innpngc as t o obscure t h o simplicity of logic on which it
is based.
'Indeed, t h e l o g i c has already been presented.
Except for a t r i c k
necessary t o make t h e system r e a l i z a b l e , and by considering an i n t e g r a l without
absolute value brackets, w e optimized t h e simple smoothing system i n problem
1
8,
Combining our knowledge of e r r o r spectral d e n s i t i e s with t h a t of the
c a l c u l u s of v a r i a t i o n s we may " a l m o s t " f i n d Wiener's answer in three lines.
....
8
Smoothing
Wetwork
(Signal and Noise Independent.)
FIGURE 12
The Basic Wiener Smoothing Problem
300
Ep 204
which is minimized* by t h e calculus of variations technique, y i e l d i n g ?
.
.
.
.
I
Problem 28: Show that i f ' the desired operation ha6 been H1(8) instead of
smoothing, tho abme H ( s ) would have been
The H(s) which
lized.
'we
have j u s t determined is interesting but cannot be rea-
As was shovm in t h e section on s p e c t r a l d e n s i t i e s :
and c o n s e q u m t l y H ( s ) = €I(-s).
The t r a n s f e r f u n c t i o n t h u s has terms such as
the second of which is e i t h e r unstable
(e
a t ) or u n r c a l i z a b l e
-a w
e
as was discussed i n t h e s e c t i o n an F o u r i e r t r a n s f o m s
Inasmuch os this o b s t a c l e t a k e s o d y a l i t t l e t r i c k e r y t o overaom, we
have solved for the famous Viener optimum filter as far as basic concepts
are ooncerned.
5
Trickery
3
L.
The purpose of the following trickery i s t o mako the v a r i a t i o n of
zero with an H ( s ) which is realizable.
*Problem la.
$
6
The t r i c k e r y enables us t o d i s c a r d
zp 201
part of t h e i n t e g r a l r e s u l t i n g f r o m t h e c a l c u l u s of v a r i a t i o n s d i f f e r e n t i a t i o n .
S e t t i n g t h e remainder of the integral equal to zero yields t h e optinnun realizable filter.
,The essence of t h e maneuver is oontained i n the values of two integrals.+
The c o n s t a n t s
4
and
P
are c o q l e x numbers whose real p a r t i s positive.
Let us define t w o useful forms by a p l u s and minus s u b s c r i p t such t h a t
A
The
2+
( 8 )
is defined a s having p o l e s in the l e f t - h a l f plane only. z - ( s )
has i t s p o l e s i n t h e r i g h t - h a l f plane only.
Then if Y+(s) and Y,(s)
is
another such p a i r of f u n c t i o n s , the i n t e g r a l s I1 and I2 demonstrate t h a t by
p a i r i n g terms i n
*
2
and
Y,
These values may be obtained most e a s i l y by contour i n t e g r a t i o n .
If such
a technique, is unfamiliar t o t h e reader, the values may be accepted on f a i t h .
I t is probably u n p r o f i t a b l e t o study tho technique j u s t for this developmont.
-
cc
vu
Equation 59 shows t h a t integrals whose integrands have poles in one halfr.
Those with mixed*poles have a non-zero value.
plane only have value zero.
Tie now proceed i o t a m p e r with the ;;;inirLzation carried out e a r l i e r . *
-JCS
TO minimize t h i s i n t e g r a l
replace H+ by H+
-k
E q+
i s subject to the same r e s t r i c t i o n a s
funotiont,
realizable
WB
-,
a l l i t s poles
s t i t u t i o n and s o t t i n g
*Problem 1 a
a
/a
=0
at
&
The arbitrary
q, namely
in the left-half plane.
2
.
=0
that it be
Performing the sub-
yields
EP 204
- 67 -
(62:
vlre can obtain a zero value, o f course,
but t h i s solution i s unsatisfactory.
by
The t r i c k c o n s i s t s of demonstrating
that part of the r i g h t hand side of the i n t e g r a l equation i s zero, regardless
of %or
the spectral densities.* . L e t us f a c t o @ m
+
zfd
The section on Fourier transforms gives us j u s t i f i c a t i o n f o r this action.
Then
1
rn
p+
ds
-Jgo
Gi3
-J @
j + '%
The f i r s t parts of these integrals, involving the integrals
ds
a n d . m
H+
v!
's
68
-
* c a n n o t be zero becauso they contain h t o r p i n d s with
mixed p o l e s (equation 59).
The second p a r t s of the intogrnla, however, aon-
taia both zero md non-zero parts.
-Y-9
d
To show t h i s , s p l i t
3
-%
and
into
sums.of p a r t i a l fractions
The zero parts o f t h e i n t e g r a l s of equation 65 are thus
, is
because the integrands c o n t a i n p o l e s i n one half-plane only,
65 without i t s zero parts me obtain
Rowritirig equation
- 69
The conditions for zero v a r i a t i o n , independent of
a r e t h e r e fore
-
These equations are e x a c t l y a l i k e except f o r a +s interchange,
(3ff/p)+is
c a l l e d t h e p h y s i c a l l y - r e a l i z a b l e p a r t of
By convention,
gff/q! and
is u s u a l l y
m i t t e n as
Tho optimum Yliener f i l t e r is t h e r e f o r e given by
H(s)
=
1
(63)
Y<s>
Problem 29:
r e o l izable
Prove t h a t i f the desired o p e r a t i o n had been H l ( s ) i n s t e a d of
smoothing, t h e optimum system would be
-
realizable
Tho d e r i v a t i o n has applied t o a signal and noise which are uncorrelated.
In t h i s case of c o r r e l a t e d s i g n a l and n o i s e the ansmr i s
vrhoro
H1(s) = desired minimum phase o p e r a t i o n
- 70 17hat are the optimum filters i n tho c a s e s listed
Problem 30:
below?
S i g n a l and n o i s e are uncorrelated.
N2
1
za
152
1
C.
How t o SDecifv a Viener O D t i m u m Svstom
The foregoing proof has demonstrated that the optimum TJiener system i s
given by
n,(s)
(2,f+ E,,,
y (4
where
w ( s )
(-s)
=E
,, +gm+iEfn +iEnf
H1(s) = Desired System Operation
)
Physically
(71)
Reali zable
(72)
and where, if the brackets are expressible
Physically
Realizable
Z
1
Bi
d i + a
(74)
EP
204
- 71 lvhoro t h e A, art; dctorrdried h; r q
Ciko
<jf
(Gcrrlner 0
zt nuinbcr of Lochniques.
Bnrncs pu.6;" 1G4).
If T is a p o s i t i v c zvrn3t.r (fut.w*s F r w i i c t i o n ) , t h e use of
= H1( 8 1
i n Chc 'dieref o p t i m ! Z ( s ) oqucticn rill c o n t r i b u t e l i t t l e o t h e r
of e
Ts t i p L ~ c cin
r tho left half-plane,
1
2:
If T is a
EP 204
-
-72
approximation n = 2 13 lisud,froin equation 'IG,
The a p p r o x i m t i o n might a l s o have boen used
.
-
r e s u l t i n g i n s f i i l d i f f e r e n t pLeR.
The best approximation probably
depsnds upon the s i g m l and noiec f u n c t i o n s considered.
2,
Misleading Operations
The oporktio1i, "physi:.ally
1 - e a l i z a b l c , " has been d c f i n o d only for
quotients o f poiynomials whose n u m e r a t o r is of lessor order than the denoininator,
O p e r n t i o n s H.i s ) which r e s u l t i n
1
n o t having a hii;?xr order der.wninntor than numerator g c i l s r d l y produce
nisleading results,
Out:
c;J?+;~
check b e f o r e a t a r t i x g +he problem xi11
Check t h a t i f no n o i s e were present, we
prevent such o:cur-Tt:::es.
would n o t be performing i
m operation y i e l d k g i n f i r i i t e output s i g n a l
values.
As an sxaar,lc, ssking the d i f f e r e n t i a t i o n of
1
--
would r e s u l t i n an o u t p u t spcs+,nun
%*t
-
-3
1
2
- sL
whsse mean sqaare vaiuo i s i r C i n i t e .
function f ( t )r e p r c s c n t e d by such LI
Stated a n t h e m t i c a l l y , the
9ff i s
almost never d i f f e r e n t i a b l e ,
EP 204
- 73 Problem
---
31:
a,
Firid t?m o p c r a t o r which clppcars Yhon
t.
Lot Xi +O.
Bra
YE
differentiating?
- 74 -
?Ic3R3 13
A Tinc-varian; L3ightlrg Furxtion
T\e error of the a p t m
tit
t h e t is
by
w
p o i n t (and which we w i l l a r b i t r a r 5 l y continuc t o u s e ) i s t h t * e
taken w i t h respect t o an en3einblo o f experiments,
mi;;ht
e l s o have been defined as a
t i i i ~ aa v c r z p
Tfic re:>
durkg
OM
l a t t e r definition was >.sed i n tho curve f i t t i n g Froblom.
moan ie
squfzz e r n r
ex?criment,
The
I f the systems are
- 75 -
ZP 204
D e f i n i t i o n One
D e f i n i t i o n Two
(Time Avorago)
w-
-2dM 2
n (t)dt
2T-T
= o
Using t h e ensoiiible d e f i n i t i o n leads to t h e assumption t h a t tho menn square
error of a s y s t o m is njnimizad i f it is ensemble-minimized a t every i n s t a n t .
Coming bnck t o t h e d e r i v a t i o n , t h s mean square (ensemble) average of
Defining soma ensemble c o r r e l a t i o n f u n c t i o n s
- 76 ive r e v r i t e t?ie
EP 201
m a n square error as
The mean squaro e r r o r i s now minjmized by t h e c a l c u l u s of v a r i a t i o n s approach,
Let h($,t)
to
Eat
be replaced by i.-('t:,t)+
E= 0,
&I(%."),
d i f f e r e n t i a t e w i t h respoct
and make t h e result independent of
(%,t). The r e s u l t is the
equation
fo0
which holds ( o n l y ) fcr
%>
0.
T h j z z q u a t i o n must now be solved for h(t8*),.
No s o l u t i o n has t e c n y - o p o s e d for tf7c timc-variant case.
xhich is t h o samo as
For t h c s t a t i o n a r y
dcrivcd earlier.
From our cnsemtle d e f i n i t i o n of man squaro error, B ~ ( E )
may be a t i m - v a r i a n t opcrator if desired.
A time-variant operator, H( 6,t).
dl1 appear much l i k e the usual constant-coefficient H ( a ) but \vi11 have %
a paramt.er.
Problam 32:
_
_
I
-
Prove equation 85 in the manner outlined in the t e x t .
Q8
Ef 204
XI11 A PHYSICAL IlITXRPRETATIOI! OF TIIE XLDJER O P T I U
.
d
-
Bode and Shannor* have given an instructive picture of tho signifioanoe
of the operations used In obtaining R i m e r ’ s optimum system.
vat.j.on i e b a s i c a l l y the 8am
RS
the ( : ; e presented hero.
Ianzuago are bath s l i g h t l y diffcrcnt.
Their deri-
Their approach and
Unfortunatoly, they negfbcted t o
vtrfto tho H ( s ) oquatlon in conclusion.
The engindor recognizes tho n e c e s s i t y of nalcing his system physically
xealiiable,
He a l s c agrees that
n(s)
=
s
ff
(Example only)
iF
- ff ’5,
c
_
i s a fine solution but imrenlizablc,
But why can% we write t h e solution
as follows?
T h reason, mathenaiic.a’lfy, i s t h c t t h e optimization d e r i v a t i o n docs not
work this i v a y ,
The F i ’ j s i c d reason is this:
the t o t a l i n p u t fvmction
c
-
(sign71 p l u s n o i s e ) possesses c o r r c i n t i o n , i . c O 3 som of the data in t h o
mderstands t h i s fact.
prising.
The f u t u r e infcjrination w i l l not be t o t a l l y sur-
Thus, even thoagh our wGibhting f u n c t i o n can only use a ~ a i l a b l c
d a t a , it can dc a ‘ t c t t c i snoothing job by doing sone p r e d i c t i n g , as well,
-
70
-
ET' 204
-
On t h e other hand, suppose tho t o t a l i n p u t s p e c t r a l d s n s i t y i o
( c o r r e l a t i o n only a t $= 0).
flat
The f u t u r s Is a oolrplete surprise. Undor
these oonditions l e t us derive a system v i t h no holds barred.
will be u n r e a l i z a b l e aa alwsys.
The result
But t h e unrealizable p a r t , t h o p a r t ueing
f u t u r e data, is vrorMng with r;o_l_rrpJatelyunpredictable lnforrnation (oorrel a t i o n of futuro i n p u t d a t a w i t h pa3t i n p u t d a t a is zoro).
a t f u t u r e behavior of the input i s t h a t It v r i l l b o zero.
Tho b c s t guess
Because t h e future
is unavailnblo, we t~5.Xl o s e l i t t l e on t h o average by a s s i g n i n g it a zero
value.
The "physically r e a l i z a b l e " o p e r a t i o n thus c o s t s
119
the l e a s t if
performed on the t r a n s f e r f u n c t i o n of a notwork whose t o t a l i n p u t s p e c t r a l
d e n s i t y is flst.
Let us apply t h i s ' r e a s o n i n g t o the R i r . n c r problem.
1, Convert t h e given input t o flat noise.
2.
The operator is 1/
y(8)r
Optimize the system from this point w i t h t h e desired o p e r a t i o n
being y ' ( s ) .
FTGUIZE 14
Physical Systsin for Ffnding 3 i e n e r Optimmm System
The optimutn H ' ( s ) ,
no holds barred, is
Ep
204
- 30 as shown cc:-li,or.
But t h i s ntay be w i t t o n
BB
3 0 Taking t h a physically r e a l i x ~ b l ep ~ r and
t
conhining into uno
t rari sfcr fimct ion
Problcm 33:
s,(S)
Derive the optimum H ( s ) to perform an o p e r a t i o n
t h s snmc rcnsoninz:,
Sovcrnl conscqienczs af the Kioncr so1ut:on
are ovidcnt,
probably most important;, ths best m y af pcrforrnin;
not t o first smooth t h c 5 . q . u t
-
using
F i r s t , and
an o p e r a t i o n 3 ( s ) is
m d t h o n o p e r a t e w i t h H (s),
1
It is not t r u e ,
for o x z q l c , t h a t tlic best dcri;ra.civl: of a f u n c t i o n is obtaincd by f i n d i n g
tha b c s t sriioothsd value and. thcn d i f f c r c n t i a t i n G it.
is mcra m a t h c ~ w t i c u lthml prc ctica3
:
,
The sccond conscquerce
Thc r o q i i i r z d c o r r c l e t i o n s arc not tho
s i g n a l and n o i s e c c r r o l u t i o n s bui; t h c tots:
c o r r e l a t i o n f u n c t i o n of t h e
i n p u t m d t h o ccrrclatfon of t>a s l z m ? cn t h e t o t a l input.
Input
I n Iter numerator prediction time conutant.
EP 204
XIV
---
MTZNSIOMS OF VIElER1S SOLUTION
I
The basio t h e o r y of t h e optimization technique i s now complete.
The
remainder of this t e x t c o n s i s t s of various a p p l i c a t i o n s of the Wiener
l u t i o n t o s p e o i a l problems.
the v a r i o u s problems.
SO-
It is t h e purpose of this m o t i o n t o d e s o r i b e
The reader may t h e n s e l e o t t h e t o p i o s of p a r t i o u l a r
i n t e r e s t in h i s ovrn problem.
A servomechanisms engineer, f o r example,
would probably s o l e c t t h e sect: c 5s on q u a s i - d i s t o r t i o n l e s s networks, the
s a t u r a t i o n c o n s t r a i n t , t r a n s i e n t e r r o r minimization, and t h e d e s i g n of
servomechanisms.
A r a d a r engineer would probably s e l e c t s e c t i o n s on
f i n i t e - t i m e f i l t e r s and matched filters.
The Wiener s o l u t i o n provided the necessary mathematical tools f o r
extensions of his work and f o r a ULification o f o t h e r e a r l i e r efforte.
The extensions are a l l i l l u s t r a t i o n s of t h e f a c t t h a t i f t h e c a l c u l u s of
v a r i a t i o n s mrks, -it also works under c o n s t r a i n t s .
Section XV
North, Dwork, and Niddleton independently derived t h e f i l t o r s bown
v a r i o u s l y as matched f i l t e r s , comb f i l t e r s , North f i l t o r s , and Ihvork
filters.
These men were i n t e r e s t e d maximking t h e p r o b a b i l i t y of d e t e c t i o n
of a pulse.
Their s o l u t i o n was e q u i v a l e n t t o minimizing t h e m a n square
n o i s e output under t h e c o n s t r a i n t t h a t t h e f i l t e r d u p l i c a t e the maximum
value of t h e pulse.
We w i l l d e r i v e t h e i r results i n a f e n l i n e s w i t h t h e
b e n e f i t of a more powerfcf technique t h a n was a v a i l a b l e t o them.
Ragazzini and Zadeh re-derived t h e Ihvork-North-Eddleton r e s u l t s for
a more g e n e r a l n o i s e spectrum than f l a t noise and under t h o c o n s t r a i n t t h a t
t h e network be r e a l i z a b l e .
t h e i r timo-domain solution.
Our d e r i v a t i o n is a s p e c t r a l d e n s i t y v e r s i o n o f
- 83 Soction XVI
Ragazzini and Zndeh a l s o t r i e d applying t h e o o n s t r a i n t t h a t polynonialr
i n t h e pass through the network without d i s t o r t i o n ,
necessity limited t o finite-time f i l t o r s ,
This o o n e t r a i n t is
Of
A f i l t e r which can remember the
s t a r t of a polynomial m s t , by d e f i n i t i o n , oontain t h e s t a r t i n g t r a n s i e n t .
Ragazzini and Zadoh roquire t h a t the ensemble mean square e r r o r be minimized over t h e i n t o r v o l considered.
This e x t e n s i o n of Wiener's s o h t i o l l
is s t r a i g h t f o r w a r d b u t d i f f i c u l t of applioation.
The dosign ong3noer
u8U-
a l l y has t o solve fivc o r s i x s i m l t a n o o u s equations t o o b t a i n t h e 8olutiOno
Section AXII
Tho Ragazzini and Zadeh, f i n i t e - t i m e f i l t o r i s usually apFrcxim-3ed by
t h e so-callod q u a s i - d i s t o r t i o n l e s s network,
This network passes p o l y n o i e a l s
without d i s t o r t i o n except f o r tho s t a r t i n g t r a n s i e n t ,
The q u a s i - d i s t 0 r t i . m -
less network can be optimized using e i t h e r t h e P h i l l i p s o r Wiener t e c h n i q u e S a ct i o n ----mrI
Newton a t 1L.I.T.
arid t h i s author at J.P.L.
considered s y s t e m in rJhich
s a t u r a t i o n e f f e c t s could so d i s r u p t t h e system as t o render operation worth-
less.
Both solved t h c problem by applying a p.ower level c o n s t r a i n t a t the
a p p r o p r i a t e systcm point.
S o c t i o n XIX
The t r m s i s n t e r r o r of a system may bo measured i n rany weys, depcnding
upon t h o ci'rcuit a p p l i c a t i o n .
transient-squared,
One common d c f i n i t i o n is t h e i n t e g r a l of the
Other d e f i n i t i o n s a r e c e r t a i n l y possiblc,
c i r c u i t s t h e d e f i n i t i o n might bo t h e la;: time of t h e c i r c u i t
I n switching
--
the t i m e
EP 204
- 01
c3-x
betwoen reoeption of a s t o p function and the f i r s t m r o o f the t r a n s i e n t .
44
t
k+7'
Lag Time
Both d e f i n i t i o n s are considered in d c r i v i n g t h e b e s t t r a n s i e n t performanos
undor n o i s e conditions,
Section 10(
R. J, Parks at J.P.L.
applied Wiener's s o l u t i o n t o guidanco systems
attempting t o c o r r e c t for o x t o r n a l d i s t u r b a n c e s , and hindered by h i g h
love1 measurement noise.
in philosophy:
This development demonstrated an important point
that f(t) i s not n e c e s s a r i l y a s i g n a l and n ( t ) n o t neces-
s a r i l y a noise in t h e expression f o r error.
signals,
B G ' may
~
be n o i s e or b o t h
The Wiener s o l u t i o n minimizes t h e "inem square error". defined by
the e r r o r equation
E ( * )= H(s)
N(s)
+ (1
- H(s))
F(s)
where F(s) is earrnarkcd s o t by the d e s i g n a t i o n "signal"' b u t by being the
c o e f f i c i e n t of 1-H(s).
of H ( s ) .
Section
Similarly, N(s) is defined by being the c o o f f i c i e n t
--
Any e r r o r equation of t h i s form is t r e a t e d by t h e Wiener s o l u t i o n .
XXI
The l a s t s e c t i o n of t h e t e x t d i s c u s s e s som r e l a t e d t o p i c s :
multi-
dimensional s y s t e m , s h o r t - t i m e r r o r s y s t e m , c r o s s - c o r r e l a t i o n d o t e c t i c n ,
and d e c i s i o n networks.
s o l u t i o n s are presented.
The discussion i s q u a l i t a t i v e i n nature.
No
E 2 204
The matched-filtor r o s u l t s prosentod here were p a r t l y known bcfore, or
indopsndently of, Tiionor.
inequality.
The results were dsrived originally using t h e Schvari
The basio problem v a s the detection of b m m pulse shapes under
high noise conditions,
The original statenont of tho problem asked . that,
at
.
the output of the detoction system, t h e poak s i g n a l strcngth bo as large
Thin c r i t e r i o n ic equiva-
a s possible r e l a t i v e t o the output n o i s e 10~0:.
lent t o mininieing the mean output noise level under tho c o n s t r a i n t tksat
the system duplicate tho signalts peak value.
The signal is assumed zcro until
%?IO
(arbitra-7) rofcrzncc tine t = 0,
I C then increases t o its peak value ( o r a . q other n l u e of i n t e r e s t ) a t a
t i m e to.
The constraint
TB
i q i o s e i s t h a t at t
0
tho system y i e l d f ( t o ) ,
-
The system w i l l n o t yield f(t) a t other tima unless by coincidencc,
The
constraint i s therefore given by:
d
F(s)
-st
+ e
O
3i-s) F(-s)
ds
The problem t o be solved is t h e m i n i m i z a t i o n of t h e o u t p u t m a n spuare n o i s o
level under thc constraint, iac.,
t h e mininizotion of
EP 204
- 86 -
Using the same plus and minus subsoript n o t a t i o n and teahnique given in the
Wiener derivation,
c y + and difforontiating
which i6 minimized by letting H+ -3 €
+
+I
respect to E at E = 0. The result 5s
with
a
By exactly the s m o technique as was used in the Wiener derivation, wo obtain
(
H(s) =
where
yn(s)
(95)
Physically Realizable
= g m and vhore K
(-s)
pl
I s t h e gain constant neccssary
to
y i o l d the desired peak va;ue.*
bvork solved t h e problcm without the restrict5 on of physical r e a l i z a -
bility.
His unswor
was
H( s) = Xs-St?F(-s)/gP.
realizable under c e r t a i n specialized conditions.
a8 siuned
-
5an
=xa2
This a n s w e r is physicnlly-
To mechanize a filter, Drvork
and that f ( t )had a f i n i t e t i m e durbtion.
those expeaiencies are unnecessaxy.
*Note thc caution on
,-“to
As seen above,
Under Bv.vkls conditions t h e E(s) becorns
usage i n XI1 D.
- 8.7 H( E ) =
EP 201,
F(-s)
(96)
The matahing of H(B) t o F(-8) results in the name "imtched f i l t e r " for
thi'a
a(8)o
A similar result v a s derived in the section on ourve fitting.
The weighting m o t i o n matched tho form of the signal f ' ~ n o t i o n .
T h e m filters
arb
of great i n t e r e s t in radar problems tvhen the gene-
ral form of tho roturn echo (or chain of echoos) is hown,.but we wish the
roturn signal t o "stand out" of the noise as m c h as possible,
A more general result can be derived $f the signal i s assumed t o have
a statietical part as well as the knovrn f(t).
+ K F(-8)
H(s) =
y
The a n m r is
e-sto
(971
('8)
Physically
Realizable
where
Problom 3 5 : ' Prove the result abwe.
Problem 36:.
Show t h a t the
of f(t) = 4
is given by
i l t e r t o maximize tho ratio of the peak value
in the presence of noise of spectrump 2/(1-s2)
where the approximation e-x =
1
is assumed.
The impulsive response and weighting function of a Dvrork "matched
f i l t c r n arc o f a particularly interesting form.
f o r H(s) w e find the impulsive response t o be
From the Daork expression
EP 204
- aa -
The lmpulsivs reaponee therefore etarts vdth a value f(t,)
and traoar the
'Xatched F i l t e r Impulsive Rcsponse
The vreighting function dragged down t h g r e c o r d t h u s looks like the first
part of f ( t ) up t o t i m o to. For non-flat n o i m , this characteristic io no
longer true
1
FIGURE 18
Matchcd F i l t e r Su?or?osition I ' n t s p t r t i o n
- 89 -
EP 204
The weighting function of t h e more general f i l t e r given i n equation (95)
may be constructed from our knowledge of F ( e ) and
Y)n(~).
I n equetian (96)
the impulsive response of H(s) i s given Sy the impulsive response of
.
k(-s)e-sto/
y
,(-*)I
P,R. passed through t h e network l/vn(8).
The im-
pulsive response of the bracket, however, i s t h e response of a network
l/Yn(s) t o f(t) played backwards from to.
The graphical construction of
t h e weighting function of H ( s ) i s i l l u s t r a t e d i n the following sketches.
A function, f ( t ) = a
were assumed,
+ bt, and
a network
y,(s) = 40+
8
2
*
EP 204
- EO
0
Response of
I m p 1 sive
Response of
?.E.
In i p 1sive
Response of
Ns)
- 91 Needless t o say, it is d i f f i c u l t t o mechanize a network H(s) which w i l l
y i e l d suoh an impulsive response.
The meohanization of the unusual responm
near to would probably r o q u i r e dolay l i n e s ,
mitted i n tho i n t e r v a l Octet,,
Rising exponentials are per-
but such e x p o n e n t i a l s are p r o h i b i t e d for
-
t g r e a t e r than to. The designer must t h e r e f o r e obtain e x a c t oancellatian
of r i s i n g exponential terms after to,
Tho optimum network s p e o i f i e d by equation ( 9 s ) is p r i m a r i l y useful
a standard of comparison f o r proposed s u b s t i t u t e s ,
network y i e l d s
Tho optinnun non-reali zable network yields
88
The optimum realizable
EP 201,
XVI The Finite-Time. Finite-Order, Filtsp
A f i n i t e - t i m e f i l t e r l e one whose impulsive response l a e t s only f o r
an i n t e r v a l T, a f t e r uhioh it 11 i d e n t i c a l l y zero.
Most reabily-mechanizablo
t r a n s f e r m n a t i o n a are e x p r e s a i b l e as q u o t i e n t s of polynomials
In
I.
The
impulsioa responses correeponding t o such t r a n a f e r f u n c t i o n s are thue s u m
of exponentlala in time,
Consequently most readily-mechanizable network6
do not e x h i b i t f i n i t e - t i m e behavior,
The o p t i m i z a t i o n of f i n i t e - t i m e systems is no d i f f e r e n t than the
o p t i m i z a t i o n of infinite-time systems, provlded t h a t our s t a t i s t i c a l averaging
For s t a t i o n a r y inputs,
is done in an ensemble f a s h i o n (over many experiments).
t h e s p e c t r a l d e n s i t y approach is s t i l l l e g i t b a t e ; error s p e c t r a l d e n s i t i e s
may be used as before.
RO
For t h e non-stationary o r t h e - v a r i a n t case. hornever,
must again distinguish between t h e mean square e r r o r averaged over an ensemble
and t h e man square error averaged over t h e i n t e r v a l T,
This text rill consider
only t h e ensemble average case, although the i n t e r v a l average I s more s l a i f i c a n t
for t h e f i n i t e - t i m e case than for t h e i n f i n i t e - t i m e
Cam.
The meaning of t h e expression, f i n i t e - o r d e r , is closely connected w i t h the
a p p l i c a t i o n of c o n s t r a i n t s t o optimization problems.
I n previoua a e c t i o n s we have
discussed c o n s t r a i n t s which s p e c i f y response c h a r a c t e r i s t i c s t o a f a i r l y g e n e r a l
EF' 204
- 93 A s t i l l d i f f e 2 e n t set of' constraints were proposed by Zedeh and Ragazzini
in the first of t w o articles on the optimization of finite-time systems*
0
T
These n constraints ere usad t o guarmtee distortion-free passage of a
s i p ~ i a lfimcti on
t h r o u g h the system inabmLzh a s it. i s e a s i l y shovm t h a t the response of the
f i l t o r t o f ( t ) i s given by
Evidently i f the best present value of f ( t ) i s desired,/cCo = 1,
= 0.
*I.
2.
Ths
I~YITI~?,finite-order,
i s applied
Zarleh 6: Rnzazzini: An k t e n s i o n of 'Xanerts Theory, J.A.P. Vol.21,July 1950
Optimum Filters for the Detection of Signals i n Noise,
Zadeh L: ;iagazsir,i:
I.R. 2.; October 1950.
Ep
204
-94to t h i s f i l t e r because the polynomial l o of finite-order, n.
It might be
remarked that any function can be inserted i n constraint form on the system
expressible a s a sum of known functions of time such that the
form_ does not change w i t h a s h i f t of the time origin.
A s an example:
y i a l d s the constraint8
Tr
J
0
T
f
2
0
+/yc
0
r
I a sin,&t + b
b L
coapt]
fUAOtiOM1
EP 204
- 9s -- t h e V i m e r
Tho vnrioua problems c o n s i d e r e d so far'
smoothing problem,
t h e niorlor o p e r a t i o n a l problem, and t h e matchoir f i l t e r problem
be c o n s t r a i n e d w s p e o i w n g h ( t
-- could all
> T) = 0. The constraint s p a i f y i n g
a finite-
ti,m f i l t o r is evidently
It
I:;
o a z i l y shown that this c o n s t r a i n t adds tho term
4.0 t h p H ( s ) a l r e a d y detciminod f o r t h e various problems considered.
y/(s)
caw,
is t h e ap?rcpriate o m f o r t h e problem considered,
1
(s))'
=
9ff + $' fh + 3nf + 9 nn
;
Tho
F o r t h e Trimer
for t h e matched f i l t e r case
Inazniuch as nr>t o n l y h ( t ) but- all of i t s derivatives and i n t e g r a l s
( f r o m 7 ) a r e a l s o zerc f o r t
> T,
the most general set of c o n s t r a i n t s viculd
rcslllt. i n the a d d i t i o n of a term
t o t h c irnrious ,Y(s),
The f i x t i o n C ( 3 )
sents a g o n e r a l i z o d Lal;range m l t i p l i e r .
i s witxiiglod i n t ; ~ o answer;
dapondent of ti.
must yet be determined; it repreUnfortunately, the v a r i c h l e tl
it. is n o t obvious how t o make t h e answer in-
EP 204
Nonetheless, several. sluos &r8 provided:
1.
Tlio firilta-timo answor will a o n s i s t of tho infinite-time anewcr
plus so:ne a d d i t i o n a l ter~ils.
2, Tile a d d i t i o n a l term w i l l probably be constants, t e r m of the
form -4s
7
7
B3- +
...,
ruid terns criiaing frcm the
rooto of the
L m mr atclr cf yt 3).
-
3.
The NYiLhtiaG fixcticln v d l i h e m o d i f i e d both vithir, the interval
Rsl;si:zilii cu~d 2udeh s;lv5d
for h ( t ) r r i t h i i t t h a
the prcijlem in t h e t i n e - d s m i n ,
j r:+,arv;ii
.-
0
t
< T are Liven below.,d
Thair cnswers
If t h e problem 13 tho best natchsd filter, t h e f i n i t e - t i r m unsvmr 38
(980)
where
T>to.
I n the > a r t i c u l a r case ~hsri.ths c o n s t r a i n t s art: given i n t h o fcrm of
The miriotis soluticvris
lams are scilutioxs of t;t;a
1, T r i t e t h e
y/ ( s )
c a n bo ioiibln!d,
inasmuch a s t h o l a s t two prob-
f i r s t p r z h l c m under c o n s t r a i n t s ,
The s-tion
"ppropriatc: f o r t h e p r o b l s n as n q u o t i e n t of two
~~ol~no
als
i n:
i
- 1
of
1. Q(s)12
=0
EP 204
- 98 3.
Tho c o c f f i c i o n t s Ad, Bj, C
j,
and D
3
mist bo found w i n g tho
c o n s t r a i n t oqtmtions and t h o a p p r o p r i a t e o p t i m i z a t i o n equations*
!.!at c h a
Filter:
T
0
R a p z z i n i and Zadoh coiisidcrad tho Yfienx-plus-finite-order
f i l t e r i n their
first a r t i c l e and t h e m t c h e d filter i n t h o i r second.
Tho f i n i t e - t i m f i l t c r i s su5joct t o t h e same d i f f i c u l t i e s of mchanizatiori that plagued t h e matched f i l t e r ,
It is useful as a standard of
comparison, nonetheless,
The f i n i t c - t i n 3 f i l t e r i s u s e f u l i n s o l v i n g t h e turn-on problem.
The
tiirn-on problem asks f o r t h e b e s t f i l t e r such t h a t t h e m a n squero e r r o r i s
miniaized over t h e i n t e r v u l
froill
t;
=;
0 (turn-on) t o t h e pre3ent t i m e t =
T,
The turn-on f i l t e r i s t h e r e f o r e a t i n o - v a r i a n t one depending on t h o para-
m t e r T,
The finite-tine f i l t e r i s also u s e f u l i n data-reduction work i n which
t h e t i n e i n t o r v a l i s l i i * ~ t e d u r i n g which c e r t a i n assumptions might be
valid,
F G exapple,
~
c c z s i d e r t h e pro51am of f i t t i n g polynomials to
-*ficndily d e r i v z b l c
iisiii,
thc: tzcldiique ,ivan i n XI1 i2.
( S O 8~5 )
JP204
- 99 sections of a curva of complex shape.
Tho polynomials are assumed v a l i d
o n l j for short intervals. A finite-time i e i g h t i n g function of appropriate
duratior,, T, w i l l proporly perform the f i t t i n g .
Prohloxn SSa:
17)ut i s tho optimum finitc-time f i l t e r t o puss
distortod i n the presonco of noiso
Gn
;=
Q
-t bt Un-
24rr,
2
9
itsPro’olcm 38b:
-
Prohlsrn 38c:
-
U h u t i s t h o optimun finite-tiym f i l t e r to find t h e derivative
2
of a + b t , undistortad, i n the presonce of n o i s o c z =
n
zd rn
oc -=
Find the answers for parts a and b for a noise spoctrum
H , = d o - s2
.
2
B o
I - h t i s t h e answer if
do=fo
?
EP 204
Netwoi*ks &to q u i t e oommon xhich produce no signal d i s t o r t i o n i n the
steady stat.e.
A transfer function H(s) whioh satisfies H ( 0 ) = 1 Will pass
DC u n d i s t o r t e d after the t r a n s i o n t has lried out,
Similarly w e may design
networks which pass c e r t a i n sine. waves and r e j e c t others.
One partioulor
notwork h a s recently came i n t o prominence because o f i t s utility
time duration problems.
If
a sipal
raenr; is completed long bcfore
6
is
50
izI
short
slowly varying that tho experi-
"cycle" might occur, an expansion bf the
signal in Fourier coefficients is almost meaningless,
A much more l e g i t i -
mate o x p n n s j o n would be in poirers of t, The p a r t i c u l a r network which passes
pGl:?nOlnicilS i n t und5storted i n t h e steady s t a t e is c a l l e d a quasi-distorti ctilesa network,
C u s t o m r i l y t h s name, q u a s i - d i s t o r t i o n l e s s , is applied
only t o networks which leave t h e polynomial u n a f f e c t e d i n the s t e a d y s t a t e .
It is c e r t a i n l y p o s s i b l e t o perform o p e r a t i o n s such as t i n e delay o r d i f f o i e n t i a t i o n w i t h x t steady-state d i s t o r t i o n , b u t t h e s e networks are not
o c m o a l y c a l l e d quasi-di s t o r t i o n i e s s ,
The q u a s i 4 i s t o r t i o n l e s s network
-m p g r f o r m o t h e r duties a t tho same tine.
Noise m y be reducedj60 cyole
h u m m y bo rejacted.
The necessary c o n d i t i o n for a network t o be q u a s i - d i s t o r t i o n l e s s t o
a poiyuornial of order n i s t h a t i t s transfer f u n c t i o n have matched numer a t o r a x ? denoinjnntor c o e f f i c i e n t s t o
Sn0
in equation 99 is q u a s i - d i s t o r t i o n l e s s t o t
n(s)
=
ao-tal s + a 2 s2
+a,
The network t.ransfer f u n c t i o n
2
.
a
8
(99)
ET 204
- 101 The proof:
I n the steady s t a t o
such t h a t a l l
If %ut
8’s
= ein =
we
may juggle the t r a n s f e r h c t i o n aquation
represent differentiation,
+
8
,
t +
t2
,t h e
equation i s satisfied i d e n t i c a l l y .
Notice t h a t t h e v a l u e s o f the a’s and b * s hove n o t been spocified.
The
choice of the a ’ s and b * s dopends.upon the further d u t i e s of the network and
upon t h e a l l o m b l e t r a n s i e n t t i m o f tho system.
The most u s e f u l extra duty
of t h e q u e s i - d i s t o r t i o n l e s s network i s n o i s e roduction,
F o r this purpose the
quasi-di s t o r t i o n l e s s net;rork also must be optimized, although such an o p t i mization i s n o t very d i r e c t ,
c e r t a i n time i n t e r n a l .
S i g n a l s are rarely polynoraials f o r more t h a n a
During this i n t e r v n l vie wish t h e s t a r t i n g t r a n s i e n t
t o d i e out as f a s t a s possible.
On t h e o t h e r hand, a quick t r a n s i e n t implies
a short-time woiyhting f u n c t i o n and
P
wide-band system.
mits a g r e a t d e a l of n o i s e to pass through the system,
The wide-band perThe engineor must
f i n d t h e best compromise,
Q u a s i - d i s t o r t i o n l e s s n e b o r k s are a c t u a l l y optimum t o ‘a p a r t i c u l a r class
of spectral d c n s i t i o s .
Reinerrbering t h a t a f l a t spectrum c o n s i s t s o f a series
of ii:i~mlses, we can set up t h c following t a b l e ,
;IGNAL
S?ZCTXkL DE:!SI TY
I
~
i
POSSTBLZ T I L 2 I;ZI!CTTON
inpul se s
steps
1/91
( 1/s2”) (-1)”
s e c t i o n s of p o l p o n i a l r
of dezree n -
EP 204
- 102 Tho steps, ramps, etc. occur in a random manner as w e have seen earlier,
noise speptral density apparently does not a f f e o t the faot that the
The
H(8)
specified for arnoothing such s i g n a l s are quasi-diatortionless in f o r n o *
Inasmuch a s the networks are determined by the niener operation, all coefficients are s p e o i f i e d .
Problem
I
_
_
-
a:
No further engineoring compromising is necoasary-
Show that quasi-distortionless networks result as the optimum
H ( s ) for t h e following s i g n a l and noise spectral densities.
I
1I
I
I
Qff
4/S2
'I
-
1
E2
The mem square signal values in all these cases are infinite
due t o the good probability ( i n f i n i t e DC energy) that t h e signal
@.
will d r i f t to iReplacing s Pyith"'5: +65 in'the signaI-,spectrd
d e n s i t i e s will-result in non-quasi-di stortionless filters. Thus,
the quasi-distortionless f i l t e r i s 8 limitiw caser
*
Shovrn by oxainple only.
- 103 XVIII THE SATURATION CONSTRfUIJT
P r a o t i c a l system3 a r e linear only w i t h i n c e r t a i n power l e v e l limits,
A
system designed t o handle 1 watt of power w i l l u s u a l l y s a t u r a t e p a r t i a l l y by
5 w a t t s and t o t a l l y by 10 watts,
The optimuxi system, as discussed t o this p a i n t , has assumed perfeot
l i n e a r i t y of a l l components.
Occasionally this assumption can lea& t o ex-
oessive power l e v e l s w i t h i n the system.
As an example, c o n s i d e r designing
H(s) in t h e system below.
*----pq -?-IT+
Control
Point
E nu = o
Output
Notor
FIGURE 19
A Saturable Sjstem
I n a p e r f e c t l y l i n e a r system, H(s) =
4 + so
The input power level is equal
t o t h o output power l e v e l , n e i t h e r of which is i n f i n i t e ,
a t the c o n t r o l point, hoxever, is infinite,
Thc power level
S a t u r a t i o n of t h e network o r
mctor i s unavoidable.
The best solLtion t o such a problem is t h e one which minimizes the
2
within specified
output e r r o r but keeps t h e control p o i n t power, q
G*
limits.
2
The output e r r o r i s given by
bE -
1
2TT j
2
EP 201
- 104 whero H is the motor transfer function 1/(1 +s).
ZA
Tho povrer level at
the contrpl 2 o i n t is
The q u c n t i t y t o be minimized is, therofore,
The minimization might be done by t h e standard technique,
c u t my be used in cases s i m i l a r t o this, however.
A useful short-
Tlie s h o r t c u t oonsists
of re-writing t h o o q u n t i m of interest (102) so t h a t it resembles an oquat i o n that has already boen solved.
In t h i s particular
case ne note the
g e n e r a l roserhlanco of e q u a t i o n 102 t o the cquatj.on doscribing the man
square error i n the stsn6ard Wiener snocthing problem:
-3 00
In this standnr2 smoothing equation,
feed. II(s) 5 s to bo found,
2ff and 5m are complctoly
e
A substitution
This problam has already Seer, solved for
W(s)
speci-
= H(s) %(SI
yields
2
n(s)if gffh%(s)]
is thought
of a8 an "bquivalont n o i s e spoctrum".
Using t h i s s h o r t c u t , it i~ s t r a i g h t -
forward t o show that
la t h e problom i l l u s t r a t i n g c o n t r o l of
ai
p a r t i c u l a r motors
If t h e power c o n s t r e i n t xere not p r e s e n t {
oxpcctod answer H ( s )
( l a r g e values of
The value of
3
= 1+
2 = 0)
-IPO
mould obtain the
If t h a power c o n s t r a i n t i s very severe
8.
t h e n H(s)+
.
A ( 1 +Qo
,d e s c r i b i n g
is ohoson t o yield the s p e c i f i e d
-B
6
C
-
an attenuator.
Thc s a t u r a t i o n c o n s t r a i n t can u s u a l l y be applied and solved i n the
s h o r t c u t manner i l l u s t r a t e d .
The equatiod t o be minimized is r e - w r i t t o n
i n t h o standard smoothing form.
d e n s i t i e s are dofinod,
Xzw e q u i v a l e n t s i g n a l and n o i s e s p e c t r a l
The smoothing answer i s w r i t t e n immediately and
u s e d t o find t h e answer t o t h e original problem.
Problem 4-0:
Solve t h e standard smoothing problem under t h e c o n s t r a i n t t h a t
t h c output noiso power be l i m i t e d t o a s p e c i f i e d value. Notioe
t h a t t h i s problem i3 equivalont to weighting noise o r r o r and
s i g n a l error d i f f e r u n t l y .
- 106 Problem
M:
Solve for t h e II(8) g i v i n g t h e minimum out ut noise level In the
transmission system below. Assume that
and
I
a, and t h a t the transmitter power is limitod t o a
gf=
-
fixed value, P
T.
- 107 .yIX TRANSIESJT mROR 12INIh4IWTION
The optimization toohnique m y be a p p l i e d t o other problem t h a n the
h g i n e e r s designing oontrol system are
minimization of menn square errorr
usually i n t e r e s t e d i n a dosign giving the minisum t r a n s i e n t s t o oertain
specified input signals.
The d e f i n i t i o n s of minimum t r a n s i e n t a r e varied.
One cormnon s p e c i f i c a t i o n asks t h a t t h e i n t e g r a l of the equare of t h e tran-
If
s i e n t be os small as possible.*
E(t) is
-
t h e t r a n s i e n t , f ( t ) t h e known
input, and H ( s ) the system t r a n s f e r function, then
If t h e minimization of this i n t e g r a l were t h e only requirement t h e r e would
be
LO
problem.
H ( s ) = 1 satisfies this condition.
Typically, t h e c o n s t r a i n t s
a r e t h o s e of proper performance i n the prosenoe of noise o r operation within
a s p e c i f i e d powor
level.
The d e f i n i t i o n of t r a n s i e n t e r r o r is almost like t h e definition of
s i g n a l d i s t o r t i o n i n the s p c c t r a l density case, except t h a t i n tho l a t t e r
case, we tako a time average.
I
Transient Error
Signal Distortion
i n Spcctral Density
Problom
= lim
T
1
W T
1
& 2 ( t ) dt
0
For exan?ple, considcr ri3nImizing the t r a n s i e n t error of tho system of f i g u r e 19
2
under the cdndition t h a t t h e output noise lovol be lese than co
FIGURE: 19
Il l u strfr t i v o System
The integral t o be minimized is t h e r e f o r e
0
The equation is almost an'cxact copy of t h e Viener smoothing problem equation.
The answer is t h e r e f o r e
H(s)
=
1
Physically Realizable
where/y/(8)12 = I F ( s ) I 2
l e v e l be less than
+ A Enand where 3 i s so
.
go2
i s the same as t h a t f o r
8
(107)
chosen t h a t t h e output noise
Notice from equations 106 and 107 t h a t t h e solution
signal spectral density
rf
= n\F(s)/
where n
Therefore t h e Yliener s o l u t i o n minimizes the t r a n s i e n t e r r o r t o a single
currence of Y,,cs)
Problem 4L:
-
whore\V/f(s)\
2
= 1.
00-
=
Minimize the man square noise o u t p u t of a system under the cons t r n i n t t h a t t h e t r a n s i e n t eiror due to a s t e p i n p u t be less than
= n2
S O ~ Gs p e c i f i e d constant E
. Assume
gn
.
Problem
-
43:
t e t h e mean squeru noise output of a s y s t e m under the double
constraint t h a t the tracsient error t o a step be El and to a
= n2
ramp, E2. Asstme
Kifijrn;
.
sn
Tho d e f i n i t i o n o h s o n for t r a m i e n t error dopentls upon the system.
For
example, ooiisider the p r o k l e m of quick-closing a switch with a s t e p iqxt
badly contardneted w i t h noise.
Assum t h a t t h e sv5txb c l s s e s and locks upon roception of a voltage of l e v e l
V.
The criterion for cjptimization is t l l e minimization of t h e l a g t i m e of the circ u i t wider the const-rhlnt. that. t.he oxitput noise l e v e l be less than (V/lO)
2
.
The la(: t i i m is d e f i n e d as t?ie t i n e i r i t e r v o l between the str& of a (noisef r e e ) s t e p mc':t h e instmt. t h a t tho c i r c u i t output f i r s t m a c h o s V.
1-
lag
I-
tiirrc
to
- 110 The time t h a t t h e c i r c u i t reaches V i s given by
j
?
The problem is considerably simplified by use' of t h e syrmnetry c o n d i t i o n of
t h o calculus of v a r i a t i o n s .
I n s t e a d of v i n i n d z i n g t, with t h e output noise
l e v o 1 f i x e d , ninimize the output n o i s e level with to fixed.
however, is e q u i v a l e n t t o the c o n s t r a i n t
& (to) = 0,
Fixine to,
Therefore the i n t e g r a l
The c o n s t r a i n t i s w r i t t e n as functions of S(s) and H ( - s )
f o r convenience.
The
i n t e g r a l equation, however, i s e x a c t l y t h e same as that. used i n the matched
f i i t e r derivation.
The answer i s t h e r e f o r e
where R i s a gain c o n s t a n t t o be d e t e m 5 n e d . l y
t h e l a g time.
n
2I).(
=
$n
and where to ia
The s o l u t i o n is completed by f i n d i n g the r e l a t i o n s h i p 'between
t.he o-dtput n o i s e l e v e l and to.
If
Gn = n2, t h e
best h(t) is a c o n s t a n t of
EP 204
- 111 h e i c h t l/to l a s t i n e for to scoonde.
The o u t p b t
to a
stop i n p u t
i.8
nally s p a c i f i e d , to = 1 C h ' f l
The output n o i s e l o v e 1 is
a ramp funotion, 0 a t
seoonda.
t = 0 and
v
2
2
=P
a t to*
/toe
EP 204
- 112 XX THE DESIGN OF SERVOI~ECHANISGIS
nvc
major techniques have been presentad which may be used to d e s i g n
somomechnnisms operating i n t h o presence of intorforenoe.
t h e P h i l l i p s technique.
The f i r s t
50
The mean square e r r o r of tho system is expressed
a s a f u n c t i o n of the input s p e c t r a l d e n s i t y c o n s t a n t s and of t h e a d j u s t a b l e
parametors of t h e system.
The moan squere e r r o r is then minimized by proper
adjustment of' the p a r o t e r a .
The complete form o f the s y s t e m l a specified.
The second major technique s t e m from Wiener's optimization solution.
Tho system is mow assumed known except for a transfer function H ( 8 ) .
t h e goal o f the techniqum t o specify H ( s ) completely,
1% is
In a servomechanism,
such a s p e c i f i c a t i o n deternines the loop trnnsfer f u n c t i o n , t h e loop gain,
and t h e l o o p t r a n s i e n t behavior.
For example, consider t h e simple asmo
i1lustra ted be 1ow.
FIGURE 20
A Simplo Serpomechanism
Given Hl(s),
what i s tho best design f o i H ( s ) t o y i e l d t h e minimum mean
square f o l l o w i n g e r r o r ? The s o l u t i o n here is most simply achieved by re-
p l a c i n g the whole servo w i t h U ( s ) ,
8 204
Knowing W ( s ) and H l ( s )
we solve f o r H ( s ) t o complete the problem.
The
important feature of t h i s tochnique i s the complete s p e c i f i c a t i o n of H ( s )
and consequently of thc whole loop.
Scvomechanisms designed in this manner q u i t e o f t e n run a f o u l of power
limitations.
A s we have seen*, however, power level constraints may be added
without d i f f i c u l t y and the problem usually solved by defining equivalent
n o i s e and signal spectral densities.
If tho servo i s asked t o p e r f o m an o p r a t i o n , H2(s),
following the input, the best design i s evidently given by
rather than simply
- 114
The foregoing i l l u s t r a t i v e semt is e quiot sorvo in t h o t no noiso or
disturbance i s introduood w i t h i n tho loop i t s e l f .
DOITO
can a l s o be solved, however.
t h i s author's knovrlodp by
The problem of t h e n o i s y
This s o l u t i o n was f i r s t demonotratod t o
R. J. Perks of J.P.L.
Consider t h e following
noisy servo.
FIGURE 21
A Noisy Servomechanism
The second noiso, n p ( t ) , might be measurement e r r o r of t h e output such
pctentiomcter n o i s e from an output p o s i t i o n i n d i c a t o r .
If
t h o servo
connected together by r a d i o l i n k s , n2(t) might be atmospheric s t a t i c .
88
is
For
i l l u s t r a t i o n , assumc t h o servo is t r y i n g t o follow f ( t ) and t h a t a l l signals
and n o i s e s are icdependeiit of each other.
The error of such a system is:
The p r o b l c m w i l l be solved by reducing it t o an cqu.ivalent smoothing pro5lom*
* For
a s i z p l e r i l l u s t r a t i c n of t h i s method of s o l u t i o n see s e c t i o n XVTII.
Raducing a problcn t o t h e equivalent standard smoothing problcm l e t s us u s e
t h e stzntlsrd smoothing answer. The d i r e c t rethod involves another c o r y l o t e
derivation.
-
115
This p a r t i c v l n r r e d u c t i o n i s espcclclly i n t e r e s t i n g bocause it i l l u a t r a t e o
t h e us0 of cross-spectral d e n s i t i e s in a s o l u t i o n even thou&h a l l the ori-
ginal inputs are assuxad unoorralotod.
Reviewing t h e quiot servo, n o t i c e t h a t the s o l u t i o n vms s i m p l i f i e d l g ~
using V ( a ) i n s t e n d o f li(s) before s t a r t i n g tho operations of f a c t o r i n g , o ;o
The s u b s t i t u t i o n i s 8lightly
The noisy servo solution i s quite eimilar,
d i f f crcnt.
The equivalent s i s a l and oquivrtlont noise ore defincd by analogy t o the
simfle smoothing error oquution:
~ ( s )
= (1-V)
fl(s)
+ Wn*(s)
(1151
Thus, t h e eq~ivalcnts i g n a l ond noise am
Equivalent S i p a l *
= H1nl
~ q u i v a ~ e n~to i s o *
=-(n2
+
BlFa P
(1161
+ F)
Botice ths cross-correloticn bctwcon o q u i v c l m t s i g n a l and equivalent noise
duo t o tho projcnco of F i n b o t 3 functions.
*
N e i t h e r t h s equivalent s i g n a l n o r t h e equivalent noise may be f'unctions
of t h e u n h o w n E ( s ) by d e f i n i t i o n of the hasic Tiienor problem.
0
EP 204
- 116
K r i t i n g t h c equivfalant ( p i m d ) s ~ e o t r s ldensities:
The s o l u t i o n m y be written by annlcgy to the s j q l o smooth5ng problom
(119)
and vhere
1 + I $ ( s ) H(s)
Fa icteresting p o i n t is Srivea homo if we allow f ( t ) t o be zero.
fOrn31 signal enters the network, only disturbances.
No
Tho e r r o r of t k e
systsn is s t i l l cxprsssible In LIZ e c p l v n l s n t smoothing f o m .
The equi-
- 117 Problom 44:
-----
EP 204
Design tha system below t o bost oountoract wind disturbanoes
under sovcre meusursrmnt error conditione.
- 118 XXI
-
ilELATED TOPICS
This t e x t has ondoavored t o present c e r t a i n fundamentals of linear
system design.
A t l o u s t four r e l a t e d eubjocts hove been almost totally
3 gnored.
A.
hlulti-dimensional System
The system discussed have a13 boen one-dimcns:.mal.
A multi-
dimensional system would be one where the best o p e r a t i o n s on many d i f f e r e n t
(but related) phonomna were t o bz spacifiod,
D.
Short-time Error Systsns
,
I n csrtain
-
cescs
i n G-hich %lis tottl t f m cf cpcretion is short ~ o m -
pared t.0 t h o tj4w necessary t o reach stTady s t a t e , ths long-tine overage
z
c)
.
Gf
It-may 97en be p r o f i t a b l e t o design an unstable
3 Day be unimportant.
system,
As a n cxaxple, consider t h o i n t e g r a t i o n cf a function in the
prcsencs of f l a t noise,
error a t t
I'iith t h o system at r c s t u n t i l t ='O,
= 0 w i l l be zcro.
is i n f i n i t c .
ht t =
g0,
t h e output
h o ~ e v e r , t h e mean square e r r o r
At intormodinte t i n e s , thc expcctcd error may s t i l l be
acccpfallc.
C,
Cross-correlation Dotection
If t h o
c x c c t C , i x dcpcndence of a signal is lmown except f o r
paraTnetErs, t h e
?ammeters
o. fen
m y be detorndnod by c o r r e l a t i n g thc t o t a l
hpt.w i t h a l o c c l s i g n a l o r s e t of signals,
For example,
D
sine wave
m y be d e t - c t c d in thc presence of noisc by c r o s s - c o r r e l a t i n g tho t o t a l
i n p u t with a local sine wavo i n phas3 with t h e incoming sine vav0.
The
tschniquc i s w i d e l y used under t h e n w o s prodcct d e t e c t i o n ar,d crossc o r r c l s t i o n dotcction.
Such dctoctior, s-rsterns arc i n h c r c n t l y i o t t o r
- 113 than auto-coimlation detectors but are more oomplox i n requiring a stable
local
80UE08.
Uultiplior
Lo-Pass
Local Oscillator
sin a
t
Output
= a(t)
+.
smaller tenna
A Comparison of Auto-Corre1ation Detection with Cross-Correlation Detection
De
Decision IJetworks
The cross-correlation detector i s a good example o f a system ue.ng
Figure 22, for example, the exact frequency and phase o f the modulation
sine vJave wcre kn0;m.
Decision networks provide another examplo of n high debrce of. a p r i o r i
knowledge.
If l i t t l e but the spectral density of a signal i s
'mown,
the
EP 204
system must be able t o produoe an i n f i n i t y of p o s s i b l e output values.
On
tho o t h d hand, If it i s known beforohand t h a t the signal i s e i t h e r
or
011
o f f , and that it c o n o i s t s of a puro sine wave of knovm amplitude and phase,
t h e problem i s quite dicforont.
Notworks which decide the question of the
prcoonco of h o t i o n s are examples of tho gonoral class, "decision nottvorks."
Such networks
two often
non-linoar.
The deoision 5s o f t e l l .
indicnLcd by tho openin;; or closinz of a switch.
Decision networks are
dozi,pad usin& the stat: s t i c a l tachniqucs of Nopan-Ponrson, Siogort,
and Ilald.
REFCRZlICZ LIST
Tho C r i t e r i a
--
I.
A.
f o r Optixun
BoZe & Shanyon:
Linear b a s t
--1950 IRZ, p a p a
T h o 3 , April
Basic A s s i ~ i t i o n , "
.
B.
Tiionor:
C,
Lawnon m d Uhlonbock:
161-167,
I).
Golay: C o r r o l n t i m VS. Linaar Transf3rms, Febmary 1953 IREa
pa;es 2 6 T a T d 7 6 9
S t a t i o n a r y Time Saries, p a p 13
Throskhold
--
S i g n a l s , pagos 149-151,
-
.
Elenenttiry
--
11.
Squaro Snoothing and Prediction
424 ~~~:-'"Discussion
of t'F;e
Calculus cf V a r i a t i o n s
Principles
;f IJathcmatical Physics, Chapter V,
----.-
A.
?;ouston:
p e e s 53
- 61,
B.
Siener:
S t a t i o n a r y Tine S e r i e s , page 14
s
-C o n s t r a i n t-
111.
A.
IIouzton: P r l c c i ? l e s of Ihthomatical Physics, Chapter V,
y q e s 53 - 6 1
e
-
B.
IV,
'
--
P h i l l i p s : Analytical Geoinetry
and Calculus, Chaptor X I T I ,
-Scctior, 209
The -Curvo
Fittin2
----- .
A.
_
Prcllcin
nzC Z e i g h t i n g Functions
~
-__.---_I_-
-- -
-
Mcthematics_--o f S t a t i s -t-i c s , pages 145
150, (The
trcetncnt] paces 152-153, ( r c l a t i v c t o problem 6
t h i s f i t i s not t h c best i f tho xioise is on y
.
)
Kemey:
-____-I_
ClRSSiC.ci1
-
B.
Ecol: I n t r o d u c t i o n t o Idathermtical
S t a t i s t i c s , Chnptar
--paLe P 7 T - Z E . 7 I
-
V,
2
-
122
--
-
BX B L I O G F i A ~
Y i Q URIER
AND LAPIACe TRhIJSFOW
Cardnar and Earnest
A.
B, Wiener:
Statlonarv
Transients In Llnear Syatemg, Chapters 111 arrd
Tima Serie8, pages 25-31, 36-37, 53-55
Jame8, Nichols, 6 P h i l l i p s :
pages 23-62
C,
Theory of Sarvomechanlama, Chapter 2.
VI THE SUPERPOSITION INTEORAL
Jameg, Nichols, & P h i l l l p a i
P36es 9-40
A,
B, Gardner and Barnes:
Theory of Servomechanism, Chapter 2,
Wanalents i n Linear System,
pages 228-236
CORRELATION FVNCTIONS
VI1
Jams. Nichols, and Phillips: Theory of Servorriechanisms, p g e s 270278. Note: S u b s c r i p t convention used by t h e s e authore i a t h e revereo
of t h a t used i n t h i s t e x t .
A.
SFECTRAL DEIJsrrY
VI11
Theory of Servomechanisms, .pages 27&-
A,
James, Nichols, andPhillips:
291
E.
Lawaon and Ublenbeck: Threshold S i m a l s , Chapter8 3 and 4. Note:
These a u t h o r s use a s p c t r a l d e n s i t y defined over positive frequenclaa
only. This usage r e s u l t s i n t h e expression for a p a r t i c u l a r spectral
density being twice t h a t used i n t h i s text.
IX THE FLAT IJOISE CONCEFT
X
A,
James, Nichols, and P h i l l i p s , Theory of Sarvomschanisms, pages 29890.
321-322.
33,
Lawson and Fmlenbeck:
Threshold Sianals,
€We8
42-46, 69-71-
THE ERROR SPECTRAL DENSITY
A.
JBIT,CS,Nichols, and Phil?.iys:
315, 317-322
Theory of Scrvomachanisms, pa6338 312-
XI PHILLIPS OPTIMIZATION TECHNI3lE
A.
Jams, Nichols, and Phillips:
315, 323-328 359-370
Theory of Servomechanisms, pages
308-
- 123 XI1
VVIENER OYTIMUh! FILTER3
'Riener;
A,
Stationary Time Series. pages 81-94,
l.29-133, 149-160
An ODtimizatlon Theory for Time-Varying
Linear Systems, h o c . IRE,' Vol. 40, pages 977-981, August 19%
B. Richard C. Booton, Jr.:
XI11 A PHYSICAL INTERP.HETATION OF THE ;VIENER OVrIMll4
Bode and ShuMOn: . A Simplified Derivation of Linear b a s t SauarQ
Smoothinj-d
Prediction Theory, Proc3edlnge of the IRE, A p r i l 1950.
A.
pages
417-1125
EXTENSIONS OF VIENER'S SOUJTION
XIV
A.
Dwork: Detection of a Pulse Suptrimposed on Flu,ctuation Noise, Proc e t d i n g s of the IRE, pol. 38, p g € S nl-n4,
1950
B.
Zadeh and Ragazzini: Optimum F i l t e r s for t h e Detection of SlRnals In
yoise, Proceedings of t h e IRE, Vol. 40, References given t o f i l t e r s on
pages 1223, 1224, October 1952
XVI THE FINITE-TIIE , FINITE-OR3ER. F I L E R
XVII
A.
G d t h , and Ragazzinit An Extension of Viener's Theory, 3-A.P.,
Vol. 21, July 1950, u a g a - h ! j ! j .
E,
Zadeh and Ragazzini: O i J t i m u m F i l t e r s fc? t h e Detection of S i m a l a
i n Noise, Proceedings IRE. Vol. 40, pages 1223-1231
QUASI-DISTORTIC)I\JLESSNETWORKS
None Teadily available
A.
George C. Newton, J r . ? : Ccmpensation of Fmdback-Control System8
Sub j T c t to Saturation, Journal of the Franklin I n s t i t u t e . Pol. 254.
No. 4 , October 1952, pages 281-413
See r*f?rcnc* i n XVIII
XX
THE DESICN OF SERVOIECHAI'VISIi'G
See rpfmrenccl i n
XVIIX -