1. science
2. mathematics
3. statistics

A course in Time Series Analysis
Suhasini Subba Rao
Email: [email protected]
October 20, 2014
Contents
1 Introduction
1.1
6
Time Series data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6
1.1.1
R code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9
Detrending a time series . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11
1.2.1
Estimation of parametric trend . . . . . . . . . . . . . . . . . . . . .
11
1.2.2
Estimation using nonparametric methods . . . . . . . . . . . . . . . .
12
1.2.3
Estimation of the period . . . . . . . . . . . . . . . . . . . . . . . . .
14
1.3
Some formalism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
21
1.4
Estimating the mean . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
22
1.5
Stationary processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
25
1.5.1
Types of stationarity (with Ergodicity thrown in) . . . . . . . . . . .
26
1.5.2
Towards statistical inference for time series . . . . . . . . . . . . . . .
31
What makes a covariance a covariance? . . . . . . . . . . . . . . . . . . . . .
32
1.2
1.6
2 Linear time series
36
2.1
Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
37
2.2
Linear time series and moving average models . . . . . . . . . . . . . . . . .
38
2.2.1
Infinite sums of random variables . . . . . . . . . . . . . . . . . . . .
38
The autoregressive model and the solution . . . . . . . . . . . . . . . . . . .
41
2.3.1
Difference equations and back-shift operators . . . . . . . . . . . . . .
41
2.3.2
Solution of two particular AR(1) models . . . . . . . . . . . . . . . .
41
2.3.3
The unique solution of a general AR(1) . . . . . . . . . . . . . . . . .
44
2.3.4
The solution of a general AR(p) . . . . . . . . . . . . . . . . . . . . .
45
2.3
1
2.3.5
Explicit solution of an AR(2) model . . . . . . . . . . . . . . . . . . .
46
2.3.6
Features of a realisation from an AR(2) . . . . . . . . . . . . . . . . .
49
2.3.7
Solution of the general AR(∞) model . . . . . . . . . . . . . . . . . .
51
An explanation as to why the backshift operator method works . . . . . . .
55
2.4.1
Representing the AR(p) as a vector AR(1) . . . . . . . . . . . . . . .
58
2.5
The ARMA model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
60
2.6
Simulating from an Autoregressive process . . . . . . . . . . . . . . . . . . .
64
2.4
3 The autocovariance function of a linear time series
3.1
3.2
3.3
The autocovariance function . . . . . . . . . . . . . . . . . . . . . . . . . . .
68
3.1.1
The rate of decay of the autocovariance of an ARMA process . . . . .
69
3.1.2
The autocovariance of an autoregressive process . . . . . . . . . . . .
70
3.1.3
The autocovariance of a moving average process . . . . . . . . . . . .
77
3.1.4
The autocovariance of an autoregressive moving average process . . .
78
The partial covariance and correlation of a time series . . . . . . . . . . . . .
79
3.2.1
A review of partial correlation in multivariate analysis . . . . . . . .
80
3.2.2
Partial correlation in time series . . . . . . . . . . . . . . . . . . . . .
83
3.2.3
The variance/covariance matrix and precision matrix of an autoregressive and moving average process . . . . . . . . . . . . . . . . . . . . .
87
Correlation and non-causal time series . . . . . . . . . . . . . . . . . . . . .
89
3.3.1
The Yule-Walker equations of a non-causal process . . . . . . . . . .
92
3.3.2
Filtering non-causal AR models . . . . . . . . . . . . . . . . . . . . .
92
4 Nonlinear Time Series Models
4.1
4.2
68
95
Data Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
97
4.1.1
Yahoo data from 1996-2014 . . . . . . . . . . . . . . . . . . . . . . .
97
4.1.2
FTSE 100 from January - August 2014 . . . . . . . . . . . . . . . . . 100
The ARCH model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
4.2.1
Features of an ARCH . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
4.2.2
Existence of a strictly stationary solution and second order stationarity
of the ARCH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
2
4.3
4.4
4.5
The GARCH model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
4.3.1
Existence of a stationary solution of a GARCH(1, 1) . . . . . . . . . . 107
4.3.2
Extensions of the GARCH model . . . . . . . . . . . . . . . . . . . . 109
4.3.3
R code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
Bilinear models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
4.4.1
Features of the Bilinear model . . . . . . . . . . . . . . . . . . . . . . 110
4.4.2
Solution of the Bilinear model . . . . . . . . . . . . . . . . . . . . . . 112
4.4.3
R code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
Nonparametric time series models . . . . . . . . . . . . . . . . . . . . . . . . 113
5 Prediction
115
5.1
Forecasting given the present and infinite past . . . . . . . . . . . . . . . . . 116
5.2
Review of vector spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
5.2.1
5.3
Spaces spanned by infinite number of elements . . . . . . . . . . . . . 125
Levinson-Durbin algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
5.3.1
A proof based on projections . . . . . . . . . . . . . . . . . . . . . . . 128
5.3.2
A proof based on symmetric Toeplitz matrices . . . . . . . . . . . . . 130
5.3.3
Using the Durbin-Levinson to obtain the Cholesky decomposition of
the precision matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
5.4
Forecasting for ARMA processes . . . . . . . . . . . . . . . . . . . . . . . . . 133
5.5
Forecasting for nonlinear models . . . . . . . . . . . . . . . . . . . . . . . . . 140
5.5.1
Forecasting volatility using an ARCH(p) model . . . . . . . . . . . . 140
5.5.2
Forecasting volatility using a GARCH(1, 1) model . . . . . . . . . . . 141
5.5.3
Forecasting using a BL(1, 0, 1, 1) model . . . . . . . . . . . . . . . . . 143
5.6
Nonparametric prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
5.7
The Wold Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
6 Estimation of the mean and covariance
6.1
An estimator of the mean . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
6.1.1
6.2
149
The sampling properties of the sample mean . . . . . . . . . . . . . . 150
An estimator of the covariance . . . . . . . . . . . . . . . . . . . . . . . . . . 152
3
6.2.1
Asymptotic properties of the covariance estimator . . . . . . . . . . . 154
6.2.2
Proof of Bartlett’s formula . . . . . . . . . . . . . . . . . . . . . . . . 155
6.3
Using Bartlett’s formula for checking for correlation . . . . . . . . . . . . . . 160
6.4
Long range dependence versus changes in the mean . . . . . . . . . . . . . . 163
7 Parameter estimation
7.1
7.2
7.3
166
Estimation for Autoregressive models . . . . . . . . . . . . . . . . . . . . . . 167
7.1.1
The Yule-Walker estimator . . . . . . . . . . . . . . . . . . . . . . . . 167
7.1.2
The Gaussian maximum likelihood . . . . . . . . . . . . . . . . . . . 170
Estimation for ARMA models . . . . . . . . . . . . . . . . . . . . . . . . . . 174
7.2.1
The Gaussian maximum likelihood estimator . . . . . . . . . . . . . . 174
7.2.2
The Hannan-Rissanen AR(∞) expansion method . . . . . . . . . . . 176
The quasi-maximum likelihood for ARCH processes . . . . . . . . . . . . . . 178
A Background: some definition and inequalities
4
180
Preface
• The material for these notes come from several different places, in particular:
– Brockwell and Davis (1998)
– Shumway and Stoffer (2006) (a shortened version is Shumway and Stoffer EZ).
– Fuller (1995)
– Pourahmadi (2001)
– Priestley (1983)
– Box and Jenkins (1970)
– A whole bunch of articles.
• Tata Subba Rao and Piotr Fryzlewicz were very generous in giving advice and sharing
homework problems.
• When doing the homework, you are encouraged to use all materials available, including
Wikipedia, Mathematica/Maple (software which allows you to easily derive analytic
expressions, a web-based version which is not sensitive to syntax is Wolfram-alpha).
• You are encouraged to use R (see David Stoffer’s tutorial). I have tried to include
Rcode in the notes so that you can replicate some of the results.
• Exercise questions will be in the notes and will be set at regular intervals.
• You will be given some projects are the start of semester which you should select and
then present in November.
5
Chapter 1
Introduction
A time series is a series of observations xt , observed over a period of time. Typically the
observations can be over an entire interval, randomly sampled on an interval or at fixed time
points. Different types of time sampling require different approaches to the data analysis.
In this course we will focus on the case that observations are observed at fixed equidistant
time points, hence we will suppose we observe {xt : t ∈ Z} (Z = {. . . , 0, 1, 2 . . .}).
Let us start with a simple example, independent, uncorrelated random variables (the
simplest example of a time series). A plot is given in Figure 1.1. We observe that there
aren’t any clear patterns in the data. Our best forecast (predictor) of the next observation
is zero (which appears to be the mean). The feature that distinguishes a time series from
classical statistics is that there is dependence in the observations. This allows us to obtain
better forecasts of future observations. Keep Figure 1.1 in mind, and compare this to the
following real examples of time series (observe in all these examples you see patterns).
1.1
Time Series data
Below we discuss four different data sets.
The Southern Oscillation Index from 1876-present
The Southern Oscillation Index (SOI) is an indicator of intensity of the El Nino effect (see
wiki). The SOI measures the fluctuations in air surface pressures between Tahiti and Darwin.
6
2
1
0
−2
−1
whitenoise
0
50
100
150
200
Time
Figure 1.1: Plot of independent uncorrelated random variables
In Figure 1.2 we give a plot of monthly SOI from January 1876 - July 2014 (note that
there is some doubt on the reliability of the data before 1930). The data was obtained
from http://www.bom.gov.au/climate/current/soihtm1.shtml. Using this data set one
−40
−20
soi1
0
20
major goal is to look for patterns, in particular periodicities in the data.
1880
1900
1920
1940
1960
1980
2000
2020
Time
Figure 1.2: Plot of monthly Southern Oscillation Index, 1876-2014
7
Nasdaq Data from 1985-present
The daily closing Nasdaq price from 1st October, 1985- 8th August, 2014 is given in Figure
1.3. The (historical) data was obtained from https://uk.finance.yahoo.com. See also
http://www.federalreserve.gov/releases/h10/Hist/. Of course with this type of data
2000
1000
nasdaq1
3000
4000
the goal is to make money! Therefore the main object is to forecast (predict future volatility).
1985
1990
1995
2000
2005
2010
2015
Time
Figure 1.3: Plot of daily closing price of Nasdaq 1985-2014
Yearly sunspot data from 1700-2013
Sunspot activity is measured by the number of sunspots seen on the sun. In recent years it has
had renewed interest because times in which there are high activity causes huge disruptions
to communication networks (see wiki and NASA).
In Figure 1.4 we give a plot of yearly sunspot numbers from 1700-2013. The data was
obtained from http://www.sidc.be/silso/datafiles. For this type of data the main aim
is to both look for patterns in the data and also to forecast (predict future sunspot activity).
Yearly and monthly temperature data
Given that climate change is a very topical subject we consider global temperature data.
Figure 1.5 gives the yearly temperature anomalies from 1880-2013 and in Figure 1.6 we plot
8
150
100
0
50
sunspot1
1700
1750
1800
1850
1900
1950
2000
Time
Figure 1.4: Plot of Sunspot numbers 1700-2013
the monthly temperatures from January 1996 - July 2014. The data was obtained from
http://data.giss.nasa.gov/gistemp/graphs_v3/Fig.A2.txt and http://data.giss.
nasa.gov/gistemp/graphs_v3/Fig.C.txt respectively. For this type of data one may be
trying to detect for global warming (a long term change/increase in the average temperatures). This would be done by fitting trend functions through the data. However, sophisticated time series analysis is required to determine whether these estimators are statistically
significant.
1.1.1
R code
A large number of the methods and concepts will be illustrated in R. If you are not familar
with this language please learn the very basics.
Here we give the R code for making the plots above.
# assuming t h e data i s s t o r e d i n your main d i r e c t o r y we s c a n t h e data i n t o R
s o i <−
s o i 1 <−
scan (”˜/ s o i . txt ”)
t s ( monthlytemp , s t a r t=c ( 1 8 7 6 , 1 ) , f r e q u e n c y =12)
# t h e f u n c t i o n t s c r e a t e s a t i m e s e r i e s o b j e c t , s t a r t = s t a r t i n g year ,
# where 1 d e n o t e s January . Frequency = number o f o b s e r v a t i o n s i n a
9
0.5
−0.5
0.0
temp
1880
1900
1920
1940
1960
1980
2000
Time
0.6
0.2
0.4
monthlytemp1
0.8
Figure 1.5: Plot of global, yearly average, temperature anomalies, 1880 - 2013
2000
2005
2010
2015
Time
Figure 1.6: Plot of global, monthly average, temperatures January, 1996 - July, 2014.
# u n i t o f time ( y e a r ) . As t h e data i s monthly i t i s 1 2 .
plot . ts ( soi1 )
10
1.2
Detrending a time series
In time series, the main focus is on modelling the relationship between observations. Time
series analysis is usually performed after the data has been detrended. In other words, if
Yt = µt + εt , where {εt } is zero mean time series, we first estimate µt and then conduct the
time series analysis on the residuals. Once the analysis has been performed, we return to
the trend estimators and use the results from the time series analysis to construct confidence
intervals etc. In this course the main focus will be on the data after detrending. However,
we start by reviewing some well known detrending methods. A very good primer is given in
Shumway and Stoffer, Chapter 2, and you are strongly encouraged to read it.
1.2.1
Estimation of parametric trend
Often a parametric trend is assumed. Common examples include a linear trend
Yt = β0 + β1 t + εt
(1.1)
Yt = β0 + β1 t + β2 t2 + εt .
(1.2)
and the quadratic trend
For example we may fit such models to the yearly average temperature data. Alternatively
we may want to include seasonal terms
Yt = β0 + β1 sin
2πt
12
+ β3 cos
2πt
12
+ εt .
For example, we may believe that the Southern Oscillation Index has a period 12 (since
the observations are taken monthly) and we use sine and cosine functions to model the
seasonality. For these type of models, least squares can be used to estimate the parameters.
Remark 1.2.1 (Taking differences to avoid fitting linear and higher order trends)
A commonly used method to avoid fitting linear trend to a model is to take first differences.
11
For example if Yt = β0 + β1 t + εt , then
Zt = Yt+1 − Yt = β1 + εt+1 − εt .
Taking higher order differences (ie. taking first differences of {Zt } removes quadratic terms)
removes higher order polynomials.
Exercise 1.1
(i) Import the yearly temperature data (file global mean temp.txt) into R
and fit the linear model in (1.1) to the data (use the R command lsfit).
(ii) Suppose the errors in (1.1) are correlated. Under the correlated assumption, explain
why the standard errors reported in the R output are unreliable.
(iii) Make a plot of the residuals after fitting the linear model in (i). Make a plot of the
first differences. What do you notice about the two plots, similar?
(What I found was quite strange)
The AIC (Akaike Information Criterion) is usually used to select the parameters in the model
(see wiki). You should have studied the AIC/AICc/BIC in several of the prerequists you
have taken. In this course it will be assumed that you are familiar with it.
1.2.2
Estimation using nonparametric methods
In Section 1.2.1 we assumed that the mean had a certain known parametric form. This may
not always be the case. If we have no apriori idea of what features may be in the mean, we
can estimate the mean trend using a nonparametric approach. If we do not have any apriori
knowledge of the mean function we cannot estimate it without placing some assumptions on
it’s structure. The most common is to assume that the mean µt is a sample from a ‘smooth’
function, ie. µt = µ( nt ). Under this assumption the following approaches are valid.
Possibly one of the most simplest methods is to use a ‘rolling window’. There are several
windows that one can use. We describe, below, the exponential window, since it can be
12
‘evaluated’ in an online way. For t = 1 let µ
ˆ1 = Y1 , then for t > 1 define
µ
ˆt = (1 − λ)ˆ
µt−1 + λYt ,
where 0 < λ < 1. The choice of λ depends on how much weight one wants to give the present
observation. It is straightforward to show that
µ
ˆt =
t−1
X
(1 − λ)
t−j
λYj =
t
X
j=1
[1 − exp(−γ)] exp [−γ(t − j)] Yj
j=1
where γ = − log(1 − λ). Let b = 1/γ and K(u) = exp(−u)I(u ≥ 0), then µ
ˆt can be written
as
1/b
µ
ˆt = (1 − e )
| {z }
≈b−1
n
X
K
j=1
t−j
b
Yj ,
This we observe that the exponential rolling window estimator is very close to a nonparametric kernel estimator of the mean, which has the form
µ
˜t =
n
X
1
j=1
b
K
t−j
b
Yj .
it is likely you came across such estimators in your nonparametric classes. The main difference between the rolling window estimator and the nonparametric kernel estimator is that
the kernel/window for the rolling window is not symmetric. This is because we are trying
to estimate the mean at time t, given only the observations up to time t. Whereas for nonparametric kernel estimators we can be observations on both sides of the neighbourhood of
t.
Other type of estimators include sieve-estimators. This is where we expand µ(u) in terms
of an orthogonal basis {φk (u); k ∈ Z}’
µ(u) =
∞
X
ak φk (u).
k=1
13
Examples of basis functions are the Fourier φk (u) = exp(iku), Haar/other wavelet functions
etc. We observe that the unknown coefficients ak are a linear in the ‘regressors’ φk . Thus
we can use least squares to estimate the coefficients, {ak }. To estimate these coefficients, we
truncate the above expansion to order M , and use least squares to estimate the coefficients
n
X
t=1
"
#2
t
Yt −
ak φk ( ) .
n
k=1
M
X
(1.3)
The orthogonality of the basis means that the least squares estimator a
ˆk is
n
1X
Yt φk
a
ˆk ≈
n t=1
t
.
n
It is worth pointing out that regardless of the method used, correlations in the errors
{εt } will play an role in quality of the estimator and even on the choice of bandwidth, b, or
equivalently the number of basis functions, M (see Hart (1991)). To understand why, suppose
t
the mean function is µt = µ( 200
) (the sample size n = 200), where µ(u) = 5×(2u−2.5u2 )+20.
We corrupt this quadratic function with both iid and dependent noise (the dependent noise is
the AR(2) process defined in equation (1.6)). The plots are given in Figure 1.7. We observe
that the dependent noise looks ‘smooth’ (dependent can induce smoothness in a realisation).
This means that in the case that the mean has been corrupted by dependent noise it difficult
to see that the underlying trend is a simple quadratic function.
1.2.3
Estimation of the period
Suppose that the observations {Yt ; t = 1, . . . , n} satisfy the following regression model
Yt = A cos(ωt) + B sin(ωt) + εt
where {εt } are iid standard normal random variables and 0 < ω < π. The parameters A, B,
and ω are real and unknown. Unlike the regression models given in (1.2.1) the model here
is nonlinear, since the unknown parameter, ω, is inside a trignometric function. Standard
least squares methods cannot be used to estimate the parameters. Assuming Gaussianity of
14
3
2
3
1
2
−3 −2 −1
0
ar2
1
0
−2 −1
iid
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.6
0.8
1.0
0.8
1.0
temp
22
quadraticar2
18
20
24
22
20
18
16
16
quadraticiid
0.4
24
temp
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
temp
0.4
0.6
temp
Figure 1.7: Top: realisations from iid random noise and dependent noise (left = iid and right
= dependent). Bottom: Quadratic trend plus corresponding noise.
{εt } the maximum likelihood corresponding to the model is
n
1X
Ln (A, B, ω) = −
(Yt − A cos(ωt) − B sin(ωt))2 .
2 t=1
Nonlinear least squares method (which would require the use of a numerical maximisation
scheme) can be employed to estimate the parameters. However, using some algebraic manipulations, explicit expressions for the estimators can be obtained (see Walker (1971) and
Exercise 1.3). These are
ω
ˆ n = arg max In (ω)
ω
where
2
n
1 X
In (ω) = Yt exp(itω)
n t=1
15
(1.4)
(we look for the maximum over the fundamental frequencies ωk =
n
2πk
n
for 1 ≤ k ≤ n),
n
X
2X
ˆn = 2
Yt cos(ˆ
ωn t) and B
Yt sin(ˆ
ωn t).
Aˆn =
n t=1
n t=1
The rather remarkable aspect of this result is that the rate of convergence of |ˆ
ωn − ω| =
O(n−3/2 ), which is faster than the standard O(n−1/2 ) that we usually encounter (we will see
this in Example 1.2.1).
In (ω) is usually called the periodogram. Searching for peaks in the periodogram is a long
established method for detecting periodicities. If we believe that there were two or more
periods in the time series, we can generalize the method to searching for the largest and
second largest peak etc. We consider an example below.
Example 1.2.1 Consider the following model
Yt = 2 sin
2πt
8
+ εt
t = 1, . . . , n.
(1.5)
where εt are iid standard normal random variables. It is clear that {Yt } is made up of
a periodic signal with period eight. We make a plot of one realisation (using sample size
n = 128) together with the periodogram I(ω) (defined in (1.4)). In Figure 1.8 we give a plot of
one realisation together with a plot of the periodogram. We observe that there is a symmetry,
this is because of the eiω in the definition of I(ω) we can show that I(ω) = I(2π − ω). Notice
there is a clear peak at frequency 2π/8 ≈ 0.78 (where we recall that 8 is the period).
This method works extremely well if the error process {εt } is uncorrelated. However, problems arise when the errors are correlated. To illustrate this issue, consider again model (1.5)
but this time let us suppose the errors are correlated. More precisely, they satisfy the AR(2)
model,
εt = 1.5εt−1 − 0.75εt−2 + t ,
(1.6)
where {t } are iid random variables (do not worry if this does not make sense to you we
define this class of models precisely in Chapter 2). As in the iid case we use a sample size
16
1 2 3
−1
−3
signal
0
20
40
60
80
100
120
Time
●
0.2
0.0
P
0.4
●
●
●
●
● ●
●●
●● ●
●●
● ● ● ●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●●●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●●
0
1
2
3
4
5
6
frequency
Figure 1.8: Left: Realisation of (1.5) with iid noise, Right: Periodogram
n = 128. In Figure 1.9 we give a plot of one realisation and the corresponding periodogram.
We observe that the peak at 2π/8 is not the highest. The correlated errors (often called
0
−5
signal2
5
coloured noise) is masking the peak by introducing new peaks. To see what happens for
0
20
40
60
80
100
120
Time
●
●
●
●
●
●
0.4
0.0
P2
0.8
●
●
●
●
●
● ●●
● ●
●●
0
●
●
●
●●
●●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
1
2
3
4
5
●
●
● ●
●
● ●
●
6
frequency
Figure 1.9: Left: Realisation of (1.5) with correlated noise and n = 128, Right: Periodogram
larger sample sizes, we consider exactly the same model (1.5) with the noise generated as
17
in (1.6). But this time we use n = 1024 (8 time the previous sample size). A plot of one
realisation, together with the periodogram is given in Figure 1.10. In contrast to the smaller
0
−5
signal2
5
sample size, a large peak is seen at 2π/8. These examples illustrates two important points:
0
200
400
600
800
1000
Time
0.0 0.2 0.4 0.6
P2
●
●
●
●
●
●
●
●
●
● ●●
●●
●
●
●
●●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0
1
2
3
4
5
6
frequency
Figure 1.10: Left: Realisation of (1.5) with correlated noise and n = 1024, Right: Periodogram
(i) When the noise is correlated and the sample size is relatively small it is difficult to
disentangle the deterministic period from the noise. Indeed we will show in Chapters
2 and 3 that linear time series can exhibit similar types of behaviour to a periodic
deterministic signal. This is a subject of on going research that dates back at least 60
years (see Quinn and Hannan (2001)).
However, the similarity is only to a point. Given a large enough sample size (which
may in practice not be realistic), the deterministic frequency dominates again.
(ii) The periodogram holds important properties about the correlations in the noise (observe the periodogram in both Figures 1.9 and 1.10), there is some interesting activity
in the lower frequencies, that appear to be due to noise.
This is called spectral analysis and is explored in Chapters ?? and ??. Indeed a lot of
time series analysis can be done within the so called frequency or time domain.
18
Exercise 1.2 (Understanding Fourier transforms)
(i) Let Yt = 1. Plot the Peri-
odogram of {Yt ; t = 1, . . . , 128}.
(ii) Let Yt = 1 + εt , where {εt } are iid standard normal random variables. Plot the Periodogram of {Yt ; t = 1, . . . , 128}.
t
(iii) Let Yt = µ( 128
) where µ(u) = 5 × (2u − 2.5u2 ) + 20. Plot the Periodogram of {Yt ; t =
1, . . . , 128}.
(iv) Let Yt = 2 × sin( 2πt
). Plot the Periodogram of {Yt ; t = 1, . . . , 128}.
8
(v) Let Yt = 2 × sin( 2πt
) + 4 × cos( 2πt
). Plot the Periodogram of {Yt ; t = 1, . . . , 128}.
8
12
Exercise 1.3
(i) Let
Sn (A, B, ω) =
X
n
Yt2
−2
t=1
n
X
t=1
1
2
2
Yt A cos(ωt) + B sin(ωt) + n(A + B ) .
2
Show that
n
n
2Ln (A, B, ω) + Sn (A, B, ω) =
X
(A2 − B 2 ) X
sin(2tω).
cos(2tω) + AB
2
t=1
t=1
and thus |Ln (A, B, ω) + 12 Sn (A, B, ω)| = O(1) (ie. the difference does not grow with
n).
Since Ln (A, B, ω) and − 12 Sn (A, B, ω) are asymptotically equivalent (i) shows that we
can maximise
−1
S (A, B, ω)
2 n
instead of the likelihood Ln (A, B, ω).
(ii) By profiling out the parameters A and B, use the the profile likelihood to show that
P
ω
ˆ n = arg maxω | nt=1 Yt exp(itω)|2 .
(iii) By using the identity (which is the one-sided Dirichlet kernel)
n
X
t=1
exp(iΩt) =


exp( 12 i(n+1)Ω) sin( 12 nΩ)
sin( 12 Ω)
0 < Ω < 2π

n
Ω = 0 or 2π.
19
(1.7)
we can show that for 0 < Ω < 2π we have
n
X
t cos(Ωt) = O(n)
t=1
n
X
t2 cos(Ωt) = O(n2 )
n
X
t sin(Ωt) = O(n)
t=1
n
X
t=1
t2 sin(Ωt) = O(n2 ).
t=1
Using the above identities, show that the Fisher Information of Ln (A, B, ω) (denoted
as I(A, B, ω)) is asymptotically equivalent to

2I(A, B, ω) = E
∂ 2 Sn 

=
∂ω 2

n2
2
n
2
0
n2
B
2
0
n
2
2
− n2 A
n2
B + O(n) − 2 A + O(n)
n3
3
+ O(n)
+ O(n)
(A2 + B 2 ) + O(n2 )



.

(iv) Use the Fisher information to show that |ˆ
ωn − ω| = O(n−3/2 ).
Exercise 1.4
(i) Simulate three hundred times from model (1.5) using n = 128. Esti-
mate ω, A and B for each simulation and obtain the empirical mean squared error
P300 ˆ
1
2
ˆ
i=1 (θi − θ) (where θ denotes the parameter and θi the estimate).
300
In your simulations, is the estimate of the period, ω superior to the estimator of coefficients, A and B?
(ii) Do the same as above but now use coloured noise given in (1.6) as the errors. How do
your estimates compare to (i)?
R Code
Simulation and periodogram for model (1.5) with iid errors:
temp <− rnorm ( 1 2 8 )
s i g n a l <− 1 . 5 ∗ s i n ( 2 ∗ p i ∗ c ( 1 : 1 2 8 ) / 8 ) + temp # t h i s s i m u l a t e s t h e s e r i e s
# Use t h e command f f t t o make t h e periodogram
P <− abs ( f f t ( s i g n a l ) / 1 2 8 ) ∗ ∗ 2
f r e q u e n c y <− 2∗ p i ∗ c ( 0 : 1 2 7 ) / 1 2 8
# To p l o t t h e s e r i e s and periodogram
20
par ( mfrow=c ( 2 , 1 ) )
plot . ts ( signal )
p l o t ( f r e q u e n c y , P , type=”o ” )
Simulation and periodogram for model (1.5) with correlated errors:
set . seed (10)
a r 2 <− arima . sim ( l i s t ( o r d e r=c ( 2 , 0 , 0 ) , a r = c ( 1 . 5 , − 0 . 7 5 ) ) , n=128)
s i g n a l 2 <− 1 . 5 ∗ s i n ( 2 ∗ p i ∗ c ( 1 : 1 2 8 ) / 8 ) + a r 2
P2 <− abs ( f f t ( s i g n a l 2 ) / 1 2 8 ) ∗ ∗ 2
f r e q u e n c y <− 2∗ p i ∗ c ( 0 : 1 2 7 ) / 1 2 8
par ( mfrow=c ( 2 , 1 ) )
plot . ts ( signal2 )
p l o t ( f r e q u e n c y , P2 , type=”o ” )
1.3
Some formalism
When we observe the time series {xt }, usually we assume that {xt } is a realisation from a random
process {Xt }. We formalise this notion below. The random process {Xt ; t ∈ Z} (where Z denotes
the integers) is defined on the probability space {Ω, F, P }. We explain what these mean below:
(i) Ω is the set of all possible outcomes. Suppose that ω ∈ Ω, then {Xt (ω)} is one realisation
from the random process. For any given ω, {Xt (ω)} is not random. In time series we will
usually assume that what we observe xt = Xt (ω) (for some ω) is a typical realisation. That
is, for any other ω ∗ ∈ Ω, Xt (ω ∗ ) will be different, but its general or overall characteristics
will be similar.
(ii) F is known as a sigma algebra. It is a set of subsets of Ω (though not necessarily the set of
all subsets, as this can be too large). But it consists of all sets for which a probability can
be assigned. That is if A ∈ F, then a probability is assigned to the set A.
(iii) P is the probability.
Different types of convergence we will be using in class:
21
a.s.
(i) Almost sure convergence: Xn → a as n → ∞ (in this course a will always be a constant).
This means for every ω ∈ Ω Xn → a, where P (Ω) = 1 (this is classical limit of a sequence,
see Wiki for a definition).
P
(ii) Convergence in probability: Xn → a. This means that for every ε > 0, P (|Xn − a| > ε) → 0
as n → ∞ (see Wiki)
2
(iii) Convergence in mean square Xn → a. This means E|Xn − a|2 → 0 as n → ∞ (see Wiki).
(iv) Convergence in distribution. This means the distribution of Xn converges to the distribution
of X, ie. for all x where FX is continuous, we have Fn (x) → FX (x) as n → ∞ (where Fn and
FX are the distribution functions of Xn and X respectively). This is the simplest definition
(see Wiki).
• Which implies which?
– (i), (ii) and (iii) imply (iv).
– (i) implies (ii).
– (iii) implies (ii).
• Central limit theorems require (iv). It is often easiest to show (iii) (since this only requires
mean and variance calculations).
1.4
Estimating the mean
Based on one realisation of a time series we want to make inference about parameters associated
with the process {Xt }, such as the mean etc. Let us consider the simplest case, estimating the
mean. We recall that in classical statistics we usually assume we observe several independent
realisations, {Xt } from a random variable X, and use the multiple realisations to make inference
¯ = 1 Pn Xk . Roughly speaking, by using several independent realisations we
about the mean: X
k=1
n
are sampling over the entire probability space and obtaining a good estimate of the mean. On the
other hand if the samples were highly dependent, then it is likely that {Xt } would be concentrated
over small parts of the probability space. In this case, the variance of the sample mean would not
converge to zero as the sample size grows.
22
A typical time series is a half way house between totally dependent data and independent data.
Unlike, classical statistics, in time series, parameter estimation is based on only one realisation
xt = Xt (ω) (not multiple, independent, replications). Therefore, it would appear impossible to
obtain a good estimator of the mean. However good estimates, of the mean, can be made, based
on just one realisation so long as certain assumptions are satisfied (i) the process has a constant
mean (a type of stationarity) and (ii) despite the fact that each time series is generated from one
realisation there is ‘short’ memory in the observations. That is, what is observed today, xt has
little influence on observations in the future, xt+k (when k is relatively large). Hence, even though
we observe one tragectory, that trajectory traverses much of the probability space. The amount
of dependency in the time series determines the ‘quality’ of the estimator. There are several ways
to measure the dependency. We know that the most common measure of linear dependency is the
covariance. The covariance in the stochastic process {Xt } is defined as
cov(Xt , Xt+k ) = E(Xt Xt+k ) − E(Xt )E(Xt+k ).
Noting that if {Xt } has zero mean, then the above reduces to cov(Xt , Xt+k ) = E(Xt Xt+k ).
Remark 1.4.1 It is worth bearing in mind that the covariance only measures linear dependence.
For some statistical analysis, such as deriving an expression for the variance of an estimator, the
covariance is often sufficient as a measure. However, given cov(Xt , Xt+k ) we cannot say anything
about cov(g(Xt ), g(Xt+k )), where g is a nonlinear function. There are occassions where we require
a more general measure of dependence (for example, to show asymptotic normality). Examples of
more general measures include mixing (and other related notions, such as Mixingales, Near-Epoch
dependence, approximate m-dependence, physical dependence, weak dependence), first introduced by
Rosenblatt in the 50s (M. and Grenander (1997)). In this course we will not cover mixing.
Returning to the sample mean example suppose that {Xt } is a time series. In order to estimate the
mean we need to be sure that the mean is constant over time (else the estimator will be meaningless).
Therefore we will assume that {Xt } is a time series with constant mean µ. We observe {Xt }nt=1
¯ = 1 Pn Xt . It is clear that this is an unbiased
and estimate the mean µ with the sample mean X
t=1
n
¯ = µ (it is unbiased). Thus to see whether it converges in mean square
estimator of µ, since E(X)
23
to µ we consider its variance
¯ =
var(X)
n
n
n
1 X
2 X X
var(X
)
+
cov(Xt , Xτ ).
t
n2
n2
t=1
(1.8)
t=1 τ =t+1
If the covariance structure decays at such a rate that the sum of all lags is finite (supt
P∞
τ =−∞ |cov(Xt , Xτ )|
∞, often called short memory), then the variance is O( n1 ), just as in the iid case. However, even
¯ in order to test/construct CI for µ.
with this assumption we need to be able to estimate var(X)
Usually this requires the stronger assumption of stationarity, which we define in Section 1.5
Example 1.4.1 (The variance of a regression model with correlated errors) Let us return
to the parametric models discussed in Section 1.2.1. The general model is
Yt = β0 +
p
X
βj ut,j + εt = β 0 ut + εt ,
j=1
where E[εt ] = 0 and we will assume that {ut,j } are nonrandom regressors. Note this includes the
parametric trend models discussed in Section 1.2.1. We use least squares to estimate β
Ln (β) =
n
X
(Yt − β 0 ut )2 ,
t=1
with
βˆn = arg min Ln (β) = (
n
X
t=1
Thus
ˆ )
∂Ln (β
n
∂β
ut u0t )−1
n
X
Yt ut .
t=1
ˆ we will derive an expression for β
ˆ − β (this
= 0. To evaluate the variance of β
n
n
expression also applies to many nonlinear estimators too). We note that by using straightforward
algebra we can show that
n
i0 X
ˆ ) ∂Ln (β) h
∂Ln (β
n
ˆ −β
ut u0t .
−
= β
n
∂β
∂β
t=1
24
(1.9)
<
Moreoover because
ˆ )
∂Ln (β
n
∂β
= 0 we have
ˆ ) ∂Ln (β)
∂Ln (β
n
−
∂β
∂β
∂Ln (β)
∂β
n
n
X
X
u t εt .
=
[Yt − β 0 ut ] ut =
| {z }
= −
t=1
(1.10)
t=1
εt
Altogether (1.9) and (1.10) give
h
ˆ −β
β
n
n
i0 X
ut u0t =
t=1
n
X
u0t εt .
t=1
and
h
n
X
i
ˆ −β =
β
n
!−1
ut u0t
t=1
n
X
u t εt .
t=1
Using this expression we can see that
h
n
i
ˆ −β =
var β
n
1X
ut u0t
n
!−1
n
var
t=1
1
n
Finally we need only evaluate var
n
var
1X
u t εt
n
!
=
t=1
=
Pn
t=1 ut εt
1X
u t εt
n
t=1
!
n
1X
ut u0t
n
!−1
.
t=1
which is
n
1 X
cov[εt , ετ ]ut u0τ
n2
t,τ =1
n
X
1
n2
|
var[εt ]ut u0t +
t=1
{z
}
expression if independent
n
n
2 X X
cov[εt , ετ ]ut u0τ
n2
t=1 τ =t+1
|
{z
}
.
additional term due to correlation in the errors
P
P
Under the assumption that n1 nt=1 ut u0t is non-singular, supt kut k1 < ∞ and supt ∞
τ =−∞ |cov(εt , ετ )| <
h
i
ˆ − β = O(n−1 ), but just as in the case of the sample mean we need to
∞, we can see that var β
n
impose some additional conditions on {εt } we want to construct confidence intervals/test for β.
1.5
Stationary processes
We have established that one of the main features that distinguish time series analysis from classical
methods is that observations taken over time (a time series) can be dependent and this dependency
25
tends to decline the further apart in time these two observations. However, to do any sort of analysis
of this time series we have to assume some sort of invariance in the time series, for example the mean
or variance of the time series does not change over time. If the marginal distributions of the time
series were totally different no sort of inference would be possible (suppose in classical statistics you
were given independent random variables all with different distributions, what parameter would
you be estimating, it is not possible to estimate anything!).
The typical assumption that is made is that a time series is stationary. Stationarity is a rather
intuitive concept, it is an invariant property which means that statistical characteristics of the time
series do not change over time. For example, the yearly rainfall may vary year by year, but the
average rainfall in two equal length time intervals will be roughly the same as would the number of
times the rainfall exceeds a certain threshold. Of course, over long periods of time this assumption
may not be so plausible. For example, the climate change that we are currently experiencing is
causing changes in the overall weather patterns (we will consider nonstationary time series towards
the end of this course). However in many situations, including short time intervals, the assumption
of stationarity is quite a plausible. Indeed often the statistical analysis of a time series is done
under the assumption that a time series is stationary.
1.5.1
Types of stationarity (with Ergodicity thrown in)
There are two definitions of stationarity, weak stationarity which only concerns the covariance of a
process and strict stationarity which is a much stronger condition and supposes the distributions
are invariant over time.
Definition 1.5.1 (Strict stationarity) The time series {Xt } is said to be strictly stationary
if for any finite sequence of integers t1 , . . . , tk and shift h the distribution of (Xt1 , . . . , Xtk ) and
(Xt1 +h , . . . , Xtk +h ) are the same.
The above assumption is often considered to be rather strong (and given a data it is very
hard to check). Often it is possible to work under a weaker assumption called weak/second order
stationarity.
Definition 1.5.2 (Second order stationarity/weak stationarity) The time series {Xt } is said
to be second order stationary if the mean is constant for all t and if for any t and k the covariance
between Xt and Xt+k only depends on the lag difference k. In other words there exists a function
26
c : Z → R such that for all t and k we have
c(k) = cov(Xt , Xt+k ).
Remark 1.5.1 (Strict and second order stationarity)
(i) If a process is strictly stationar-
ity and E|Xt2 | < ∞, then it is also second order stationary. But the converse is not necessarily
true. To show that strict stationarity (with E|Xt2 | < ∞) implies second order stationarity,
suppose that {Xt } is a strictly stationary process, then
cov(Xt , Xt+k ) = E(Xt Xt+k ) − E(Xt )E(Xt+k )
Z
=
xy PXt ,Xt+k (dx, dy) − PXt (dx)PXt+k (dy)
Z
=
xy [PX0 ,Xk (dx, dy) − PX0 (dx)PXk (dy)] = cov(X0 , Xk ),
where PXt ,Xt+k and PXt is the joint distribution and marginal distribution of Xt , Xt+k respectively. The above shows that cov(Xt , Xt+k ) does not depend on t and {Xt } is second order
stationary.
(ii) If a process is strictly stationary but the second moment is not finite, then it is not second
order stationary.
(iii) It should be noted that a weakly stationary Gaussian time series is also strictly stationary too
(this is the only case where weakly stationary implies strictly stationary).
Example 1.5.1 (The sample mean and it’s variance under stationarity) Returning the variance of the sample mean discussed (1.8), if a time series is second order stationary, then the sample
¯ is estimating the mean µ and the variance of X
¯ is
mean X
¯ =
var(X)
n
n
1
2 X X
var(X0 ) + 2
cov(Xt , Xτ )
n
T
t=1 τ =t+1
=
n
1
2 X n − r
var(X0 ) +
cov(X0 , Xr ) .
|
{z
}
n | {z } n
n
r=1
c(0)
c(r)
P
We approximate the above, by using that the covariances
r |c(r)| < ∞. Therefore for all r,
Pn
P
(1 − r/n)c(r) → c(r) and | r=1 (1 − |r|/n)c(r)| ≤ r |c(r)|, thus by dominated convergence (see
27
Chapter A)
Pn
r=1 (1
− r/n)c(r) →
P∞
r=1 c(r).
This implies that
∞
¯ ≈
var(X)
1
2X
1
c(0) +
c(r) = O( ).
n
n
n
r=1
The above is often called the long term variance. The above implies that
¯ − µ)2 = var(X)
¯ → 0,
E(X
P
¯→
which we recall is convergence in mean square. Thus we have convergence in probability X
µ.
The example above illustrates how second order stationarity gave a rather elegant expression
for the variance. We now motivate the concept of ergodicity.
Sometimes, it is difficult to evaluate the mean and variance of an estimator, but often we
may only require almost sure or convergence in probability. Therefore, we may want to find an
alternative method to evaluating the mean squared error. To see whether this is possible we recall
that for iid random variables we have the very useful law of large numbers
n
1X
a.s.
Xt → µ
n
t=1
and in general
1
n
Pn
t=1 g(Xt )
a.s.
→ E[g(X0 )] (if E[g(X0 )] < ∞). Does such a result exists in time
series? It does, but we require the slightly stronger condition that a time series is ergodic (which
is a slightly stronger condition than the strictly stationary).
Definition 1.5.3 (Ergodicity - very rough) Let (Ω, F, P ) be a probability space. A transformation T : Ω → Ω is said to be measure preserving if for every set A ∈ F, P (T −1 A) = P (A).
Moreover, it is said to be an ergodic transformation if T −1 A = A implies that P (A) = 0 or 1.
It is not obvious what this has to do with stochastic processes, but we attempt to make a link. Let
us suppose that X = {Xt } is a strictly stationary process defined on the probability space (Ω, F, P ).
By strict stationarity the transformation (shifting a sequence by one)
T (x1 , x2 , . . .) = (x2 , x3 , . . .),
is a measure preserving transformation. Thus a process which is stationarity is measure preserving.
28
To understand ergodicity we define the set A, where
A = {ω : (X1 (ω), X0 (ω), . . .) ∈ H}. = {ω : X−1 (ω), . . . , X−2 (ω), . . .) ∈ H}.
The stochastic process is said to be ergodic, if the only sets which satisfies the above are such that
P (A) = 0 or 1. Roughly, this means there cannot be too many outcomes ω which generate sequences
which ‘repeat’ itself (are periodic in some sense).
See Billingsley (1994), page 312-314, for examples and a better explanation.
The definition of ergodicity, given above, is quite complex and is rarely used in time series analysis.
However, one consequence of ergodicity is the ergodic theorem, which is extremely useful in time
series. It states that if {Xt } is an ergodic stochastic process then
n
1X
a.s.
g(Xt ) → E[g(X0 )]
n
t=1
for any function g(·). And in general for any shift τ1 , . . . , τk and function g : Rk+1 → R we have
n
1X
a.s.
g(Xt , Xt+τ1 , . . . , Xt+τk ) → E[g(X0 , . . . , Xt+τk )]
n
(1.11)
t=1
(often (1.11) is used as the definition of ergodicity, as it is an iff with the ergodic definition). Later
you will see how useful this.
(1.11) gives us an idea of what constitutes an ergodic process. Suppose that {εt } is an ergodic
process (a classical example are iid random variables) then any reasonable (meaning measurable)
function of Xt is also ergodic. More precisely, if Xt is defined as
Xt = h(. . . , εt , εt−1 , . . .),
(1.12)
where {εt } are iid random variables and h(·) is a measureable function, then {Xt } is an Ergodic
process. For full details see Stout (1974), Theorem 3.4.5.
Remark 1.5.2 As mentioned above all Ergodic processes are stationary, but a stationary process
is not necessarily ergodic. Here is one simple example. Suppose that {εt } are iid random variables
and Z is a Bernoulli random variable with outcomes {1, 2} (where the chance of either outcome is
29
half ). Suppose that Z stays the same for all t. Define

 µ 1 + εt Z = 1
Xt =
 µ + ε Z = 2.
2
t
It is clear that E(Xt |Z = i) = µi and E(Xt ) = 12 (µ1 + µ2 ). This sequence is stationary. However,
P
we observe that T1 Tt=1 Xt will only converge to one of the means, hence we do not have almost
sure convergence (or convergence in probability) to 12 (µ1 + µ2 ).
Exercise 1.5 State, with explanation, which of the following time series is second order stationary,
which are strictly stationary and which are both.
(i) {εt } are iid random variables with mean zero and variance one.
(ii) {εt } are iid random variables from a Cauchy distributon.
(iii) Xt+1 = Xt + εt , where {εt } are iid random variables with mean zero and variance one.
(iv) Xt = Y where Y is a random variable with mean zero and variance one.
(iv) Xt = Ut +Ut−1 +Vt , where {(Ut , Vt )} is a strictly stationary vector time series with E[Ut2 ] < ∞
and E[Vt2 ] < ∞.
Example 1.5.2 In Chapter 6 we consider estimation of the autocovariance function. However for
now use R command acf. In Figure 1.11 we give the sample acf plots of the Southern Oscillation
Index and the Sunspot data. We observe that are very different. The acf of the SOI decays rapidly,
but there does appear to be some sort of ‘pattern’ in the correlations. On the other hand, there is
more persistence in the acf of the Sunspot data. The correlations of the acf data appear to decay
but over a longer period of time and there is a clear periodicity in the correlation.
Exercise 1.6
(i) Make an ACF plot of the monthly temperature data from 1996-2014.
(ii) Make and ACF plot of the yearly temperature data from 1880-2013.
(iii) Make and ACF plot of the residuals (after fitting a line through the data (using the command
lsfit(..)$res)) of the yearly temperature data from 1880-2013.
Briefly describe what you see.
30
0.4
0.0
ACF
0.8
Series soi
0
50
100
150
200
250
300
40
50
60
Lag
−0.4 0.0 0.4 0.8
ACF
Series sunspot
0
10
20
30
Lag
Figure 1.11: Top: ACF of Southern Oscillation data. Bottom ACF plot of Sunspot data.
R code
To make the above plots we use the commands
par ( mfrow=c ( 2 , 1 ) )
a c f ( s o i , l a g . max=300)
a c f ( sunspot , l a g . max=60)
1.5.2
Towards statistical inference for time series
Returning to the sample mean Example 1.5.1. Suppose we want to construct CIs or apply statistical
tests on the mean. This requires us to estimate the long run variance (assuming stationarity)
∞
¯ ≈
var(X)
1
2X
c(0) +
c(r).
n
n
r=1
There are several ways this can be done, either by fitting a model to the data and from the model
estimate the covariance or doing it nonparametrically. This example motivates the contents of the
course:
(i) Modelling, finding suitable time series models to fit to the data.
(ii) Forecasting, this is essentially predicting the future given current and past observations.
31
(iii) Estimation of the parameters in the time series model.
(iv) The spectral density function and frequency domain approaches, sometimes within the frequency domain time series methods become extremely elegant.
(v) Analysis of nonstationary time series.
(vi) Analysis of nonlinear time series.
(vii) How to derive sampling properties.
1.6
What makes a covariance a covariance?
The covariance of a stationary process has several very interesting properties. The most important
is that it is positive semi-definite, which we define below.
Definition 1.6.1 (Positive semi-definite sequence)
(i) A sequence {c(k); k ∈ Z} (Z is the
set of all integers) is said to be positive semi-definite if for any n ∈ Z and sequence x =
(x1 , . . . , xn ) ∈ Rn the following is satisfied
n
X
c(i − j)xi xj ≥ 0.
i,j=1
(ii) A function is said to be an even positive semi-definite sequence if (i) is satisfied and c(k) =
c(−k) for all k ∈ Z.
An extension of this notion is the positive semi-definite function.
Definition 1.6.2 (Positive semi-definite function)
(i) A function {c(u); u ∈ R} is said to
be positive semi-definite if for any n ∈ Z and sequence x = (x1 , . . . , xn ) ∈ Rn the following
is satisfied
n
X
c(ui − uj )xi xj ≥ 0.
i,j=1
(ii) A function is said to be an even positive semi-definite function if (i) is satisfied and c(u) =
c(−u) for all u ∈ R.
32
Remark 1.6.1 You have probably encountered this positive definite notion before, when dealing
with positive definite matrices. Recall the n × n matrix Σn is positive semi-definite if for all x ∈ Rn
x0 Σn x ≥ 0. To see how this is related to positive semi-definite matrices, suppose that the matrix Σn
P
has a special form, that is the elements of Σn are (Σn )i,j = c(i−j). Then x0 Σn x = ni,j c(i−j)xi xj .
We observe that in the case that {Xt } is a stationary process with covariance c(k), the variance
covariance matrix of X n = (X1 , . . . , Xn ) is Σn , where (Σn )i,j = c(i − j).
We now take the above remark further and show that the covariance of a stationary process is
positive semi-definite.
Theorem 1.6.1 Suppose that {Xt } is a discrete time/continuous stationary time series with covariance function {c(k)}, then {c(k)} is a positive semi-definite sequence/function. Conversely for
any even positive semi-definite sequence/function there exists a stationary time series with this
positive semi-definite sequence/function as its covariance function.
PROOF. We prove the result in the case that {Xt } is a discrete time time series, ie. {Xt ; t ∈ Z}.
We first show that {c(k)} is a positive semi-definite sequence. Consider any sequence x =
Pn
(x1 , . . . , xn ) ∈ Rn , and the double sum
i,j xi c(i − j)xj . Define the random variable Y =
Pn
Pn
0
i=1 xi Xi . It is straightforward to see that var(Y ) = x var(X n )x =
i,j=1 c(i−j)xi xj where X n =
P
(X1 , . . . , Xn ). Since for any random variable Y , var(Y ) ≥ 0, this means that ni,j=1 xi c(i−j)xj ≥ 0,
hence {c(k)} is a positive definite sequence.
To show the converse, that is for any positive semi-definite sequence {c(k)} we can find a
corresponding stationary time series with the covariance {c(k)} is relatively straightfoward, but
depends on defining the characteristic function of a process and using Komologorov’s extension
theorem. We omit the details but refer an interested reader to Brockwell and Davis (1998), Section
1.5.
In time series analysis usually the data is analysed by fitting a model to the data. The model
(so long as it is correctly specified, we will see what this means in later chapters) guarantees the
covariance function corresponding to the model (again we cover this in later chapters) is positive
definite. This means, in general we do not have to worry about positive definiteness of the covariance
function, as it is implicitly implied.
On the other hand, in spatial statistics, often the object of interest is the covariance function
and specific classes of covariance functions are fitted to the data. In which case it is necessary to
33
ensure that the covariance function is semi-positive definite (noting that once a covariance function
has been found by Theorem 1.6.1 there must exist a spatial process which has this covariance
function). It is impossible to check for positive definiteness using Definitions 1.6.1 or 1.6.1. Instead
an alternative but equivalent criterion is used. The general result, which does not impose any
conditions on {c(k)} is stated in terms of positive measures (this result is often called Bochner’s
theorem). Instead, we place some conditions on {c(k)}, and state a simpler version of the theorem.
Theorem 1.6.2 Suppose the coefficients {c(k); k ∈ Z} are absolutely summable (that is
P
k
|c(k)| <
∞). Then the sequence {c(k)} is positive semi-definite if an only if the function f (ω), where
f (ω) =
∞
1 X
c(k) exp(ikω),
2π
k=−∞
is nonnegative for all ω ∈ [0, 2π].
We also state a variant of this result for positive semi-definite functions. Suppose the function
R
{c(u); k ∈ R} is absolutely summable (that is R |c(u)|du < ∞). Then the function {c(u)} is positive
semi-definite if and only if the function f (ω), where
1
f (ω) =
2π
Z
∞
c(u) exp(iuω)du,
−∞
is nonnegative for all ω ∈ R.
PROOF. See Section ??.
Example 1.6.1 We will show that sequence c(0) = 1, c(1) = 0.5, c(−1) = 0.5 and c(k) = 0 for
|k| > 1 a positive definite sequence.
From the definition of spectral density given above we see that the ‘spectral density’ corresponding
to the above sequence is
f (ω) = 1 + 2 × 0.5 × cos(ω).
Since | cos(ω)| ≤ 1, f (ω) ≥ 0, thus the sequence is positive definite. An alternative method is to
find a model which has this as the covariance structure. Let Xt = εt + εt−1 , where εt are iid random
variables with E[εt ] = 0 and var(εt ) = 0.5. This model has this covariance structure.
34
We note that Theorem 1.6.2 can easily be generalized to higher dimensions, d, by taking Fourier
transforms over Zd or Rd .
Exercise 1.7 Which of these sequences can used as the autocovariance function of a second order
stationary time series?
(i) c(−1) = 1/2, c(0) = 1, c(1) = 1/2 and for all |k| > 1, c(k) = 0.
(ii) c(−1) = −1/2, c(0) = 1, c(1) = 1/2 and for all |k| > 1, c(k) = 0.
(iii) c(−2) = −0.8, c(−1) = 0.5, c(0) = 1, c(1) = 0.5 and c(2) = −0.8 and for all |k| > 2,
c(k) = 0.
Exercise 1.8
(i) Show that the function c(u) = exp(−a|u|) where a > 0 is a positive semi-
definite function.
(ii) Show that the commonly used exponential spatial covariance defined on R2 , c(u1 , u2 ) =
p
exp(−a u21 + u22 ), where a > 0, is a positive semi-definite function.
35
Chapter 2
Linear time series
Prerequisites
• Familarity with linear models.
• Solve polynomial equations.
• Be familiar with complex numbers.
• Understand under what conditions the partial sum Sn =
P∞
P
(ie. if ∞
j=1 aj .
j=1 |aj | < ∞, then Sn → S, where S =
Pn
j=1 aj
has a well defined limits
Objectives
• Understand what causal and invertible is.
• Know what an AR, MA and ARMA time series model is.
• Know how to find a solution of an ARMA time series, and understand why this is important (how the roots determine causality and why this is important to know - in terms of
characteristics in the process and also simulations).
• Understand how the roots of the AR can determine ‘features’ in the time series and covariance
structure (such as pseudo periodicities).
36
2.1
Motivation
The objective of this chapter is to introduce the linear time series model. Linear time series models
are designed to model the covariance structure in the time series. There are two popular subgroups of linear time models (a) the autoregressive and (a) the moving average models, which can
be combined to make the autoregressive moving average models.
We motivate the autoregressive from the perspective of classical linear regression. We recall one
objective in linear regression is to predict the response variable given variables that are observed.
To do this, typically linear dependence between response and variable is assumed and we model Yi
as
Yi =
p
X
aj Xij + εi ,
j=1
where εi is such that E[εi |Xij ] = 0 and more commonly εi and Xij are independent. In linear
regression once the model has been defined, we can immediately find estimators of the parameters,
do model selection etc.
Returning to time series, one major objective is to predict/forecast the future given current and
past observations (just as in linear regression our aim is to predict the response given the observed
variables). At least formally, it seems reasonable to represent this as
Xt =
p
X
aj Xt−j + εt ,
(2.1)
j=1
where we assume that {εt } are independent, identically distributed, zero mean random variables.
Model (2.1) is called an autoregressive model of order p (AR(p) for short). It is easy to see that
E(Xt |Xt−1 , . . . , Xt−p ) =
p
X
aj Xt−j
j=1
(this is the exected value of Xt given that Xt−1 , . . . , Xt−p have already been observed), thus the
past values of Xt have a linear influence on the conditional mean of Xt (compare with the linear
P
P
model Yt = pj=1 aj Xt,j + εt , then E(Yt |Xt,j ) = pj=1 aj Xt,j ). Conceptionally, the autoregressive
model appears to be a straightforward extension of the linear regression model. Don’t be fooled
by this, it is a more complex object. Unlike the linear regression model, (2.1) is an infinite set of
linear difference equations. This means, for this systems of equations to be well defined, it needs
37
to have a solution which is meaningful. To understand why, recall that equation is defined for all
t ∈ Z, so let us start the equation at the beginning of time (t = −∞) and run it on. Without any
constraint on the parameters {aj }, there is no reason to believe the solution is finite (contrast this
with linear regression where these issues are not relevant). Therefore, the first thing to understand
is under what conditions will the AR model (2.1) have a well defined stationary solution and what
features in a time series is the solution able to capture.
Of course, one could ask why go through to the effort. One could simply use least squares to
estimate the parameters. This is possible, but without a proper analysis it is not clear whether
model has a meaningful solution (for example in Section 3.3 we show that the least squares estimator can lead to misspecified models), it’s not even possible to make simulations of the process.
Therefore, there is a practical motivation behind our theoretical treatment.
In this chapter we will be deriving conditions for a strictly stationary solution of (2.1). We
will place moment conditions on the innovations {εt }, these conditions will be sufficient but not
necessary conditions Under these conditions we obtain a strictly stationary solution but not a
second order stationary process. In Chapter 3 we obtain conditions for (2.1) to have be both a
strictly and second order stationary solution. It is possible to obtain strictly stationary solution
under far weaker conditions (see Theorem 4.0.1), but these won’t be considered here.
Example 2.1.1 How would you simulate from the model
Xt = φ1 Xt−1 + φ2 Xt−1 + εt .
2.2
2.2.1
Linear time series and moving average models
Infinite sums of random variables
Before defining a linear time series, we define the MA(q) model which is a subclass of linear time
series. Let us supppose that {εt } are iid random variables with mean zero and finite variance. The
time series {Xt } is said to have a MA(q) representation if it satisfies
Xt =
q
X
ψj εt−j ,
j=0
38
where E(εt ) = 0 and var(εt ) = 1. It is clear that Xt is a rolling finite weighted sum of {εt },
therefore {Xt } must be well defined (which for finite sums means it is almost surely finite, this
you can see because it has a finite variance). We extend this notion and consider infinite sums of
random variables. Now, things become more complicated, since care must be always be taken with
anything involving infinite sums. More precisely, for the sum
∞
X
ψj εt−j ,
j=−∞
to be well defined (has a finite limit), the partial sums Sn =
Pn
j=−n ψj εt−j
should be (almost
surely) finite and the sequence Sn should converge (ie. |Sn1 − Sn2 | → 0 as n1 , n2 → ∞). Below, we
give conditions under which this is true.
Lemma 2.2.1 Suppose {Xt } is a strictly stationary time series with E|Xt | < ∞, then {Yt } defined
by
Yt =
∞
X
ψj Xt−j ,
j=−∞
P∞
< ∞, is a strictly stationary time series. Furthermore, the partial sum converges
P
almost surely, Yn,t = nj=0 ψj Xt−j → Yt . If var(Xt ) < ∞, then {Yt } is second order stationary
where
j=0 |ψj |
and converges in mean square (that is E(Yn,t − Yt )2 → 0).
PROOF. See Brockwell and Davis (1998), Proposition 3.1.1 or Fuller (1995), Theorem 2.1.1 (page
31) (also Shumway and Stoffer (2006), page 86).
Example 2.2.1 Suppose {Xt } is a strictly stationary time series with var(Xt ) < ∞. Define {Yt }
as the following infinite sum
Yt =
∞
X
j k ρj |Xt−j |
j=0
where |ρ| < 1. Then {Yt } is also a strictly stationary time series with a finite variance.
We will use this example later in the course.
Having derived conditions under which infinite sums are well defined (good), we can now define
the general class of linear and MA(∞) processes.
39
Definition 2.2.1 (The linear process and moving average (MA)(∞)) Suppose that {εt }
P
are iid random variables, ∞
j=0 |ψj | < ∞ and E(|εt |) < ∞.
A time series is said to be a linear time series if it can be represented as
∞
X
Xt =
ψj εt−j ,
j=−∞
where {εt } are iid random variables with finite variance.
(ii)
(i) The time series {Xt } has a MA(∞) representation if it satisfies
Xt =
∞
X
ψj εt−j .
j=0
Note that since that as these sums are well defined by equation (1.12) {Xt } is a strictly stationary
(ergodic) time series.
The difference between an MA(∞) process and a linear process is quite subtle. A linear process
involves both past, present and future innovations {εt }, whereas the MA(∞) uses only past and
present innovations.
Definition 2.2.2 (Causal and invertible)
(i) A process is said to be causal if it has the representation
Xt =
∞
X
aj εt−j ,
j=0
(ii) A process is said to be invertible if it has the representation
Xt =
∞
X
bj Xt−j + εt ,
j=1
(so far we have yet to give conditions under which the above has a well defined solution).
Causal and invertible solutions are useful in both estimation and forecasting (predicting the future
based on the current and past).
A very interesting class of models which have MA(∞) representations are autoregressive and
autoregressive moving average models. In the following sections we prove this.
40
2.3
The autoregressive model and the solution
In this section we will examine under what conditions the AR process has a stationary solution.
2.3.1
Difference equations and back-shift operators
The autoregressive model is defined in terms of inhomogenuous difference equations. Difference
equations can often be represented in terms of backshift operators, so we start by defining them
and see why this representation may be useful (and why it should work).
The time series {Xt } is said to be an autoregressive (AR(p)) if it satisfies the equation
Xt − φ1 Xt−1 − . . . − φp Xt−p = εt ,
t ∈ Z,
where {εt } are zero mean, finite variance random variables. As we mentioned previously, the autoregressive model is a difference equation (which can be treated as a infinite number of simultaneous
equations). Therefore for it to make any sense it must have a solution. To obtain a general solution
we write the autoregressive model in terms of backshift operators:
Xt − φ1 BXt − . . . − φp B p Xt = εt ,
where φ(B) = 1 −
Pp
j=1 φj B
j,
⇒
φ(B)Xt = εt
B is the backshift operator and is defined such that B k Xt = Xt−k .
Simply rearranging φ(B)Xt = εt , gives the ‘solution’ of the autoregressive difference equation to
be Xt = φ(B)−1 εt , however this is just an algebraic manipulation, below we investigate whether it
really has any meaning. To do this, we start with an example.
2.3.2
Solution of two particular AR(1) models
Below we consider two different AR(1) models and obtain their solutions.
(i) Consider the AR(1) process
Xt = 0.5Xt−1 + εt ,
t ∈ Z.
(2.2)
Notice this is an equation (rather like 3x2 + 2x + 1 = 0, or an infinite number of simultaneous
equations), which may or may not have a solution. To obtain the solution we note that
41
Xt = 0.5Xt−1 + εt and Xt−1 = 0.5Xt−2 + εt−1 . Using this we get Xt = εt + 0.5(0.5Xt−2 +
εt−1 ) = εt + 0.5εt−1 + 0.52 Xt−2 . Continuing this backward iteration we obtain at the kth
P
iteration, Xt = kj=0 (0.5)j εt−j + (0.5)k+1 Xt−k . Because (0.5)k+1 → 0 as k → ∞ by taking
P
j
the limit we can show that Xt = ∞
j=0 (0.5) εt−j is almost surely finite and a solution of
(2.2). Of course like any other equation one may wonder whether it is the unique solution
(recalling that 3x2 + 2x + 1 = 0 has two solutions). We show in Section 2.3.3 that this is the
unique stationary solution of (2.2).
Let us see whether we can obtain a solution using the difference equation representation. We
recall, that by crudely taking inverses, the solution is Xt = (1 − 0.5B)−1 εt . The obvious
P∞
j
question is whether this has any meaning. Note that (1 − 0.5B)−1 =
j=0 (0.5B) , for
|B| ≤ 2, hence substituting this power series expansion into Xt we have
∞
X
X
X
Xt = (1 − 0.5B)−1 εt = ( (0.5B)j )εt = ( (0.5j B j ))εt =
(0.5)j εt−j ,
j=0
j=0
j=0
which corresponds to the solution above. Hence the backshift operator in this example helps
us to obtain a solution. Moreover, because the solution can be written in terms of past values
of εt , it is causal.
(ii) Let us consider the AR model, which we will see has a very different solution:
Xt = 2Xt−1 + εt .
Doing what we did in (i) we find that after the kth back iteration we have Xt =
(2.3)
Pk
j
j=0 2 εt−j +
2k+1 Xt−k . However, unlike example (i) 2k does not converge as k → ∞. This suggest that if
P
j
we continue the iteration Xt = ∞
j=0 2 εt−j is not a quantity that is finite (when εt are iid).
P
j
Therefore Xt = ∞
j=0 2 εt−j cannot be considered as a solution of (2.3). We need to write
(2.3) in a slightly different way in order to obtain a meaningful solution.
Rewriting (2.3) we have Xt−1 = 0.5Xt + 0.5εt . Forward iterating this we get Xt−1 =
P
−(0.5) kj=0 (0.5)j εt+j − (0.5)t+k+1 Xt+k . Since (0.5)t+k+1 → 0 we have
Xt−1 = −(0.5)
∞
X
j=0
as a solution of (2.3).
42
(0.5)j εt+j
Let us see whether the difference equation can also offer a solution. Since (1 − 2B)Xt = εt ,
using the crude manipulation we have Xt = (1 − 2B)−1 εt . Now we see that (1 − 2B)−1 =
P∞
P∞ j j
j
j=0 (2B) for |B| < 1/2. Using this expansion gives Xt =
j=0 2 B Xt , but as pointed out
above this sum is not well defined. What we find is that φ(B)−1 εt only makes sense (is well
defined) if the series expansion of φ(B)−1 converges in a region that includes the unit circle
|B| = 1.
What we need is another series expansion of (1 − 2B)−1 which converges in a region which
includes the unit circle |B| = 1 (as an aside, we note that a function does not necessarily
have a unique series expansion, it can have difference series expansions which may converge
in different regions). We now show that a convergent series expansion needs to be defined in
terms of negative powers of B not positive powers. Writing (1 − 2B) = −(2B)(1 − (2B)−1 ),
therefore
∞
X
(1 − 2B)−1 = −(2B)−1
(2B)−j ,
j=0
which converges for |B| > 1/2. Using this expansion we have
Xt = −
∞
∞
X
X
(0.5)j+1 B −j−1 εt = −
(0.5)j+1 εt+j+1 ,
j=0
j=0
which we have shown above is a well defined solution of (2.3).
In summary (1 − 2B)−1 has two series expansions
∞
X
1
=
(2B)−j
(1 − 2B)
j=0
which converges for |B| < 1/2 and
∞
X
1
= −(2B)−1
(2B)−j ,
(1 − 2B)
j=0
which converges for |B| > 1/2. The one that is useful for us is the series which converges
when |B| = 1.
It is clear from the above examples how to obtain the solution of a general AR(1). We now
show that this solution is the unique stationary solution.
43
2.3.3
The unique solution of a general AR(1)
Consider the AR(1) process Xt = φXt−1 + εt , where |φ| < 1. Using the method outlined in (i), it is
P
j
straightforward to show that Xt = ∞
j=0 φ εt−j is it’s stationary solution, we now show that this
solution is unique.
P
j
We first show that Xt = ∞
j=0 φ εt−j is well defined (that it is almost surely finite). We note
P∞
P∞
j
j
that |Xt | ≤
j=0 |φ | · |εt−j |. Thus we will show that
j=0 |φ | · |εt−j | is almost surely finite,
which will imply that Xt is almost surely finite. By montone convergence we can exchange sum
P
P
and expectation and we have E(|Xt |) ≤ E(limn→∞ nj=0 |φj εt−j |) = limn→∞ nj=0 |φj |E|εt−j |) =
P∞ j
P
j
E(|ε0 |) ∞
j=0 φ εt−j is a well defined solution of
j=0 |φ | < ∞. Therefore since E|Xt | < ∞,
Xt = φXt−1 + εt .
To show that it is the unique stationary causal solution, let us suppose there is another (causal)
solution, call it Yt (note that this part of the proof is useful to know as such methods are often used
when obtaining solutions of time series models). Clearly, by recursively applying the difference
equation to Yt , for every s we have
Yt =
s
X
φj εt−j + φs Yt−s−1 .
j=0
Evaluating the difference between the two solutions gives Yt − Xt = As − Bs where As = φs Yt−s−1
P
j
and Bs = ∞
j=s+1 φ εt−j for all s. To show that Yt and Xt coincide almost surely we will show that
P
for every > 0, ∞
s=1 P (|As − Bs | > ε) < ∞ (and then apply the Borel-Cantelli lemma). We note
if |As − Bs | > ε), then either |As | > ε/2 or |Bs | > ε/2. Therefore P (|As − Bs | > ε) ≤ P (|As | >
ε/2)+P (|Bs | > ε/2). To bound these two terms we use Markov’s inequality. It is straightforward to
show that P (|Bs | > ε/2) ≤ Cφs /ε. To bound E|As |, we note that |Ys | ≤ |φ| · |Ys−1 | + |εs |, since {Yt }
is a stationary solution then E|Ys |(1 − |φ|) ≤ E|εs |, thus E|Yt | ≤ E|εt |/(1 − |φ|) < ∞. Altogether
P
this gives P (|As − Bs | > ε) ≤ Cφs /ε (for some finite constant C). Hence ∞
s=1 P (|As − Bs | > ε) <
P∞
s
s=1 Cφ /ε < ∞. Thus by the Borel-Cantelli lemma, this implies that the event {|As − Bs | > ε}
happens only finitely often (almost surely). Since for every ε, {|As −Bs | > ε} occurs (almost surely)
P
j
only finite often for all ε, then Yt = Xt almost surely. Hence Xt = ∞
j=0 φ εt−j is (almost surely)
the unique causal solution.
44
2.3.4
The solution of a general AR(p)
Let us now summarise our observation for the general AR(1) process Xt = φXt−1 + εt . If |φ| < 1,
then the solution is in terms of past values of {εt }, if on the other hand |φ| > 1 the solution is in
terms of future values of {εt }.
Now we try to understand this in terms of the expansions of the characteristic polynomial
φ(B) = 1 − φB (using the AR(1) as a starting point). From what we learnt in the previous
section, we require the characteristic polynomial of the AR process to have a convergent power
series expansion in the region including the ring |B| = 1. In terms of the AR(1) process, if the root
of φ(B) is greater than one, then the power series of φ(B)−1 is in terms of positive powers, if it is
less than one, then φ(B)−1 is in terms of negative powers.
Generalising this argument to a general polynomial, if the roots of φ(B) are greater than one,
then the power series of φ(B)−1 (which converges for |B| = 1) is in terms of positive powers (hence
the solution φ(B)−1 εt will be in past terms of {εt }). On the other hand, if the roots are both less
than and greater than one (but do not lie on the unit circle), then the power series of φ(B)−1 will
be in both negative and positive powers. Thus the solution Xt = φ(B)−1 εt will be in terms of both
past and future values of {εt }. We summarize this result in a lemma below.
Lemma 2.3.1 Suppose that the AR(p) process satisfies the representation φ(B)Xt = εt , where
none of the roots of the characteristic polynomial lie on the unit circle and E|εt | < ∞. Then {Xt }
has a stationary, almost surely unique, solution.
We see that where the roots of the characteristic polynomial φ(B) lie defines the solution of
the AR process. We will show in Sections 2.3.6 and 3.1.2 that it not only defines the solution but
determines some of the characteristics of the time series.
Exercise 2.1 Suppose {Xt } satisfies the AR(p) representation
Xt =
p
X
φj Xt−j + εt ,
j=1
where
Pp
j=1 |φj |
< 1 and E|εt | < ∞. Show that {Xt } will always have a causal stationary solution.
45
2.3.5
Explicit solution of an AR(2) model
Specific example
Suppose {Xt } satisfies
Xt = 0.75Xt−1 − 0.125Xt−2 + εt ,
where {εt } are iid random variables. We want to obtain a solution for the above equations.
It is not easy to use the backward (or forward) iterating techique for AR processes beyond
order one. This is where using the backshift operator becomes useful. We start by writing Xt =
0.75Xt−1 − 0.125Xt−2 + εt as φ(B)Xt = ε, where φ(B) = 1 − 0.75B + 0.125B 2 , which leads to what
is commonly known as the characteristic polynomial φ(z) = 1 − 0.75z + 0.125z 2 . If we can find a
power series expansion of φ(B)−1 , which is valid for |B| = 1, then the solution is Xt = φ(B)−1 εt .
We first observe that φ(z) = 1 − 0.75z + 0.125z 2 = (1 − 0.5z)(1 − 0.25z). Therefore by using
partial fractions we have
1
−1
2
1
=
=
+
.
φ(z)
(1 − 0.5z)(1 − 0.25z)
(1 − 0.5z) (1 − 0.25z)
We recall from geometric expansions that
∞
X
−1
=−
(0.5)j z j
(1 − 0.5z)
∞
|z| ≤ 2,
j=0
X
2
=2
(0.25)j z j
(1 − 0.25z)
|z| ≤ 4.
j=0
Putting the above together gives
∞
X
1
=
{−(0.5)j + 2(0.25)j }z j
(1 − 0.5z)(1 − 0.25z)
|z| < 2.
j=0
The above expansion is valid for |z| = 1, because
P∞
j=0 |
− (0.5)j + 2(0.25)j | < ∞ (see Lemma
2.3.2). Hence
Xt = {(1 − 0.5B)(1 − 0.25B)}−1 εt =
∞
X
∞
X
{−(0.5)j + 2(0.25)j }B j εt =
{−(0.5)j + 2(0.25)j }εt−j ,
j=0
j=0
which gives a stationary solution to the AR(2) process (see Lemma 2.2.1).
The discussion above shows how the backshift operator can be applied and how it can be used
46
to obtain solutions to AR(p) processes.
The solution of a general AR(2) model
We now generalise the above to general AR(2) models
Xt = (a + b)Xt−1 − abXt−2 + εt ,
the characteristic polynomial of the above is 1 − (a + b)z + abz 2 = (1 − az)(1 − bz). This means
the solution of Xt is
Xt = (1 − Ba)−1 (1 − Bb)−1 εt ,
thus we need an expansion of (1 − Ba)−1 (1 − Bb)−1 . Assuming that a 6= b, as above we have
1
b
a
1
=
−
(1 − za)(1 − zb)
b − a 1 − bz 1 − az
Cases:
(i) |a| < 1 and |b| < 1, this means the roots lie outside the unit circle. Thus the expansion is
∞
∞
j=0
j=0
X
X
1
1
=
b
bj z j − a
aj z j ,
(1 − za)(1 − zb)
(b − a)
which leads to the causal solution
1
Xt =
b−a
X
∞
j+1
b
j+1
−a
)εt−j .
(2.4)
j=0
(ii) Case that |a| > 1 and |b| < 1, this means the roots lie inside and outside the unit circle and
we have the expansion
1
(1 − za)(1 − zb)
=
1
b−a
=
1
(b − a)
47
b
a
−
1 − bz (az)((az)−1 − 1)
X
∞
∞
X
bj z j + z −1
a−j z −j ,
b
j=0
j=0
(2.5)
which leads to the non-causal solution
Xt =
∞
∞
X
X
1
bj+1 εt−j +
a−j εt+1+j .
b−a
j=0
(2.6)
j=0
Later we show that the non-causal solution has the same correlation structure as the causal
solution when a = a−1 .
This solution throws up additional interesting results. Let us return to the expansion in (2.5)
and apply it to Xt

Xt =


1
b
1 
1


εt =
εt +
ε


t
−1
−1
(1 − Ba)(1 − Bb)
b − a  |1 −{z
bB }
B(1 − a B ) 
{z
}
|
causal AR(1)
=
noncausal AR(1)
1
(Yt + Zt+1 )
b−a
where Yt = bYt−1 + εt and Zt+1 = a−1 Zt+2 + εt+1 . In other words, the noncausal AR(2)
process is the sum of a causal and a‘future’ AR(1) process. This is true for all noncausal
time series (except when there is multiplicity in the roots) and is discussed further in Section
2.6.
Several authors including Richard Davis, Jay Breidt and Beth Andrews argue that noncausal
time series can model features in data which causal time series cannot.
(iii) a = b > 1. The characteristic polynomial is (1 − az)2 . To obtain the convergent expansion
when |z| = 1 we note that (1 − az)−2 = (−1) d(1−az)
d(az)
−1
. Thus
∞
X
(−1)
=
(−1)
j(az)j−1 .
(1 − az)2
j=0
This leads to the causal solution
Xt = (−1)
∞
X
jaj−1 εt−j .
j=1
Exercise 2.2 Show for the AR(2) model Xt = φ1 Xt−1 + φ2 Xt−2 + εt to have a causal stationary
48
solution the parameters φ1 , φ2 must lie in the region
φ2 + φ1 < 1,
Exercise 2.3
φ2 − φ1 < 1
|φ2 | < 1.
(a) Consider the AR(2) process
Xt = φ1 Xt−1 + φ2 Xt−2 + εt ,
where {εt } are iid random variables with mean zero and variance one. Suppose the roots of
the characteristic polynomial 1 − φ1 z − φ2 z 2 are greater than one. Show that |φ1 | + |φ2 | < 4.
(b) Now consider a generalisation of this result. Consider the AR(p) process
Xt = φ1 Xt−1 + φ2 Xt−2 + . . . φp Xt−p + εt .
Suppose the roots of the characteristic polynomial 1 − φ1 z − . . . − φp z p are greater than one.
Show that |φ1 | + . . . + |φp | ≤ 2p .
2.3.6
Features of a realisation from an AR(2)
We now explain why the AR(2) (and higher orders) can characterise some very interesting behaviour
(over the rather dull AR(1)). For now we assume that Xt is a causal time series which satisfies the
AR(2) representation
Xt = φ1 Xt−1 + φ2 Xt−2 + εt
where {εt } are iid with mean zero and finite variance. The characteristic polynomial is φ(B) =
1 − φ1 B − φ2 B 2 . Let us assume the roots of φ(B) are complex, since φ1 and φ2 are real, the roots
are complex conjugates. Thus by using case (i) above we have
1
1
=
1 − φ1 B − φ2 B 2
λ−λ
where λ−1 and λ
−1
¯ λ
λ
−
,
¯
1 − λB 1 − λB
are the roots of the characteristic. Thus
Xt = C
∞
X
j
λ εt−j
− C¯
j=0
∞
X
j=0
49
¯ j εt−j ,
λ
(2.7)
where C = [λ − λ]−1 . Since λ and C are complex we use the representation λ = r exp(iθ) and
C = α exp(iβ) (noting that |r| < 1), and substitute these expressions for λ and C into (2.7) to give
Xt = α
∞
X
rj cos(θj + β)εt−j .
j=0
We can see that Xt is effectively the sum of cosines with frequency θ that have been modulated by
the iid errors and exponentially damped. This is why for realisations of autoregressive processes
you will often see periodicities (depending on the roots of the characteristic). These arguments can
be generalised to higher orders p.
Exercise 2.4
(a) Obtain the stationary solution of the AR(2) process
2
7
Xt = Xt−1 − Xt−2 + εt ,
3
3
where {εt } are iid random variables with mean zero and variance σ 2 .
Does the solution have an MA(∞) representation?
(b) Obtain the stationary solution of the AR(2) process
√
4× 3
42
Xt =
Xt−1 − 2 Xt−2 + εt ,
5
5
where {εt } are iid random variables with mean zero and variance σ 2 .
Does the solution have an MA(∞) representation?
(c) Obtain the stationary solution of the AR(2) process
Xt = Xt−1 − 4Xt−2 + εt ,
where {εt } are iid random variables with mean zero and variance σ 2 .
Does the solution have an MA(∞) representation?
Exercise 2.5 Construct a causal stationary AR(2) process with pseudo-period 17. Using the R
function arima.sim simulate a realisation from this process (of length 200) and make a plot of the
periodogram. What do you observe about the peak in this plot?
Below we now consider solutions to general AR(∞) processes.
50
2.3.7
Solution of the general AR(∞) model
AR(∞) models are more general than the AR(p) model and are able to model more complex
behaviour, such as slower decay of the covariance structure. It is arguable how useful these models
are in modelling data, however recently it has become quite popular in time series bootstrap
methods.
In order to obtain the stationary solution of an AR(∞), we need to define an analytic function
and its inverse.
Definition 2.3.1 (Analytic functions in the region Ω) Suppose that z ∈ C. φ(z) is an analytic complex function in the region Ω, if it has a power series expansion which converges in Ω, that
P
j
is φ(z) = ∞
j=−∞ φj z .
P
˜ j
˜
˜
˜
If there exists a function φ(z)
= ∞
j=−∞ φj z such that φ(z)φ(z) = 1 for all z ∈ Ω, then φ(z)
is the inverse of φ(z) in the region Ω.
Well known examples of analytic functions include
(i) Finite order polynomials such as φ(z) =
(ii) The expansion (1 − 0.5z)−1 =
Pp
j=0 φj z
P∞
j
j=0 (0.5z)
j
for Ω = C.
for Ω = {z; |z| ≤ 2}.
We observe that for AR processes we can represent the equation as φ(B)Xt = εt , which formally
gives the solution Xt = φ(B)−1 εt . This raises the question, under what conditions on φ(B)−1 is
φ(B)−1 εt a valid solution. For φ(B)−1 εt to make sense φ(B)−1 should be represented as a power
series expansion. Below, we give conditions on the power series expansion which give a stationary
solution. It is worth noting this is closely related to Lemma 2.2.1.
P∞
j
is finite on a region that includes |z| = 1 (hence
P
it is analytic) and {Xt } is a strictly stationary process with E|Xt | < ∞. Then ∞
j=−∞ |φj | < ∞
P∞
and Yt = φ(B)Xt−j = j=−∞ φj Xt−j is almost surely finite and strictly stationary time series.
Lemma 2.3.2 Suppose that φ(z) =
j=−∞ φj z
P
j
PROOF. It can be shown that if sup|z|=1 |φ(z)| < ∞, in other words on the unit circle ∞
j=−∞ φj z <
P
∞, then ∞
j=−∞ |φj | < ∞. Since the coefficients are absolutely summable, then by Lemma 2.2.1
P
we have that Yt = φ(B)Xt−j = ∞
j=−∞ φj Xt−j is almost surely finite and strictly stationary.
Using the above we can obtain the solution of an AR(∞) (which includes an AR(p) as a special
case).
51
Corollary 2.3.1 Suppose that
Xt =
∞
X
φj Xt−j + εt
j=1
and φ(z) has an inverse ψ(z) =
P∞
j=−∞ ψj z
j
which is analytic in a region including |z| = 1, then
the solution of Xt is
Xt =
∞
X
ψj εt−j .
j=−∞
Corollary 2.3.2 Let Xt be an AR(p) time series, where
Xt =
p
X
φj Xt−j + εt .
j=1
Suppose the roots of the characteristic polynomial φ(B) = 1 −
Pp
j=1 φj B
j
do not lie on the unit
circle |B| = 1, then Xt admits a strictly stationary solution.
In addition suppose the roots of φ(B) all lie outside the unit circle, then Xt admits a strictly
stationary, causal solution.
This summarises what we observed in Section 2.3.4.
Rules of the back shift operator:
(i) If a(z) is analytic in a region Ω which includes the unit circle |z| = 1 in it’s interior, then
a(B)Xt is a well defined random variable.
(ii) The operator is commutative and associative, that is [a(B)b(B)]Xt = a(B)[b(B)Xt ] =
[b(B)a(B)]Xt (the square brackets are used to indicate which parts to multiply first). This
may seems obvious, but remember matrices are not commutative!
(iii) Suppose that a(z) and its inverse
1
a(z)
are both have solutions in the region Ω which includes
the unit circle |z| = 1 in it’s interior. If a(B)Xt = Zt , then Xt =
1
a(B) Zt .
(i) Clearly a(z) = 1 − 0.5z is analytic for all z ∈ C,
P
1
j
and has no zeros for |z| < 2. The inverse is a(z)
= ∞
j=0 (0.5z) is well defined in the region
Example 2.3.1 (Analytic functions)
|z| < 2.
52
(ii) Clearly a(z) = 1 − 2z is analytic for all z ∈ C, and has no zeros for |z| > 1/2. The inverse is
P∞
1
−1
−1
j
a(z) = (−2z) (1 − (1/2z)) = (−2z) ( j=0 (1/(2z)) ) well defined in the region |z| > 1/2.
(iii) The function a(z) =
1
(1−0.5z)(1−2z)
is analytic in the region 0.5 < z < 2.
(iv) a(z) = 1 − z, is analytic for all z ∈ C, but is zero for z = 1. Hence its inverse is not well
defined for regions which involve |z| = 1 (see Example 2.3.2).
Example 2.3.2 (Unit root/integrated processes and non-invertible processes)
(i) If the difference equation has root one, then an (almost sure) stationary solution of the AR model
do not exist. The simplest example is the ‘random walk’ Xt = Xt−1 + εt (φ(z) = (1 − z)). This is
an example of an Autoregressive Integrated Moving Average ARIMA(0, 1, 0) model (1 − B)Xt = εt .
To see that it does not have a stationary solution, we iterate the equation n steps backwards
P
P
and we see that Xt = nj=0 εt−j + Xt−n . St,n = nj=0 εt−j is the partial sum, but it is clear that
the partial sum St,n does not have a limit, since it is not a Cauchy sequence, ie. |St,n − St,m | does
not have a limit. However, given some initial value X0 , for t > 0 we can define the unit process
P
Xt = Xt−1 +ε. Notice that the nonstationary solution of this sequence is Xt = X0 + tj=1 εt−j which
has variance var(Xt ) = var(X0 ) + t (assuming that {εt } are iid random variables with variance one
and independent of X0 ).
We observe that we can ‘stationarize’ the process by taking first differences, ie. defining Yt =
Xt − Xt−1 = εt .
(ii) The unit process described above can be generalised ARIMA(0, d, 0), where (1 − B)d Xt = εt .
In this case to stationarize the sequence we take d differences, ie. let Yt,0 = Xt and for 1 ≤ i ≤ d
define the iteration
Yt,i = Yt,i−1 − Yt−1,i−1
and Yt = Yt,d will be a stationary sequence. Or define
Yt =
d
X
j=0
d!
(−1)j Xt−j ,
j!(d − j)!
in which case Yt as defined above will be a stationary sequence.
53
(iii) The general ARIMA(p, d, q) is defined as (1 − B)d φ(B)Xt = θ(B)εt , where φ(B) and θ(B) are
p and q order polynomials respectively and the roots of φ(B) lie outside the unit circle.
Another way of describing the above model is that after taking d differences (as detailed in (ii))
the resulting process is an ARMA(p, q) process (see Section 2.5 for the definition of an ARMA
model).
To illustrate the difference between stationary ARMA and ARIMA processes, in Figure 2.1
(iv) In examples (i) and (ii) a stationary solution does not exist. We now consider an example
where the process is stationary but an autoregressive representation does not exist.
Consider the MA(1) model Xt = εt − εt−1 . We recall that this can be written as Xt = φ(B)εt
where φ(B) = 1 − B. From Example 2.3.1(iv) we know that φ(z)−1 does not exist, therefore it does
20
−60
−40
−5
−20
0
ar2
ar2I
0
40
60
5
80
not have an AR(∞) representation since (1 − B)−1 Xt = εt is not well defined.
0
100
200
300
400
0
Time
100
200
300
400
Time
(a) Xt = 1.5Xt−1 − 0.75Xt−2 + εt
(b) (1 − B)Yt = Xt , where is defined in (a)
Figure 2.1: Realisations from an AR process and it’s corresponding integrated process, using
N (0, 1) innovations (generated using the same seed).
54
2.4
An explanation as to why the backshift operator
method works
To understand why the magic backshift operator works, we use matrix notation to rewrite the
AR(p) model as an infinite set of difference equations




..
.

... ... ... ...
...
...
...


  Xt


 ... 0
1 −φ1 . . . −φp . . .  

 X

  t−1
 ... 0
0
1
−φ1 . . . −φp  

 X
 t−2
 .
... ... ... ...
...
...
...
..

..
.
 
 
  εt
 
 
= ε
  t−1
 
  ε
  t−2
  .
..






.





The above is an infinite dimensional equation (and the matrix is an infinite upper triangular matrix).
Formally to obtain a simulation we invert the matrix to get a solution of Xt in terms of εt . Of course
in reality it is not straightfoward to define this inverse. Instead let us consider a finite (truncated)
version of the above matrix equation. Except for the edge effects this is a circulant matrix (where
the rows are repeated, but each time shifted by one, see wiki for a description). Truncating the
matrix to have dimension n, we approximate the above by the finite set of n-equations

1
−φ1
...


 0
1
−φ1


 ...
...
...

−φ1 −φ2 . . .
−φp
0
...
...
−φp
...
...
...
0
1


Xn


  Xn−1

  ..
 .

X0

εn





 ε
 =  n−1

 ..

 .


ε0








⇒ C n X n ≈ εn .
The approximation of the AR(p) equation only arises in the first p-equations, where
X0 −
p
X
φj Xn−j
= ε0
φj Xn+1−j
= ε1
j=1
X1 − φ1 X0 −
p
X
j=2
..
.
Xp −
p−1
X
..
.
φj Xp−j − φp Xn = εp .
j=1
55
We now define the n × n matrix Un , where




Un = 



0 1 0 0 ... 0
0 0 1 0 ... 0
.. .. .. .. .. ..
. . . . . .




.



1 0 0 0 ... 0
We observe that Un is a ‘deformed diagonal matrix’ where all the ones along the diagonal have
been shifted once to the right, and the ‘left over’ one is placed in the bottom left hand corner. Un
is another example of a circulant matrix, moreover Un2 shifts once again all the ones to the right






2
Un = 




0 1 0 0 ... 0
0 0 1 0 ... 0
.. .. .. .. .. ..
. . . . . .
1 0 0 0 ... 0






.




0 1 0 0 ... 0
Un3 shifts the ones to the third off-diagonal and so forth until Unn = I. Thus all circulant matrices
can be written in terms of powers of Un (the matrix Un can be considered as the building blocks
of circulant matrices). In particular
Cn = In −
p
X
φj Unj ,
j=1
[In −
Pp
j
j=1 φj Un ]X n
= εn and the solution to the equation is
X n = (In −
p
X
φj Unj )−1 εn .
j=1
Our aim is to write (In −
Pp
j −1
j=1 φj Un )
as a power series in Un , with Un playing the role of the
backshift operator.
P
To do this we recall the similarity between the matrix In − pj=1 φj Unj and the characteristic
P
equation φ(B) = 1 − pj=1 φj z j . In particular since we can factorize the characteristic equation as
Q
P
Q
φ(B) = pj=1 [1 − λj B], we can factorize the matrix In − pj=1 φj Unj = pj=1 [In − λj Un ]. To obtain
the inverse, for simplicity, we assume that the roots of the characteristic function are greater than
56
one (ie. |λj | < 1, which we recall corresponds to a causal solution) and are all different. Then there
exists constants cj where
[In −
p
X
φj Unj ]−1
=
j=1
p
X
cj (In − λj Un )−1
j=1
P
(just as in partial fractions) - to see why multiply the above by [In − pj=1 φj Unj ]. Finally, we
P
j
recall that if the eigenvalues of A are less than one, then (1 − A)−1 = ∞
j=0 A . The eigenvalues
of Un are {exp( 2πij
n ); j = 1, . . . , n}, thus the eigenvalues of λj Un are less than one. This gives
P
k k
(In − λj Un )−1 = ∞
k=0 λj Un and
[In −
p
X
φj Unj ]−1 =
j=1
p
X
cj
j=1
∞
X
λkj Unk .
(2.8)
k=0
Therefore, the solution of Cn X n = εn is

X n = Cn−1 εn = 
p
X
cj
j=1
∞
X

λkj Unk  εn .
k=0
Let us focus on the first element of the vector X n , which is Xn . Since Unk εn shifts the elements of
εn up by k (note that this shift is with wrapping of the vector) we have
Xn =
p
X
j=1
cj
n
X
λkj εn−k
k=0
+
p
X
cj
j=1
∞
X
λkj εn−k
{z
|
mod (n) .
(2.9)
k=n+1
→0
}
Note that the second term decays geometrically fast to zero. Thus giving the stationary solution
P
P
k
Xn = pj=1 cj ∞
k=0 λj εn−k .
P
To recollect, we have shown that [In − pj=1 φj Unj ]−1 admits the solution in (2.8) (which is
the same as the solution of the inverse of φ(B)−1 ) and that Unj εn plays the role of the backshift
operator. Therefore, we can use the backshift operator in obtaining a solution of an AR process
because it plays the role of the matrix Un .
57
Example 2.4.1 The AR(1) model, Xt − φ1 Xt−1 = εt is written as

1


 0


 ...

−φ1
−φ1
1
...
0
0
−φ1 . . .
...
...
0
0
0 0
0
0
0
Xn


  Xn−1

  ..
 .

X0
... ...
0

1


εn





 ε
 =  n−1

 ..

 .


ε0








⇒ C n X n = εn .
The approximation of the AR(1) is only for the first equation, where X0 − φ1 Xn = ε0 . Using the
matrix Un , the above equation can be written as (In − φ1 Un )X n = εn , which gives the solution
X n = (In − φ1 Un )−1 εn .
Let us suppose that |φ1 | > 1 (ie, the root lies inside the unit circle and the solution is noncausal),
−1
then to get a convergent expansion of (1n −φ1 Un )−1 we rewrite (In −φ1 Un ) = −φ1 Un (In −φ−1
1 Un ).
Thus we have
"
(In − φ1 Un )
−1
=−
∞
X
#
−k
φ−k
1 Un
(φ1 Un )−1 .
k=0
Therefore the solution is
Xn =
−
∞
X
!
Un−k+1
φ−k+1
1
εn ,
k=0
which in it’s limit gives the same solution as Section 2.3.2(ii).
Notice that Unj and B j are playing the same role.
2.4.1
Representing the AR(p) as a vector AR(1)
Let us suppose Xt is an AR(p) process, with the representation
Xt =
p
X
φj Xt−j + εt .
j=1
For the rest of this section we will assume that the roots of the characteristic function, φ(z), lie
outside the unit circle, thus the solution causal. We can rewrite the above as a Vector Autoregressive
58
(VAR(1)) process
X t = AX t−1 + εt
(2.10)
where

φ1 φ2 . . . φp−1 φp


 1


 0

0
0
...
0
1
...
0
0
...
1



0 
,

0 

0
(2.11)
X 0t = (Xt , . . . , Xt−p+1 ) and ε0t = (εt , 0, . . . , 0). It is straightforward to show that the eigenvalues of
A are the inverse of the roots of φ(z) (since
det(A − zI) = z p −
p
X
φi z p−i = z p (1 −
i=1
p
X
φi z −i )),
i=1
|
{z
=z p φ(z −1 )
}
thus the eigenvalues of A lie inside the unit circle. It can be shown that for any |λmax (A)| < δ < 1,
there exists a constant Cδ such that |kAj kspec ≤ Cδ δ j (see Appendix A). Note that result is
extremely obvious if the eigenvalues are distinct (in which case the spectral decomposition can be
used), in which case |kAj kspec ≤ Cδ |λmax (A)|j (note that kAkspec is the spectral norm of A, which
is the largest eigenvalue of the symmetric matrix AA0 ).
We can apply the same back iterating that we did for the AR(1) to the vector AR(1). Iterating
(2.10) backwards k times gives
Xt =
k−1
X
Aj εt−j + Ak X t−k .
j=0
P
Since kAk X t−k k2 ≤ kAk kspec kX t−k k → 0 we have
Xt =
∞
X
Aj εt−j .
j=0
59
2.5
The ARMA model
Up to now, we have defined the moving average and the autoregressive model. The MA(q) average
has the feature that after q lags there isn’t any correlation between two random variables. On
the other hand, there are correlations at all lags for an AR(p) model. In addition as we shall see
later on, it is much easier to estimate the parameters of an AR model than an MA. Therefore,
there are several advantages in fitting an AR model to the data (note that when the roots are
of the characteristic polynomial lie inside the unit circle, then the AR can also be written as an
MA(∞), since it is causal). However, if we do fit an AR model to the data, what order of model
should we use? Usually one uses the AIC (BIC or similar criterion) to determine the order. But
for many data sets, the selected order tends to be relative large, for example order 14. The large
order is usually chosen when correlations tend to decay slowly and/or the autcorrelations structure
is quite complex (not just monotonically decaying). However, a model involving 10-15 unknown
parameters is not particularly parsimonious and more parsimonious models which can model the
same behaviour would be useful. A very useful generalisation which can be more flexible (and
parsimonious) is the ARMA(p, q) model, in this case Xt satisfies
Xt −
p
X
φi Xt−i = εt +
i=1
q
X
θj εt−j .
j=1
Definition 2.5.1 (Summary of AR, ARMA and MA models)
(i) The autoregressive AR(p)
model: {Xt } satisfies
Xt =
p
X
φi Xt−i + εt .
(2.12)
i=1
Observe we can write it as φ(B)Xt = εt
(ii) The moving average M A(q) model: {Xt } satisfies
Xt = εt +
q
X
j=1
Observe we can write Xt = θ(B)εt
60
θj εt−j .
(2.13)
(iii) The autoregressive moving average ARM A(p, q) model: {Xt } satisfies
Xt −
p
X
q
X
φi Xt−i = εt +
i=1
θj εt−j .
(2.14)
j=1
We observe that we can write Xt as φ(B)Xt = θ(B)εt .
Below we give conditions for the ARMA to have a causal solution and also be invertible. We
also show that the coefficients of the MA(∞) representation of Xt will decay exponentially.
Lemma 2.5.1 Let us suppose Xt is an ARMA(p, q) process with representation given in Definition
2.5.1.
(i) If the roots of the polynomial φ(z) lie outside the unit circle, and are greater than (1 + δ) (for
some δ > 0), then Xt almost surely has the solution
Xt =
∞
X
aj εt−j ,
(2.15)
j=0
where for j > q, aj = [Aj ]1,1 +
Pq
i=1 θi [A




A=



φ1
1
..
.
0
where
P
j
φ2
0
..
.
j−i ]
1,1 ,
with
. . . φp−1 φp
...
..
.
... ...
...
..
.
0
..
.
1
0




.



|aj | < ∞ (we note that really aj = aj (φ, θ) since its a function of {φi } and {θi }).
Moreover for all j,
|aj | ≤ Kρj
(2.16)
for some finite constant K and 1/(1 + δ) < ρ < 1.
(ii) If the roots of φ(z) lie both inside or outside the unit circle and are larger than (1 + δ) or less
than (1 + δ)−1 for some δ > 0, then we have
Xt =
∞
X
j=−∞
61
aj εt−j ,
(2.17)
(a vector AR(1) is not possible), where
|aj | ≤ Kρ|j|
(2.18)
for some finite constant K and 1/(1 + δ) < ρ < 1.
(iii) If the absolute value of the roots of θ(z) = 1 +
Pq
j=1 θj z
j
are greater than (1 + δ), then (2.14)
can be written as
Xt =
∞
X
bj Xt−j + εt .
(2.19)
j=1
where
|bj | ≤ Kρj
(2.20)
for some finite constant K and 1/(1 + δ) < ρ < 1.
PROOF. We first prove (i) There are several way to prove the result. The proof we consider here,
uses the VAR expansion given in Section 2.4.1. We write the ARMA process as a vector difference
equation
X t = AX t−1 + εt
where X 0t = (Xt , . . . , Xt−p+1 ), ε0t = (εt +
Pq
j=1 θj εt−j , 0, . . . , 0).
Xt =
∞
X
(2.21)
Now iterating (2.21), we have
Aj εt−j ,
(2.22)
j=0
concentrating on the first element of the vector X t we see that
Xt =
∞
X
[Ai ]1,1 (εt−i +
i=0
q
X
θj εt−i−j ).
j=1
Comparing (2.15) with the above it is clear that for j > q, aj = [Aj ]1,1 +
Pq
i=1 θi [A
j−i ] .
1,1
Observe
that the above representation is very similar to the AR(1). Indeed as we will show below the Aj
behaves in much the same way as the φj in AR(1) example. As with φj , we will show that Aj
converges to zero as j → ∞ (because the eigenvalues of A are less than one). We now show that
62
|Xt | ≤ K
P∞
j
j=1 ρ |εt−j |
for some 0 < ρ < 1, this will mean that |aj | ≤ Kρj . To bound |Xt | we use
(2.22)
|Xt | ≤ kX t k2 ≤
∞
X
kAj kspec kεt−j k2 .
j=0
Hence, by using Section 2.4.1 we have |kAj kspec ≤ Cρ ρj (for any |λmax (A)| < ρ < 1), which gives
the corresponding bound for |aj |.
To prove (ii) we consider a power series expansion of
θ(z)
φ(z) .
If the roots of φ(z) are distinct,
then it is straightforward to write φ(z)−1 in terms of partial fractions and a convergent power
series for |z| = 1. This expansion immediately gives the the linear coefficients aj and show that
|aj | ≤ C(1 + δ)−|j| for some finite constant C. On the other hand, if there are multiple roots, say
P
the roots of φ(z) are λ1 , . . . , λs with multiplicity m1 , . . . , ms (where sj=1 ms = p) then we need to
adjust the partial fraction expansion. It can be shown that |aj | ≤ C|j|maxs |ms | (1 + δ)−|j| . We note
that for every (1 + δ)−1 < ρ < 1, there exists a constant such that |j|maxs |ms | (1 + δ)−|j| ≤ Cρ|j| ,
thus we obtain the desired result.
To show (iii) we use a similar proof to (i), and omit the details.
Corollary 2.5.1 An ARMA process is invertible if the roots of θ(B) (the MA coefficients) lie
outside the unit circle and causal if the roots of φ(B) (the AR coefficients) lie outside the unit
circle.
The representation of an ARMA process is unique upto AR and MA polynomials θ(B) and
φ(B) having common roots. A simplest example is Xt = εt , this also satisfies the representation
Xt − φXt−1 = εt − φεt−1 etc. Therefore it is not possible to identify common factors in the
polynomials.
One of the main advantages of the invertibility property is in prediction and estimation. We will
consider this in detail below. It is worth noting that even if an ARMA process is not invertible, one
can generate a time series which has identical correlation structure but is invertible (see Section
3.3).
63
2.6
Simulating from an Autoregressive process
Simulating from a Gaussian AR process
It is straightforward to simulate from an AR process with Gaussian innovations, {εt }. Given the
autoregressive structure we can deduce the correlation structure (see Chapter 3) (regardless of
the distribution of the innovations). Furthermore, from Lemma 2.5.1(ii) we observe that all AR
processes can be written as the infinite sum of the innovations. Thus if the innovations are Gaussian,
so is the AR process. This allows us to deduce the joint distribution of X1 , . . . , Xp , which in turn
allows us generate the AR(p) process.
We illustrate the details with with an AR(1) process. Suppose Xt = φ1 Xt−1 + εt where {εt }
are iid standard normal random variables (note that for Gaussian processes it is impossible to
discriminate between causal and non-causal processes - see Section 3.3, therefore we will assume
|φ1 | < 1). We will show in Section 3.1, equation (3.1) that the autocovariance of an AR(1) is
c(r) = φr1
∞
X
φ2j
1 =
j=0
φr1
.
1 − φ21
Therefore, the marginal distribution of Xt is Gaussian with variance (1 − φ21 )−1 . Therefore, to
simulate an AR(1) Gaussian time series, we draw from a Gaussian time series with mean zero and
variance (1 − φ21 )−1 , calling this X1 . We then iterate for 2 ≤ t, Xt = φ1 Xt−1 + εt . This will give us
a stationary realization from an AR(1) Gaussian time series.
Note the function arima.sim is a routine in R which does the above. See below for details.
Simulating from a non-Gaussian AR model
Unlike the Gaussian AR process it is difficult to simulate a non-Gaussian model, but we can obtain
a very close approximation. This is because if the innovations are non-Gaussian but known it is not
clear what the distribution of Xt will be. Here we describe how to obtain a close approximation in
the case that the AR process is causal.
Again we describe the method for the AR(1). Let {Xt } be an AR(1) process, Xt = φ1 Xt−1 + εt ,
which has stationary, causal solution
Xt =
∞
X
φj1 εt−j .
j=0
64
˜ 1 = 0. Then obtain the iteration X
˜ t = φ1 X
˜ t−1 + εt for
To simulate from the above model, we set X
t ≥ 2. We note that the solution of this equation is
˜t =
X
t
X
φj1 εt−j .
j=0
˜ t | ≤ |φ1 |t P∞ |φj ε−j |, which converges geometrically
We recall from Lemma 2.5.1 that |Xt − X
j=0 1
˜ t ; t ≥ n} in the simulations
fast to zero. Thus if we choose a large n to allow ‘burn in’ and use {X
we have a simulation which is close to a stationary solution from an AR(1) process.
Simulating from an Integrated process
To simulate from an integrated process ARIMA(p, 1, q) (1 − B)Yt = Xt , where Xt is a causal
ARMA(p, q) process. We first simulate {Xt } using the method above. Then we define the recursion
Y1 = X1 and for t > 1
Yt = Yt−1 + Xt .
Thus giving a realisation from an ARIMA(p, 1, q).
Simulating from a non-Gaussian non-causal model
Suppose that Xt satisfies the representation
Xt =
p
X
φj Xt−j + εt ,
j=1
whose characteristic function have roots both inside and outside the unit circle. Thus, the stationary
solution of this equation is not causal. It is not possible to simulate from this equation. To see why,
consider directly simulating from Xt = 2Xt−1 + εt without rearranging it as Xt−1 = 12 Xt − 12 εt , the
solution would explode. Now if the roots are both inside and outside the unit circle, there would
not be a way to rearrange the equation to iterate a stationary solution. There are two methods to
remedy this problem:
65
(i) From Lemma 2.5.1(ii) we recall that Xt has the solution
Xt =
∞
X
aj εt−j ,
(2.23)
j=−∞
where the coefficients aj are determined from the characteristic equation. Thus to simulate
the process we use the above representation, though we do need to truncate the number of
terms in (2.23) and use
˜t =
X
M
X
aj εt−j .
j=−M
(ii) The above is a brute force method is an approximation which is also difficult to evaluate.
There is a simpler method, if one studies the roots of the characteristic equation.
Let us suppose that {λj1 ; j1 = 1, . . . , p1 } are the roots of φ(z) which lie outside the unit circle
and {µj2 ; j2 = 1, . . . , p2 } are the roots which lie inside the unit circle. For ease of calculation
we will assume the roots are distinct. We can rewrite φ(z)−1 as
φ(z)−1 =
=
=
hQ
p1
j1 =1 (1
p1
X
j1 =1
p1
X
j1 =1
1
i hQ
i
p2
− λj1 z) ·
(1
−
µ
z)
j
2
j2 =1
p2
X
djd
cj1
+
(1 − λj1 z)
(1 − µjd z)
cj1
−
(1 − λj1 z)
j2 =1
p2
X
j2
djd
−1
µ z(1 − µ−1
jd z )
=1 jd
Thus the solution of Xt is
Xt = φ(B)−1 εt =
p2
X
cj1
djd
εt −
−1 −1 εt
(1 − λj1 B)
µ
B(1
−
µ
j
d
jd B )
j =1
=1
p1
X
j1
2
Let Yj1 ,t = λj1 Yj1 ,t−1 + εt and Zj2 ,t = µj2 Zj2 ,t−1 + εt (thus the stationary solution is gener−1
ated with Zj2 ,t−1 = µ−1
j2 Zj2 ,t − µj2 εt ). Generate the time series {Yj1 ,t ; j1 = 1, . . . , p1 } and
{Yj1 ,t ; j1 = 1, . . . , p1 } using the method described above. Then the non-causal time series
can be generated by using
Xt =
p1
X
cj1 Yj1 ,t −
j1 =1
p2
X
j2 =1
66
dj2 Zj2 ,t .
Comments:
– Remember Yj,t is generated using the past εt and Zj,t is generated using future innovations. Therefore to ensure that the generated {Yj,t } and {Zj,t } are close to the
stationary we need to ensure that the initial value of Yj,t is far in the past and the
initial value for Zj,t is far in the future.
– If the roots are complex conjugates, then the corresponding {Yj,t } or {Zj,t } should be
written as AR(2) models (to avoid complex processes).
R functions
Shumway and Stoffer (2006) and David Stoffer’s website gives a comprehensive introduction to time
series R-functions.
The function arima.sim simulates from a Gaussian ARIMA process. For example,
arima.sim(list(order=c(2,0,0), ar = c(1.5, -0.75)), n=150) simulates from the AR(2) model
Xt = 1.5Xt−1 − 0.75Xt−2 + εt , where the innovations are Gaussian.
Exercise 2.6 In the following simulations, use non-Gaussian innovations.
(i) Simulate an AR(4) process with characteristic function
2π
φ(z) = 1 − 0.8 exp(i )z
13
2π
2π
2π
1 − 0.8 exp(−i )z 1 − 1.5 exp(i )z 1 − 1.5 exp(−i )z .
13
5
5
(ii) Simulate an AR(4) process with characteristic function
2π
φ(z) = 1 − 0.8 exp(i )z
13
2π
2
2π
2
2π
1 − 0.8 exp(−i )z 1 − exp(i )z 1 − exp(−i )z .
13
3
5
3
5
Do you observe any differences between these realisations?
67
Chapter 3
The autocovariance function of a
linear time series
Objectives
• Be able to determine the rate of decay of an ARMA time series.
• Be able ‘solve’ the autocovariance structure of an AR process.
• Understand what partial correlation is and how this may be useful in determining the order
of an AR model.
• Understand why autocovariance is ‘blind’ to processes which are non-causal. But the higher
order cumulants are not ‘blind’ to causality.
3.1
The autocovariance function
The autocovariance function (ACF) is defined as the sequence of covariances of a stationary process.
That is suppose that {Xt } is a stationary process with mean zero, then {c(k) : k ∈ Z} is the ACF
of {Xt } where c(k) = E(X0 Xk ). Clearly different time series give rise to different features in the
ACF. We will explore some of these features below.
Before investigating the structure of ARMA processes we state a general result connecting linear
time series and the summability of the autocovariance function.
Lemma 3.1.1 Suppose the stationary time series Xt satisfies the linear representation
68
P∞
j=−∞ ψj εt−j .
(i) If
P∞
(ii) If
P∞
(iii) If
P∞
j=∞ |ψj |
< ∞, then
j=∞ |jψj |
j=∞ |ψj |
2
P
|c(k)| < ∞.
k
< ∞, then
P
k
|k · c(k)| < ∞.
< ∞, then we cannot say anything about summability of the covariance.
PROOF. It is straightforward to show that
c(k) = var[εt ]
X
ψj ψj−k .
j
P
Using this result, it is easy to see that
k
|c(k)| ≤
P P
k
j
|ψj | · |ψj−k |, thus
P
k
|c(k)| < ∞, which
proves (i).
P
The proof of (ii) is similar. To prove (iii), we observe that j |ψj |2 < ∞ is a weaker condition
P
then j |ψj | < ∞ (for example the sequence ψj = |j|−1 satisfies the former condition but not the
latter). Thus based on the condition we cannot say anything about summability of the covariances.
First we consider a general result on the covariance of a causal ARMA process (always to obtain
the covariance we use the MA(∞) expansion - you will see why below).
3.1.1
The rate of decay of the autocovariance of an ARMA process
We evaluate the covariance of an ARMA process using its MA(∞) representation. Let us suppose
that {Xt } is a causal ARMA process, then it has the representation in (2.17) (where the roots of
φ(z) have absolute value greater than 1 + δ). Using (2.17) and the independence of {εt } we have
cov(Xt , Xτ ) = cov(
∞
X
aj1 εt−j1 ,
j1 =0
=
∞
X
∞
X
aj2 ετ −j2 )
j2 =0
aj1 aj2 cov(εt−j , ετ −j ) =
j=0
∞
X
aj aj+|t−τ | var(εt )
(3.1)
j=0
(here we see the beauty of the MA(∞) expansion). Using (2.18) we have
|cov(Xt , Xτ )| ≤
var(εt )Cρ2
∞
X
j j+|t−τ |
ρ ρ
≤
Cρ2 ρ|t−τ |
∞
X
j=0
j=0
for any 1/(1 + δ) < ρ < 1.
69
ρ2j =
ρ|t−τ |
,
1 − ρ2
(3.2)
The above bound is useful, it tells us that the ACF of an ARMA process decays exponentially
fast. In other words, there is very little memory in an ARMA process. However, it is not very
enlightening about features within the process. In the following we obtain an explicit expression for
the ACF of an autoregressive process. So far we have used the characteristic polynomial associated
with an AR process to determine whether it was causal. Now we show that the roots of the
characteristic polynomial also give information about the ACF and what a ‘typical’ realisation of
a autoregressive process could look like.
3.1.2
The autocovariance of an autoregressive process
Let us consider the zero mean AR(p) process {Xt } where
Xt =
p
X
φj Xt−j + εt .
(3.3)
j=1
From now onwards we will assume that {Xt } is causal (the roots of φ(z) lie outside the unit circle).
Given that {Xt } is causal we can derive a recursion for the covariances. It can be shown that
multipying both sides of the above equation by Xt−k (k ≤ 0) and taking expectations, gives the
equation
E(Xt Xt−k ) =
p
X
j=1
φj E(Xt−j Xt−k ) + E(εt Xt−k ) =
| {z }
=0
p
X
φj E(Xt−j Xt−k ).
(3.4)
j=1
It is worth mentioning that if the process were not causal this equation would not hold, since εt
and Xt−k are not necessarily independent. These are the Yule-Walker equations, we will discuss
them in detail when we consider estimation. For now letting c(k) = E(X0 Xk ) and using the above
we see that the autocovariance satisfies the homogenuous difference equation
c(k) −
p
X
φj c(k − j) = 0,
(3.5)
j=1
for k ≥ 0. In other words, the autocovariance function of {Xt } is the solution of this difference
equation. The study of difference equations is a entire field of research, however we will now scratch
the surface to obtain a solution for (3.5). Solving (3.5) is very similar to solving homogenuous
differential equations, which some of you may be familar with (do not worry if you are not).
P
Recall the characteristic polynomial of the AR process φ(z) = 1 − pj=1 φj z j = 0, which has
70
the roots λ1 , . . . , λp . In Section 2.3.4 we used the roots of the characteristic equation to find the
stationary solution of the AR process. In this section we use the roots characteristic to obtain the
solution (3.5). It can be shown if the roots are distinct (the roots are all different) the solution of
(3.5) is
c(k) =
p
X
Cj λ−k
j ,
(3.6)
j=1
where the constants {Cj } are chosen depending on the initial values {c(k) : 1 ≤ k ≤ p} and are
such that they ensure that c(k) is real (recalling that λj ) can be complex.
The simplest way to prove (3.6) is to use a plugin method. Plugging c(k) =
Pp
−k
j=1 Cj λj
into
(3.5) gives
c(k) −
p
X
φj c(k − j) =
j=1
p
X
p
X
−(k−i)
−k
φi λ j
Cj λ j −
i=1
j=1
=
p
X
p
X
−k
i
Cj λj
1−
φi λj = 0.
j=1
i=1
|
{z
φ(λi )
}
In the case that the roots of φ(z) are not distinct, let the roots be λ1 , . . . , λs with multiplicity
P
m1 , . . . , ms ( sk=1 mk = p). In this case the solution is
c(k) =
s
X
λ−k
j Pmj (k),
(3.7)
j=1
where Pmj (k) is mj th order polynomial and the coefficients {Cj } are now ‘hidden’ in Pmj (k). We
now study the covariance in greater details and see what it tells us about a realisation. As a
motivation consider the following example.
Example 3.1.1 Consider the AR(2) process
Xt = 1.5Xt−1 − 0.75Xt−2 + εt ,
(3.8)
where {εt } are iid random variables with mean zero and variance one. The corresponding characp
teristic polynomial is 1 − 1.5z + 0, 75z 2 , which has roots 1 ± i3−1/2 = 4/3 exp(iπ/6). Using the
71
discussion above we see that the autocovariance function of {Xt } is
p
c(k) = ( 4/3)−k (C1 exp(−ikπ/6) + C¯1 exp(ikπ/6)),
for a particular value of C1 . Now write C1 = a exp(ib), then the above can be written as
π
p
c(k) = a( 4/3)−k cos k + b .
6
We see that the covariance decays at an exponential rate, but there is a periodicity within the decay.
This means that observations separated by a lag k = 12 are more closely correlated than other lags,
this suggests a quasi-periodicity in the time series. The ACF of the process is given in Figure 3.1,
notice that it has decays to zero but also observe that it undulates. A plot of a realisation of the time
series is given in Figure 3.2, notice the quasi-periodicity of about 2π/12. Let is now briefly return
P
to the definition of the periodogram given in Section 1.2.3 (In (ω) = n1 | nt=1 Xt exp(itω)|2 ). We
used the periodogram to identify the periodogram of a deterministic signal, but showed that when
dependent, correlated noise was added to the dignal that the periodogram exhibited more complex
behaviour. In Figure 6.1 we give a plot of the periodogram corresponding to Figure 3.2. Recall
that this AR(2) gives a quasi-periodicity of 12, which corresponds to the frequency 2π/12 ≈ 0.52,
which matches the main peaks in periodogram. We will learn later that the periodogram is a ‘crude’
(meaning inconsistent) estimator of the spectral density function. The spectral density if given in
the lower plot of Figure 6.1.
We now generalise the above example. Let us consider the general AR(p) process defined in (3.3).
Suppose the roots of the corresponding characteristic polynomial are distinct and let us split them
into real and complex roots. Because the characteristic polynomial is comprised of real coefficients,
the complex roots come in complex conjugate pairs. Hence let us suppose the real roots are {λj }rj=1
(p−r)/2
and the complex roots are {λj , λj }j=r+1 . The covariance in (3.6) can be written as
c(k) =
r
X
j=1
(p−2)/2
Cj λ−k
j
+
X
aj |λj |−k cos(kθj + bj )
(3.9)
j=r+1
where for j > r we write λj = |λj | exp(iθj ) and aj and bj are real constants. Notice that as the
example above the covariance decays exponentially with lag, but there is undulation. A typical
realisation from such a process will be quasi-periodic with periods at θr+1 , . . . , θ(p−r)/2 , though the
72
1.0
0.8
0.6
0.4
−0.4
−0.2
0.0
0.2
acf
0
10
20
30
40
50
lag
0
−2
−4
ar2
2
4
6
Figure 3.1: The ACF of the time series Xt = 1.5Xt−1 − 0.75Xt−2 + εt
0
24
48
72
96
120
144
Time
Figure 3.2: The a simulation of the time series Xt = 1.5Xt−1 − 0.75Xt−2 + εt
73
● ●
0.5
● ●
0.3
●
●
●
●
0.2
Periodogram
0.4
●
●
●
0.1
●
●
●
●
●
●
● ●
0.0
0
●
●
●
●
●
●
1
2
3
4
5
●
● ●
●
●
● ●
● ●
●
●
●
●● ●
●●●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●●
●● ●
● ●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
6
frequency
30
0
10
20
spectrum
40
50
60
Autoregressive
0.0
0.1
0.2
0.3
0.4
0.5
frequency
Figure 3.3: Top: Periodogram of Xt = 1.5Xt−1 − 0.75Xt−2 + εt for sample size n = 144.
Lower: The corresponding spectral density function (note that 0.5 of the x-axis on spectral
density corresponds to π on the x-axis of the periodogram).
74
magnitude of each period will vary.
An interesting discussion on covariances of an AR process and realisation of an AR process is
given in Shumway and Stoffer (2006), Chapter 3.3 (it uses the example above). A discussion of
difference equations is also given in Brockwell and Davis (1998), Sections 3.3 and 3.6 and Fuller
(1995), Section 2.4.
Example 3.1.2 (Autocovariance of an AR(2)) Let us suppose that Xt satisfies the model Xt =
(a + b)Xt−1 − abXt−2 + εt . We have shown that if |a| < 1 and |b| < 1, then it has the solution
Xt =
∞
X
1
bj+1 − aj+1 )εt−j .
b−a
j=0
By writing a ‘timeline’ it is straightfoward to show that for r > 1
cov(Xt , Xt−r ) =
∞
X
(bj+1 − aj+1 )(bj+1+r − aj+1+r ).
j=0
Example 3.1.3 The autocorrelation of a causal and noncausal time series Let us consider the two
AR(1) processes considered in Section 2.3.2. We recall that the model
Xt = 0.5Xt−1 + εt
has the stationary causal solution
Xt =
∞
X
0.5j εt−j .
j=0
Assuming the innovations has variance one, the ACF of Xt is
cX (0) =
1
1 − 0.52
cX (k) =
On the other hand the model
Yt = 2Yt−1 + εt
75
0.5|k|
1 − 0.52
has the noncausal stationary solution
Yt = −
∞
X
(0.5)j+1 εt+j+1 .
j=0
Thus process has the ACF
cY (0) =
0.52
1 − 0.52
cX (k) =
0.52+|k|
.
1 − 0.52
Thus we observe that except for a factor (0.5)2 both models has an identical autocovariance function.
Indeed their autocorrelation function would be same. Furthermore, by letting the innovation of Xt
have standard deviation 0.5, both time series would have the same autocovariance function.
Therefore, we observe an interesting feature, that the non-causal time series has the same
correlation structure of a causal time series. In Section 3.3 that for every non-causal time series
there exists a causal time series with the same autocovariance function. Therefore autocorrelation
is ‘blind’ to non-causality.
Exercise 3.1 Recall the AR(2) models considered in Exercise 2.4. Now we want to derive their
ACF functions.
(i)
(a) Obtain the ACF corresponding to
2
7
Xt = Xt−1 − Xt−2 + εt ,
3
3
where {εt } are iid random variables with mean zero and variance σ 2 .
(b) Obtain the ACF corresponding to
√
4× 3
42
Xt =
Xt−1 − 2 Xt−2 + εt ,
5
5
where {εt } are iid random variables with mean zero and variance σ 2 .
(c) Obtain the ACF corresponding to
Xt = Xt−1 − 4Xt−2 + εt ,
where {εt } are iid random variables with mean zero and variance σ 2 .
76
(ii) For all these models plot the true ACF in R. You will need to use the function ARMAacf.
BEWARE of the ACF it gives for non-causal solutions. Find a method of plotting a causal
solution in the non-causal case.
Exercise 3.2 In Exercise 2.5 you constructed a causal AR(2) process with period 17.
Load Shumway and Stoffer’s package asta into R (use the command install.packages("astsa")
and then library("astsa").
Use the command arma.spec to make a plot of the corresponding spectral density function. How
does your periodogram compare with the ‘true’ spectral density function?
R code
We use the code given in Shumway and Stoffer (2006), page 101 to make Figures 3.1 and 3.2.
To make Figure 3.1:
a c f = ARMAacf( a r=c ( 1 . 5 , − 0 . 7 5 ) ,ma=0 ,50)
p l o t ( a c f , type=”h ” , x l a b=” l a g ” )
a b l i n e ( h=0)
To make Figures 3.2 and 6.1:
set . seed (5)
a r 2 <− arima . sim ( l i s t ( o r d e r=c ( 2 , 0 , 0 ) , a r = c ( 1 . 5 , − 0 . 7 5 ) ) , n=144)
p l o t . t s ( ar2 , a x e s=F ) ; box ( ) ; a x i s ( 2 )
axis (1 , seq (0 ,144 ,24))
a b l i n e ( v=s e q ( 0 , 1 4 4 , 1 2 ) , l t y =”d o t t e d ” )
p l o t ( f r e q u e n c y , Periodogram , type=”o ” )
l i b r a r y (” a s t s a ”)
arma . s p e c ( a r = c ( 1 . 5 , −0.75) , l o g = ”no ” , main = ” A u t o r e g r e s s i v e ” )
3.1.3
The autocovariance of a moving average process
Suppose that {Xt } satisfies
Xt = εt +
q
X
j=1
77
θj εt−j .
The covariance is
cov(Xt , Xt−k ) =
 P
p

i=0 θi θi−k
k = −q, . . . , q
0

otherwise
where θ0 = 1 and θi = 0 for i < 0 and i ≥ q. Therefore we see that there is no correlation when
the lag between Xt and Xt−k is greater than q.
3.1.4
The autocovariance of an autoregressive moving average process
We see from the above that an MA(q) model is only really suitable when we believe that there
is no correlaton between two random variables separated by more than a certain distance. Often
autoregressive models are fitted. However in several applications we find that autoregressive models
of a very high order are needed to fit the data. If a very ‘long’ autoregressive model is required
a more suitable model may be the autoregressive moving average process. It has several of the
properties of an autoregressive process, but can be more parsimonuous than a ‘long’ autoregressive
process. In this section we consider the ACF of an ARMA process.
Let us suppose that the causal time series {Xt } satisfies the equations
Xt −
p
X
φi Xt−i = εt +
i=1
q
X
θj εt−j .
j=1
We now define a recursion for ACF, which is similar to the ACF recursion for AR processes. Let
us suppose that the lag k is such that k > q, then it can be shown that the autocovariance function
of the ARMA process satisfies
E(Xt Xt−k ) −
p
X
φi E(Xt−i Xt−k ) = 0
i=1
On the other hand, if k ≤ q, then we have
E(Xt Xt−k ) −
p
X
i=1
φi E(Xt−i Xt−k ) =
q
X
θj E(εt−j Xt−k ) =
j=1
We recall that Xt has the MA(∞) representation Xt =
78
q
X
θj E(εt−j Xt−k ).
j=k
P∞
j=0 aj εt−j
(see (2.17)), therefore for
k ≤ j ≤ q we have E(εt−j Xt−k ) = aj−k var(εt ) (where a(z) = θ(z)φ(z)−1 ). Altogether the above
gives the difference equations
c(k) −
c(k) −
p
X
i=1
p
X
φi c(k − i) = var(εt )
q
X
θj aj−k
for 1 ≤ k ≤ q
(3.10)
j=k
φi c(k − i) = 0, for k > q,
i=1
where c(k) = E(X0 Xk ). (3.10) is homogenuous difference equation, then it can be shown that the
solution is
c(k) =
s
X
λ−k
j Pmj (k),
j=1
P
where λ1 , . . . , λs with multiplicity m1 , . . . , ms ( k ms = p) are the roots of the characteristic
P
polynomial 1 − pj=1 φj z j . Observe the similarity to the autocovariance function of the AR process
(see (3.7)). The coefficients in the polynomials Pmj are determined by the initial condition given
in (3.10).
You can also look at Brockwell and Davis (1998), Chapter 3.3 and Shumway and Stoffer (2006),
Chapter 3.4.
3.2
The partial covariance and correlation of a time
series
We see that by using the autocovariance function we are able to identify the order of an MA(q)
process: when the covariance lag is greater than q the covariance is zero. However the same is
not true for AR(p) processes. The autocovariances do not enlighten us on the order p. However
a variant of the autocovariance, called the partial autocovariance is quite informative about order
of AR(p). We start by reviewing the partial autocovariance, and it’s relationship to the inverse
variance/covariance matrix (often called the precision matrix).
79
3.2.1
A review of partial correlation in multivariate analysis
Partial correlation
Suppose X = (X1 , . . . , Xd ) is a zero mean random vector (we impose the zero mean condition to
simplify notation and it’s not necessary). The partial correlation is the covariance between Xi and
Xj , conditioned on the other elements in the vector. In other words, the covariance between the
residuals of Xi conditioned on X −(ij) (the vector not containing Xi and Xj ) and the residual of Xj
conditioned on X −(ij) . That is the partial covariance between Xi and Xj given X −(ij) is defined
as
cov Xi − var[X −(ij) ]−1 E[X −(ij) Xi ]X −(ij) , Xj − var[X −(ij) ]−1 E[X −(ij) Xj ]X −(ij)
= cov[Xi Xj ] − E[X −(ij) Xi ]0 var[X −(ij) ]−1 E[X −(ij) Xj ].
Taking the above argument further, the variance/covariance matrix of the residual of X ij =
(Xi , Xj )0 given X −(ij) is defined as
var X ij − E[X ij ⊗ X −(ij) ]0 var[X −(ij) ]−1 X −(ij) = Σij − c0ij Σ−1
−(ij) cij
(3.11)
where Σij = var(X ij ), cij = E(X ij ⊗ X −(ij) ) (=cov(X ij , X −(ij) )) and Σ−(ij) = var(X −(ij) )
(⊗ denotes the tensor product). Let sij denote the (i, j)th element of the (2 × 2) matrix Σij −
c0ij Σ−1
−(ij) cij . The partial correlation between Xi and Xj given X −(ij) is
ρij = √
s12
,
s11 s22
observing that
(i) s12 is the partial covariance between Xi and Xj .
(ii) s11 = E(Xi −
P
k6=i,j
βi,k Xk )2 (where βi,k are the coefficients of the best linear predictor of
Xi given {Xk ; k 6= i, j}).
(ii) s22 = E(Xj −
P
k6=i,j
βj,k Xk )2 (where βj,k are the coefficients of the best linear predictor of
Xj given {Xk ; k 6= i, j}).
In the following section we relate partial correlation to the inverse of the variance/covariance matrix
(often called the precision matrix).
80
The precision matrix and it’s properties
Let us suppose that X = (X1 , . . . , Xd ) is a zero mean random vector with variance Σ. The (i, j)th
element of Σ the covariance cov(Xi , Xj ) = Σij . Here we consider the inverse of Σ, and what
information the (i, j)th of the inverse tells us about the correlation between Xi and Xj . Let Σij
denote the (i, j)th element of Σ−1 . We will show that with appropriate standardisation, Σij is the
negative partial correlation between Xi and Xj . More precisely,
√
Σij
Σii Σjj
= −ρij .
(3.12)
The proof uses the inverse of block matrices. To simplify the notation, we will focus on the (1, 2)th
element of Σ and Σ−1 (which concerns the correlation between X1 and X2 ). Let X 1,2 = (X1 , X2 )0 ,
X −(1,2) = (X3 , . . . , Xd )0 , Σ−(1,2) = var(X −(1,2) ), c1,2 = cov(X (1,2) , X −(1,2) ) and Σ1,2 = var(X 1,2 ).
Using this notation it is clear that

var(X) = Σ = 

Σ1,2
c1,2
c01,2
Σ−(1,2)
(3.13)

It is well know that the inverse of the above block matrix is

Σ−1 = 
−P −1 c01,2 Σ−1
−(1,2)
P −1
−1 P −1 + Σ−1
−1 c0 Σ−1
−Σ−1
1,2 −(1,2)
−(1,2) c1,2 P
−(1,2) c1,2 P

,
(3.14)
where P = (Σ1,2 − c01,2 Σ−1
−(1,2) c1,2 ). Comparing P with (3.11), we see that P is the 2 × 2 variance/covariance matrix of the residuals of X(1,2) conditioned on X −(1,2) . Thus the partial correlation
between X1 and X2 is
P1,2
ρ1,2 = p
P1,1 P2,2
(3.15)
where Pij denotes the elements of the matrix P . Inverting P (since it is a two by two matrix), we
see that

P −1 =
P2,2
1

2
P1,1 P2,2 − P1,2
−P1,2
81
−P1,2
P11

.
(3.16)
Thus, by comparing (3.14) and (3.16) and by the definition of partial correlation given in (3.15) we
have
−1
P1,2
= −ρ1,2 .
Let Σij denote the (i, j)th element of Σ−1 . Thus we have shown (3.12):
ρij = − √
Σij
Σii Σjj
.
In other words, the (i, j)th element of Σ−1 divided by the square root of it’s diagonal gives negative
partial correlation. Therefore, if the partial correlation between Xi and Xj given Xij is zero, then
Σi,j = 0.
The precision matrix, Σ−1 , contains many other hidden treasures. For example, the coefficients
of Σ−1 convey information about the best linear predictor Xi given X −i = (X1 , . . . , Xi−1 , Xi+1 , . . . , Xd )
(all elements of X except Xi ). Let
Xi =
X
βi,j Xj + εi ,
j6=i
where {βi,j } are the coefficients of the best linear predictor. Then it can be shown that
βi,j = −
Σij
Σii
and
Σii =
E[Xi −
1
P
j6=i βi,j Xj ]
2
.
(3.17)
The proof uses the same arguments as those in (3.13).
Exercise 3.3 By using the decomposition

var(X) = Σ = 
Σ1
c1
c01
Σ−(1)


(3.18)
where Σ1 = var(X1 ), c1 = E[X1 X 0−1 ] and Σ−(1) = var[X −1 ] prove (3.17).
The Cholesky decomposition and the precision matrix
We now represent the precision matrix through it’s Cholesky decomposition. It should be mentioned
that Mohsen Pourahmadi has done a lot of interesting research in this area and he recently wrote
82
a review paper, which can be found here.
We define the sequence of linear equations
Xt =
t−1
X
βt,j Xj + εt ,
t = 2, . . . , k,
(3.19)
j=1
where {βt,j ; 1 ≤ j ≤ t − 1} are the coefficeints of the best linear predictor of Xt given X1 , . . . , Xt−1 .
P
2
2
Let σt2 = var[εt ] = E[Xt − t−1
j=1 βt,j Xj ] and σ1 = var[X1 ]. We standardize (3.19) and define
t
X

γt,j Xj =
j=1
1 
Xt −
σt
t−1
X

βt,j Xj  ,
(3.20)
j=1
where we note that γt,t = σt−1 and for 1 ≤ j < t − 1, γt,j = −βt,j /σi . By construction it is clear
that var(LX) = Ik , where

γ1,1
0
0
...
0
0


0 ...
0
0
 γ2,1 γ2,2


L =  γ3,1 γ3,2 γ3,3 . . .
0
0

 ..
..
..
..
..
..
 .
.
.
.
.
.

γk,1 γk,2 γk,3 . . . γk,k−1 γk,k











(3.21)
and LL0 = Σ−1 (see ?, equation (18)), where Σ = var(X k ). Let Σt = var[X t ], then
Σij
t
=
t
X
γik γjk .
k=1
We use apply these results to the analysis of the partial correlations of autoregressive processes
and the inverse of it’s variance/covariance matrix.
3.2.2
Partial correlation in time series
The partial covariance/correlation of a time series is defined in a similar way.
Definition 3.2.1 The partial covariance/correlation between Xt and Xt+k+1 is defined as the partial covariance/correlation between Xt and Xt+k+1 after conditioning out the ‘inbetween’ time series
Xt+1 , . . . , Xt+k .
83
We now obtain an expression for the partial correlation between Xt and Xt+k+1 in terms of their
autocovariance function (for the final result see equation (3.22)). As the underlying assumption
is that the time series is stationary it is the same as the partial covariance/correlation Xk+1 and
X0 . In Chapter 5 we will introduce the idea of linear predictor of a future time point given the
present and the past (usually called forecasting) this can be neatly described using the idea of
projections onto subspaces. This notation is quite succinct, therefore we derive an expression for
the partial correlation using projection notation. The projection of Xk+1 onto the space spanned
by X k = (X1 , X2 , . . . , Xk ), is the best linear predictor of Xk+1 given X k . We will denote the
projection of Xk onto the space spanned by X1 , X2 , . . . , Xk as PX k (Xk+1 ) (note that this is the
same as the best linear predictor). Thus
PX k (Xk+1 ) =
X 0k (var[X k ]−1 E[Xk+1 X k ])−1
=
X 0k Σ−1
k ck
:=
k
X
φk,j Xj ,
j=1
where Σk = var(X k ) and ck = E(Xk+1 X k ). To derive a similar expression for PX k (X0 ) we use the
stationarity property
PX k (Xk+1 ) = X 0k (var[X k ]−1 E[X0 X k ])
= X 0k (var[X k ]−1 Ek E[Xk+1 X k ])
−1
0
= X 0k Σ−1
k Ek ck = X k Ek Σk ck :=
k
X
φk,k+1−j Xj ,
j=1
where Ek is a matrix which swops round all the elements in a vector




Ek = 



0 0 0 ... 0 1
0 0 0 ... 1 0
.. .. .. .. ..
. . . . .
.
1 0 .. 0 0 0




.



Thus the partial correlation between Xt and Xt+k (where k > 0) is the correlation X0 − PX k (X0 )
and Xk+1 − PX k (Xk+1 ), which is
cov(Xk+1 − PX k (Xk+1 ), X0 − PX k (X0 ))
= cov(Xk+1 X0 ) − c0k Σ−1
k Ek ck .
84
(3.22)
We consider an example.
Example 3.2.1 (The PACF of an AR(1) process) Consider the causal AR(1) process Xt =
0.5Xt−1 + εt where E(εt ) = 0 and var(εt ) = 1. Using (3.1) it can be shown that cov(Xt , Xt−2 ) =
2×0.52 (compare with the MA(1) process Xt = εt +0.5εt−1 , where the covariance cov(Xt , Xt−2 ) = 0).
We evaluate the partial covariance between Xt and Xt−2 . Remember we have to ‘condition out’ the
random variables inbetween, which in this case is Xt−1 . It is clear that the projection of Xt onto
Xt−1 is 0.5Xt−1 (since Xt = 0.5Xt−1 + εt ). Therefore Xt − Psp(X
Xt = Xt − 0.5Xt−1 = εt . The
¯
t−1 )
projection of Xt−2 onto Xt−1 is a little more complicated, it is Psp(X
Xt−2 =
¯
t−1 )
E(Xt−1 Xt−2 )
Xt−1 .
2 )
E(Xt−1
Therefore the partial correlation between Xt and Xt−2
cov Xt − PXt−1 Xt , Xt−2 − PXt−1 ) Xt−2
E(Xt−1 Xt−2 )
Xt−1
= cov εt , Xt−2 −
2 )
E(Xt−1
= 0.
In fact the above is true for the partial covariance between Xt and Xt−k , for all k ≥ 2. Hence we
see that despite the covariance not being zero for the autocovariance of an AR process greater than
order two, the partial covariance is zero for all lags greater than or equal to two.
Using the same argument as above, it is easy to show that partial covariance of an AR(p) for
lags greater than p is zero. Hence in may respects the partial covariance can be considered as an
analogue of the autocovariance. It should be noted that though the covariance of MA(q) is zero
for lag greater than q, the same is not true for the parial covariance. Whereas partial covariances
removes correlation for autoregressive processes it seems to ‘add’ correlation for moving average
processes!
Model identification:
• If the autocovariances after a certain lag are zero q, it may be appropriate to fit an MA(q)
model to the time series.
On the other hand, the autocovariances of any AR(p) process will only decay to zero as the
lag increases.
• If the partial autocovariances after a certain lag are zero p, it may be appropriate to fit an
AR(p) model to the time series.
On the other hand, the partial covariances of any MA(p) process will only decay to zero as
the lag increases.
85
Exercise 3.4 (The partial correlation of an invertible MA(1)) Let φt,t denote the partial correlation between Xt+1 and X1 . It is well known (this is the Levinson-Durbin algorithm, which we
cover in Chapter 5) that φt,t can be deduced recursively from the autocovariance funciton using the
algorithm:
Step 1 φ1,1 = c(1)/c(0) and r(2) = E[X2 − X2|1 ]2 = E[X2 − φ1,1 X1 ]2 = c(0) − φ1,1 c(1).
Step 2 For j = t
φt,t =
φt,j
c(t) −
= φt−1,j
Pt−1
j=1 φt−1,j c(t
r(t)
− φt,t φt−1,t−j
− j)
1 ≤ j ≤ t − 1,
and r(t + 1) = r(t)(1 − φ2t,t ).
(i) Using this algorithm show that the PACF of the MA(1) process Xt = εt + θεt−1 , where |θ| < 1
(so it is invertible) is
φt,t =
(−1)t+1 (θ)t (1 − θ2 )
.
1 − θ2(t+1)
(ii) Explain how this partial correlation is similar to the ACF of the AR(1) model Xt = −θXt−1 +
εt .
Exercise 3.5 (Comparing the ACF and PACF of an AR process) Compare the below plots:
(i) Compare the ACF and PACF of the AR(2) model Xt = 1.5Xt−1 − 0.75Xt−2 + εt using
ARIMAacf(ar=c(1.5,-0.75),ma=0,30) and ARIMAacf(ar=c(1.5,-0.75),ma=0,pacf=T,30).
(ii) Compare the ACF and PACF of the MA(1) model Xt = εt −0.5εt using ARIMAacf(ar=0,ma=c(-1.5),30)
and ARIMAacf(ar=0,ma=c(-1.5),pacf=T,30).
(ii) Compare the ACF and PACF of the ARMA(2, 1) model Xt − 1.5Xt−1 + 0.75Xt−2 = εt − 0.5εt
using ARIMAacf(ar=c(1.5,-0.75),ma=c(-1.5),30) and
ARIMAacf(ar=c(1.5,0.75),ma=c(-1.5),pacf=T,30).
Exercise 3.6 Compare the ACF and PACF plots of the monthly temperature data from 1996-2014.
Would you fit an AR, MA or ARMA model to this data?
86
Rcode
The sample partial autocorrelation of a time series can be obtained using the command pacf.
However, remember just because the sample PACF is not zero, does not mean the true PACF is
non-zero. This is why we require error bars!
3.2.3
The variance/covariance matrix and precision matrix of an
autoregressive and moving average process
Let us suppose that {Xt } is a stationary time series. In this section we consider the variance/covariance matrix var(X k ) = Σk , where X k = (X1 , . . . , Xk )0 . We will consider two cases (i) when
Xt follows an MA(p) models and (ii) when Xt follows an AR(p) model. The variance and inverse
of the variance matrices for both cases yield quite interesting results. We will use classical results
from multivariate analysis, stated in Section 3.2.1.
We recall that the variance/covariance matrix of a stationary time series has a (symmetric)
Toeplitz structure (see wiki for a definition). Let X k = (X1 , . . . , Xk )0 , then




Σk = var(X k ) = 



c(0)
c(1)
0
. . . c(k − 2) c(k − 1)
c(1) . . . c(k − 3) c(k − 2)
..
..
..
.
.
.
..
c(k − 1) c(k − 2)
.
...
c(1)
c(0)
c(1)
..
.
c(0)
..
.




.



Σk for AR(p) and MA(p) models
(i) If {Xt } satisfies an MA(p) model and k > p, then Σk will be bandlimited, where p offdiagonals above and below the diagonal will be non-zero and the rest of the off-diagonal will
be zero.
(ii) If {Xt } satisfies an AR(p) model, then Σk will not be bandlimited.
Σ−1
k for an AR(p) model
We now consider the inverse of Σk . Warning: note that the inverse of a Toeplitz is not necessarily
Toeplitz (unlike the circulant which is). We use the results in Section 3.2.1. Suppose that we have
an AR(p) process and we consider the precision matrix of X k = (X1 , . . . , Xk ), where k > p.
87
Recall the (i, j)th element of Σ−1
k divided by the square roots of the corresponding diagonals is
the negative partial correlation of between Xi and Xj conditioned on all the elements in X k . In
Section 3.2.2 we showed that if |i − j| > p, then the partial correlation between Xi and Xj given
Xi+1 , . . . , Xj−1 (assuming without loss of generality that i < j) is zero. We now show that the
ij
precision matrix of Σ−1
k will be bandlimited (note that it is not immediate obvious since Σk is the
negative partial correlation between Xi and Xj given X−(ij) not just the elements between Xi and
Xj ). To show this we use the Cholesky decomposition given in (??) and (3.23). Since Xt is an
autoregressive process of order p and plugging this information into (3.19), for t > p we have
Xt =
t−1
X
βt,j Xj + εt =
j=1
p
X
φj Xt−j + εt ,
j=1
thus βt,t−j = φj for 1 ≤ j ≤ p otherwise βt,t−j = 0. Moreover, for t > p we have σt2 = var(εt ) = 1.
For t ≤ p we use the same notation as that used in (3.19). This gives the lower triangular pbandlimited matrix

γ1,1
0


 γ2,1
γ2,2

 ..
..
 .
.


 −φp −φp−1

 .
..
.
Lk = 
.
 .

 0
0



0
 0

.
..
 .
 .
.

0
0
...
0
0
...
0
0
...
..
.
0
..
.
0
..
.
...
..
.
0
..
.
0
..
.
1
..
.
...
..
.
0
..
.
0
..
.
. . . −φ1
..
..
.
.
. . . −φp −φp−1 . . . −φ1
...
..
.
0
..
.
−φp
..
.
...
0
0
1
. . . −φ2 −φ1
..
..
..
.
.
.
...
0
0
0 ... 0



0 ... 0 

.. .. .. 
. . . 


0 ... 0 

.. .. .. 
. . . 


0 ... 0 



1 ... 0 

.. .. .. 
. . . 

0 ... 1
(3.23)
(the above matrix has not been formated well, but after the first p − 1 rows, there are ones along
the diagonal and the p lower off-diagonals are non-zero).
0
We recall that Σ−1
k = Lk Lk , thus we observe that since Lk is a lower triangular bandlimited
0
matrix, Σ−1
k = Lk Lk is a bandlimited matrix with the p off-diagonals either side of the diagonal
(i,j) = 0 if |i − j| > p.
non-zero. Let Σij denote the (i, j)th element of Σ−1
k . Then we observe that Σ
P
Moreover, if 0 < |i − j| ≤ p and either i or j is greater than p, then Σij = 2 pk=|i−j| φk φk−|i−j|+1 −
φ|i−j| .
88
The coefficients Σ(i,j) gives us a fascinating insight into the prediction of Xt given the past
and future observations. We recall from equation (3.17) that −Σij /Σii are the coffficients of the
best linear predictor of Xi given X −i . This result tells if the observations came from a stationary
AR(p) process, then the best linear predictor of Xi given Xi−1 , . . . , Xi−a and Xi+1 , . . . , Xi+b (where
a and b > p) is the same as the best linear predictor of Xi given Xi−1 , . . . , Xi−p and Xi+1 , . . . , Xi+p
(knowledge of other values will not improve the prediction).
There is an interesting duality between the AR and MA model which we will explore further
in the course.
3.3
Correlation and non-causal time series
Here we demonstrate that it is not possible to identify whether a process is noninvertible/noncausal
from its covariance structure. The simplest way to show result this uses the spectral density
function, which will now define and then return to and study in depth in Chapter ??.
Definition 3.3.1 (The spectral density) Given the covariances c(k) (with
P
k
|c(k)|2 < ∞) the
spectral density function is defined as
f (ω) =
X
c(k) exp(ikω).
k
The covariances can be obtained from the spectral density by using the inverse fourier transform
c(k) =
1
2π
Z
2π
f (ω) exp(−ikω).
0
Hence the covariance yields the spectral density and visa-versa.
For reference below, we point out that the spectral density function uniquely identifies the autocovariance function.
Let us suppose that {Xt } satisfies the AR(p) representation
Xt =
p
X
φi Xt−i + εt
i=1
where var(εt ) = 1 and the roots of φ(z) = 1 −
Pp
j=1 φj z
j
can lie inside and outside the unit circle,
but not on the unit circle (thus it has a stationary solution). We will show in Chapter ?? that the
89
spectral density of this AR process is
f (ω) =
|1 −
1
.
2
j=1 φj exp(ijω)|
(3.24)
Pp
• Factorizing f (ω).
P
Let us supose the roots of the characteristic polynomial φ(z) = 1 + qj=1 φj z j are {λj }pj=1 ,
P
Q
thus we can factorize φ(x) 1 + pj=1 φj z j = pj=1 (1 − λj z). Using this factorization we have
(3.24) can be written as
f (ω) =
1
.
2
j=1 |1 − λj exp(iω)|
(3.25)
Qp
As we have not assumed {Xt } is causal, the roots of φ(z) can lie both inside and outside the
unit circle. We separate the roots, into those outside the unit circle {λO,j1 ; j1 = 1, . . . , p1 }
and inside the unit circle {λI,j2 ; j2 = 1, . . . , p2 } (p1 + p2 = p). Thus
φ(z) = [
p1
Y
(1 − λO,j1 z)][
j1 =1
p2
Y
(1 − λI,j2 z)]
j2 =1
p
1
Y
p2
Y
j1 =1
j2 =1
= (−1)p2 λI,j2 z −p2 [
(1 − λO,j1 z)][
(1 − λ−1
I,j2 z)].
(3.26)
Thus we can rewrite the spectral density in (3.27)
f (ω) =
1
1
.
Qp 1
Qp2
−1
2
2
2
j2 =1 |λI,j2 |
j1 =1 |1 − λO,j exp(iω)|
j2 =1 |1 − λI,j2 exp(iω)|
Qp 2
(3.27)
Let
fO (ω) =
Then f (ω) =
Qp 1
j1 =1 |1
− λO,j
1
Qp 2
exp(iω)|2
j2 =1 |1
2
− λ−1
I,j2 exp(iω)|
.
Qp 2
−2
j2 =1 |λI,j2 | fO (ω).
• A parallel causal AR(p) process with the same covariance structure always exists.
We now define a process which has the same autocovariance function as {Xt } but is causal.
90
Using (3.26) we define the polynomial
e
φ(z)
=[
p1
Y
(1 − λO,j1 z)][
j1 =1
p2
Y
(1 − λ−1
I,j2 z)].
(3.28)
j2 =1
By construction, the roots of this polynomial lie outside the unit circle. We then define the
AR(p) process
e
e t = εt ,
φ(B)
X
(3.29)
et } has a stationary, almost sure unique solution. Morefrom Lemma 2.3.1 we know that {X
over, because the roots lie outside the unit circle the solution is causal.
et } is fe(ω). We know that the spectral density
By using (3.24) the spectral density of {X
et }
function uniquely gives the autocovariance function. Comparing the spectral density of {X
with the spectral density of {Xt } we see that they both are the same up to a multiplicative
constant. Thus they both have the same autocovariance structure up to a multiplicative
constant (which can be made the same, if in the definition (3.29) the innovation process has
Q
variance pj22=1 |λI,j2 |−2 ).
Therefore, for every non-causal process, there exists a causal process with the same autocovariance function.
By using the same arguments above, we can generalize to result to ARMA processes.
Definition 3.3.2 An ARMA process is said to have minimum phase when the roots of φ(z) and
θ(z) both lie outside of the unit circle.
Remark 3.3.1 For Gaussian random processes it is impossible to discriminate between a causal
and non-causal time series, this is because the mean and autocovariance function uniquely identify
the process.
However, if the innovations are non-Gaussian, even though the autocovariance function is ‘blind’
to non-causal processes, by looking for other features in the time series we are able to discriminate
between a causal and non-causal process.
91
3.3.1
The Yule-Walker equations of a non-causal process
Once again let us consider the zero mean AR(p) model
Xt =
p
X
φj Xt−j + εt ,
j=1
and var(εt ) < ∞. Suppose the roots of the corresponding characteristic polynomial lie outside the
unit circle, then {Xt } is strictly stationary where the solution of Xt is only in terms of past and
present values of {εt }. Moreover, it is second order stationary with covariance {c(k)}. We recall
from Section 3.1.2, equation (3.4) that we derived the Yule-Walker equations for causal AR(p)
processes, where
E(Xt Xt−k ) =
p
X
φj E(Xt−j Xt−k ) ⇒ c(k) −
j=1
p
X
φj c(k − j) = 0.
(3.30)
j=1
Let us now consider the case that the roots of the characteristic polynomial lie both outside
and inside the unit circle, thus Xt does not have a causal solution but it is still strictly and second
order stationary (with autocovariance, say {c(k)}). In the previous section we showed that there
P
e
e
et = εt (where φ(B) and φ(B)
exists a causal AR(p) φ(B)
X
= 1 − pj=1 φ˜j z j are the characteristic
polynomials defined in (3.26) and (3.28)). We showed that both have the same autocovariance
structure. Therefore,
c(k) −
p
X
φ˜j c(k − j) = 0
j=1
˜ t }.
This means the Yule-Walker equations for {Xt } would actually give the AR(p) coefficients of {X
Thus if the Yule-Walker equations were used to estimate the AR coefficients of {Xt }, in reality we
˜ t }.
would be estimating the AR coefficients of the corresponding causal {X
3.3.2
Filtering non-causal AR models
Here we discuss the surprising result that filtering a non-causal time series with the corresponding
causal AR parameters leaves a sequence which is uncorrelated but not independent. Let us suppose
92
that
Xt =
p
X
φj Xt−j + εt ,
j=1
where εt are iid, E(εt ) = 0 and var(εt ) < ∞. It is clear that given the input Xt , if we apply the
P
filter Xt − pj=1 φj Xt−j we obtain an iid sequence (which is {εt }).
P
Suppose that we filter {Xt } with the causal coefficients {φej }, the output εet = Xt − pj=1 φej Xt−j
is not an independent sequence. However, it is an uncorrelated sequence. We illustrate this with an
example.
Example 3.3.1 Let us return to the AR(1) example, where Xt = φXt−1 + εt . Let us suppose that
φ > 1, which corresponds to a non-causal time series, then Xt has the solution
Xt = −
∞
X
1
εt+j+1 .
φj
j=1
et =
The causal time series with the same covariance structure as Xt is X
1 e
φ Xt−1
+ ε (which has
backshift representation (1 − 1/(φB))Xt = εt ). Suppose we pass Xt through the causal filter
εet = (1 −
(1 − φ1 B)
1
1
B)Xt = Xt − Xt−1 = −
1 εt
φ
φ
B(1 − φB
)
∞
1
1 X 1
= − εt + (1 − 2 )
εt+j .
φ
φ
φj−1
j=1
Evaluating the covariance of the above (assuming wlog that var(ε) = 1) is
∞
1
1 1
1 X 1
cov(e
εt , εet+r ) = − (1 − 2 ) r + (1 − 2 )2
= 0.
φ
φ φ
φ
φ2j
j=0
Thus we see that {e
εt } is an uncorrelated sequence, but unless it is Gaussian it is clearly not independent. One method to study the higher order dependence of {e
εt }, by considering it’s higher order
cumulant structure etc.
The above above result can be generalised to general AR models, and it is relatively straightforward
to prove using the Cr´
amer representation of a stationary process (see Section ??, Theorem ??).
93
Exercise 3.7
(i) Consider the causal AR(p) process
Xt = 1.5Xt−1 − 0.75Xt−2 + εt .
Derive a parallel process with the same autocovariance structure but that is non-causal (it
should be real).
(ii) Simulate both from the causal process above and the corresponding non-causal process with
non-Gaussian innovations (see Section 2.6). Show that they have the same ACF function.
(iii) Find features which allow you to discriminate between the causal and non-causal process.
94
Chapter 4
Nonlinear Time Series Models
Prerequisites
• A basic understanding of expectations, conditional expectations and how one can use conditioning to obtain an expectation.
Objectives:
• Use relevant results to show that a model has a stationary, solution.
• Derive moments of these processes.
• Understand the differences between linear and nonlinear time series.
So far we have focused on linear time series, that is time series which have the representation
Xt =
∞
X
ψj εt−j ,
j=−∞
where {εt } are iid random variables. Such models are extremely useful, because they are designed
to model the autocovariance structure and are straightforward to use for forecasting. These are
some of the reasons that they are used widely in several applications.
A typical realisation from a linear time series, will be quite regular with no suddent bursts
or jumps. This is due to the linearity of the system. However, if one looks at financial data, for
example, there are sudden bursts in volatility (variation) and extreme values, which calm down
after a while. It is not possible to model such behaviour well with a linear time series. In order to
capture ‘nonlinear behaviour several nonlinear models have been proposed. The models typically
95
consists of products of random variables which make possible the sudden irratic bursts seen in
the data. Over the past 30 years there has been a lot research into nonlinear time series models.
Probably one of the first nonlinear models proposed for time series analysis is the bilinear model,
this model is used extensively in signal processing and engineering. A popular model for modelling
financial data are (G)ARCH-family of models. Other popular models are random autoregressive
coefficient models and threshold models, to name but a few (see, for example, Subba Rao (1977),
Granger and Andersen (1978), Nicholls and Quinn (1982), Engle (1982), Subba Rao and Gabr
(1984), Bollerslev (1986), Terdik (1999), Fan and Yao (2003), Straumann (2005) and Douc et al.
(2014)).
Once a model has been defined, the first difficult task is to show that it actually has a solution
which is almost surely finite (recall these models have dynamics which start at the −∞, so if they
are not well defined they could be ‘infinite’), with a stationary solution. Typically, in the nonlinear
world, we look for causal solutions. I suspect this is because the mathematics behind existence of
non-causal solution makes the problem even more complex.
We state a result that gives sufficient conditions for a stationary, causal solution of a certain
class of models. These models include ARCH/GARCH and Bilinear models. We note that the
theorem guarantees a solution, but does not give conditions for it’s moments. The result is based
on Brandt (1986), but under stronger conditions.
Theorem 4.0.1 (Brandt (1986)) Let us suppose that {X t } is a d-dimensional time series defined by the stochastic recurrence relation
X t = At X t−1 + B t ,
(4.1)
where {At } and {Bt } are iid random matrices and vectors respectively. If E log kAt k < 0 and
E log kB t k < ∞ (where k · k denotes the spectral norm of a matrix), then
X t = Bt +
∞
X
k−1
Y
k=1
i=0
!
At−i
B t−k
(4.2)
converges almost surely and is the unique strictly stationary causal solution.
Note: The conditions given above are very strong and Brandt (1986) states the result under
which weaker conditions, we outline the differences here. Firstly, the assumption {At , Bt } are iid
can be relaxed to their being Ergodic sequences. Secondly, the assumption E log kAt k < 0 can be
96
relaxed to E log kAt k < ∞ and that {At } has a negative Lyapunov exponent, where the Lyapunov
Q
exponent is defined as limn→∞ n1 k nj=1 Aj k = γ, with γ < 0 (see Brandt (1986)).
The conditions given in the above theorem may appear a little cryptic. However, the condition
E log |At | < 0 (in the unvariate case) becomes quite clear if you compare the SRE model with
the AR(1) model Xt = ρXt−1 + εt , where |ρ| < 1 (which is the special case of the SRE, where
P
j
the coefficients is deterministic). We recall that the solution of the AR(1) is Xt = ∞
k=1 ρ εt−j .
The important part in this decomposition is that |ρj | decays geometrically fast to zero. Now let
Q
us compare this to (4.2), we see that ρj plays a similar role to k−1
i=0 At−i . Given that there are
Q
similarities between the AR(1) and SRE, it seems reasonable that for (4.2) to converge, k−1
i=0 At−i
should converge geometrically too (at least almost surely). However analysis of a product is not
straight forward, therefore we take logarithms to turn it into a sum
k−1
k−1
Y
1
1X
a.s.
log
At−i =
log At−i → E[log At ] := γ,
k
k
i=0
i=0
since it is the sum of iid random variables. Thus taking anti-logs
k−1
Y
At−i ≈ exp[kγ],
i=0
which only converges to zero if γ < 0, in other words E[log At ] < 0. Thus we see that the condition
E log |At | < 0 is quite a logical conditional afterall.
4.1
4.1.1
Data Motivation
Yahoo data from 1996-2014
We consider here the closing share price of the Yahoo daily data downloaded from https://uk.
finance.yahoo.com/q/hp?s=YHOO. The data starts from from 10th April 1996 to 8th August 2014
(over 4000 observations). A plot is given in Figure 4.1. Typically the logarithm of such data taken,
and in order to remove linear and/or stochastic trend the first difference of the logarithm is taken
(ie. Xt = log St −log St−1 ). The hope is that after taking differences the data has been stationarized
(see Example 2.3.2). However, the data set spans almost 20 years and this assumption is rather
precarious and will be investigated later. A plot of the data after taking first differences together
97
From the QQplot in Figure 4.2, we observe that log
0
100
200
yahoo
300
400
with the QQplot is given in Figure 4.2.
0
1000
2000
3000
4000
Time
Figure 4.1: Plot of daily closing Yahoo share price 1996-2014
0.4
0.2
−0.4
−0.6
−0.2
Sample Quantiles
0.0
0.4
−0.2
−0.4
−0.6
yahoo.log.diff
0.0
0.2
Normal Q−Q Plot
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
−0.8
−0.8
●
0
1000
3000
●
●
●
−2
Time
0
2
Theoretical Quantiles
Figure 4.2: Plot of log differences of daily Yahoo share price 1996-2014 and the corresponding
QQplot
differences {Xt } appears to have very thick tails, which may mean that higher order moments of
the log returns do not exist (not finite).
In Figure 4.3 we give the autocorrelation (ACF) plots of the log differences, absolute log differences and squares of the log differences. Note that the sample autocorrelation is defined as
ρb(k) =
b
c(k)
,
b
c(0)
where
b
c(k) =
T −|k|
1 X
¯
¯
(Xt − X)(X
t+k − X).
T
(4.3)
t=1
The dotted lines are the errors bars (the 95% confidence of the sample correlations constructed
under the assumption the observations are independent, see Section 6.2.1). From Figure 4.3a
we see that there appears to be no correlation in the data. More precisely, most of the sample
98
correlations are within the errors bars, the few that are outside it could be by chance, as the error
bars are constructed pointwise. However, Figure 4.3b the ACF plot of the absolutes gives significant
large correlations. In contrast, in Figure 4.3c we give the ACF plot of the squares, where there
does not appear to be any significant correlations.
0
5
10
15
20
25
30
35
Lag
0.8
0.6
ACF
0.0
0.2
0.4
ACF
0.0
0.0
0.2
0.2
0.4
0.4
ACF
0.6
0.6
0.8
0.8
1.0
Series (yahoo.log.diff)^2
1.0
Series abs(yahoo.log.diff)
1.0
Series yahoo.log.diff
0
5
10
15
20
25
30
35
Lag
0
5
10
15
20
25
30
35
Lag
(a) ACF plot of the log differ- (b) ACF plot of the absolute (c) ACF plot of the square of
ences
of the log differences
the log differences
Figure 4.3: ACF plots of the transformed Yahoo data
To summarise, {Xt } appears to be uncorrelated (white noise). However, once absolutes have
been taken there does appear to be dependence. This type of behaviour cannot be modelled with
a linear model. What is quite interesting is that there does not appear to be any significant
correlation in the squares. However, on explanation for this can be found in the QQplot. The
data has extremely thick tails which suggest that the forth moment of the process may not exist
(the empirical variance of Xt will be extremely large). Since correlation is defined as (4.3) involves
division by b
c(0), which could be extremely large, this would ‘hide’ the sample covariance.
R code for Yahoo data
Here we give the R code for making the plots above.
yahoo <- scan ("~/ yahoo304 .96.8.14. txt ")
yahoo <- yahoo [ c ( length ( yahoo ):1)] # switches the entries to ascending order 19
yahoo . log . diff <- log ( yahoo [ -1]) - log ( yahoo [ - length ( yahoo )]) # Take
log differences
par ( mfrow = c (1 ,1))
plot . ts ( yahoo )
par ( mfrow = c (1 ,2))
99
plot . ts ( yahoo . log . diff )
qqnorm ( yahoo . log . diff )
qqline ( yahoo . log . diff )
par ( mfrow = c (1 ,3))
acf ( yahoo . log . diff ) # ACF plot of log differences
acf ( abs ( yahoo . log . diff )) # ACF plot of absolute log differences
acf (( yahoo . log . diff )**2) # ACF plot of square of log differences
4.1.2
FTSE 100 from January - August 2014
For completeness we discuss a much shorter data set, the daily closing price of the FTSE 100
from 20th January - 8th August, 2014 (141 observations). This data was downloaded from http:
//markets.ft.com/research//Tearsheets/PriceHistoryPopup?symbol=FTSE:FSI.
Exactly the same analysis that was applied to the Yahoo data is applied to the FTSE data and
6500
6600
ftse
6700
6800
the plots are given in Figure 4.4-4.6.
0
20
40
60
80
100
120
140
Time
Figure 4.4: Plot of daily closing FTSE price Jan-August, 2014
We observe that for this (much shorter) data set, the observations do not appear to deviate
much from normality. Furthermore, the ACF plot of the log differences, absolutes and squares
do not suggest any evidence of correlation. Could it be, that after taking log differences, there is
no dependence in the data (the data is a realisation from iid random variables). Or that there is
dependence but it lies in a ‘higher order structure’ or over more sophisticated transformations.
Comparing this to the Yahoo data, may be we ‘see’ dependence in the Yahoo data because it
is actually nonstationary. The mystery continues. It would be worth while conducting a similar
100
Normal Q−Q Plot
0
20
60
100
140
−0.015 −0.010 −0.005 0.000 0.005 0.010 0.015
Sample Quantiles
−0.015 −0.010 −0.005 0.000 0.005 0.010 0.015
ftse.log.diff
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
−2
Time
−1
0
1
2
Theoretical Quantiles
Figure 4.5: Plot of log differences of daily FTSE price Jan-August, 2014 and the corresponding QQplot
0
5
10
15
1.0
0.8
−0.2
0.0
0.2
0.4
ACF
0.6
0.8
−0.2
0.0
0.2
0.4
ACF
0.6
0.8
0.6
0.4
−0.2
0.0
0.2
ACF
Series (ftse.log.diff)^2
1.0
Series abs(ftse.log.diff)
1.0
Series ftse.log.diff
0
5
Lag
10
15
0
Lag
5
10
15
Lag
(a) ACF plot of the log differ- (b) ACF plot of the absolute (c) ACF plot of the square of
ences
of the log differences
the log differences
Figure 4.6: ACF plots of the transformed FTSE data
analysis on a similar portion of the Yahoo data.
4.2
The ARCH model
During the early 80s Econometricians were trying to find a suitable model for forecasting stock
prices. They were faced with data similar to the log differences of the Yahoo data in Figure 4.2. As
Figure 4.3a demonstrates, there does not appear to be any linear dependence in the data, which
makes the best linear predictor quite useless for forecasting. Instead, they tried to predict the
variance of future prices given the past, var[Xt+1 |Xt , Xt−1 , . . .]. This called for a model that has a
zero autocorrelation function, but models the conditional variance.
To address this need, Engle (1982) proposed the autoregressive conditionally heteroskadastic
101
(ARCH) model (note that Rob Engle, together with Clive Granger, in 2004, received the Noble prize
for Economics for Cointegration). He proposed the ARCH(p) which satisfies the representation
σt2
Xt = σt Zt
= a0 +
p
X
2
aj Xt−j
,
j=1
where Zt are iid random variables where E(Zt ) = 0 and var(Zt ) = 1, a0 > 0 and for 1 ≤ j ≤ p
aj ≥ 0.
Before, worrying about whether a solution of such a model exists, let us consider the reasons
behind why this model was first proposed.
4.2.1
Features of an ARCH
Let us suppose that a causal, stationary solution of the ARCH model exists (Xt is a function of
Zt , Zt−1 , Zt−1 , . . .) and all the necessary moments exist. Then we obtain the following.
(i) The first moment:
E[Xt ] = E[Zt σt ] = E[E(Zt σt |Xt−1 , Xt−2 , . . .)] = E[σt E(Zt |Xt−1 , Xt−2 , . . .)]
|
{z
}
σt function of Xt−1 ,...,Xt−p
= E[σt
E(Zt ) ] = E[0 · σt ] = 0.
| {z }
by causality
Thus the ARCH process has a zero mean.
(ii) The conditional variance:
var(Xt |Xt−1 , Xt−2 , . . . , Xt−p ) = E(Xt2 |Xt−1 , Xt−2 , . . . , Xt−p )
= E(Zt2 σt2 |Xt−1 , Xt−2 , . . . , Xt−p ) = σt2 E[Zt2 ] = σt2 .
Thus the conditional variance is σt2 = a0 +
past).
(iii) The autocovariance function:
102
Pp
2
j=1 aj Xt−j
(a weighted sum of the squared
Without loss of generality assume k > 0
cov[Xt , Xt+k ] = E[Xt Xt+k ] = E[Xt E(Xt+k |Xt+k−1 , . . . , Xt )]
= E[Xt σt+k E(Zt+k |Xt+k−1 , . . . , Xt )] = E[Xt σt+k E(Zt+k )] = E[Xt σt+k · 0] = 0.
The autocorrelation function is zero (it is a white noise process).
P
(iv) We will show in Section 4.2.2 that E[X 2d ] < ∞ iff [ pj=1 aj ]E[Zt2d ]1/d < 1. It is well known
that even for Gaussian innovations E[Zt2d ]1/d grows with d, therefore if any of the aj are
non-zero (recall all need to be positive), there will exist a d0 such that for all d ≥ d0 E[Xtd ]
will not be finite. Thus the we see that the ARCH process has thick tails.
Usually we measure the thickness of tails in data using the Kurtosis measure (see wiki).
Points (i-iv) demonstrate that the ARCH model is able to model many of the features seen in the
stock price data.
In some sense the ARCH model can be considered as a generalisation of the AR model. That
is the squares of ARCH model satisfy
Xt2 = σ 2 Zt2 = a0 +
p
X
2
aj Xt−j
+ (Zt2 − 1)σt2 ,
(4.4)
j=1
with characteristic polynomial φ(z) = 1 −
Pp
j=1 aj z
j.
It can be shown that if
Pp
j=1 aj
< 1, then the
roots of the characteristic polynomial φ(z) lie outside the unit circle (see Exercise 2.1). Moreover,
the ‘innovations’ t = (Zt2 − 1)σt2 are martingale differences (see wiki). This can be shown by noting
that
E[(Zt2 − 1)σt2 |Xt−1 , Xt−2 , . . .] = σt2 E(Zt2 − 1|Xt−1 , Xt−2 , . . .) = σt2 E(Zt2 − 1) = 0.
| {z }
=0
Thus cov(t , s ) = 0 for s 6= t. Martingales are a useful asymptotic tool in time series, we demonstrate how they can be used in Chapter ??.
To summarise, in many respects the ARCH(p) model resembles the AR(p) except that the
innovations {t } are martingale differences and not iid random variables. This means that despite
the resemblence, it is not a linear time series.
103
We show that a unique, stationary causal solution of the ARCH model exists and derive conditions under which the moments exist.
4.2.2
Existence of a strictly stationary solution and second order
stationarity of the ARCH
To simplify notation we will consider the ARCH(1) model
Xt = σt Zt
2
σt2 = a0 + a1 Xt−1
.
(4.5)
It is difficult to directly obtain a solution of Xt , instead we obtain a solution for σt2 (since Xt can
2
2 Z2
immediately be obtained from this). Using that Xt−1
= σt−1
t−1 and substituting this into (4.5)
we obtain
2
2
2
σt2 = a0 + a1 Xt−1
= (a1 Zt−1
)σt−1
+ a0 .
(4.6)
We observe that (4.6) can be written in the stochastic recurrence relation form given in (4.1)
2
2 ] = log a +
with At = a1 Zt−1
and Bt = a0 . Therefore, by using Theorem 4.0.1, if E[log a1 Zt−1
1
2 ] < 0, then σ 2 has the strictly stationary causal solution
E[log Zt−1
t
σt2
= a0 + a0
∞
X
ak1
k=1
k
Y
2
Zt−j
.
j=1
The condition for existence using Theorem 4.0.1 and (4.6) is
E[log(a1 Zt2 )] = log a1 + E[log Zt2 ] < 0,
(4.7)
which is immediately implied if a1 < 1 (since E[log Zt2 ] ≤ log E[Zt2 ] = 0), but it is also satisfied
under weaker conditions on a1 .
To obtain the moments of Xt2 we use that it has the solution is

Xt2 = Zt2 a0 + a0
∞
X
k=1
104
ak1
k
Y
j=1

2 
Zt−j
,
therefore taking expectations we have

E[Xt2 ] = E[Zt2 ]E a0 + a0
∞
X
ak1
k=1
k
Y

2 
Zt−j
= a0 + a0
j=1
∞
X
ak1 .
k=1
Thus E[Xt2 ] < ∞ if and only if a1 < 1 (heuristically we can see this from E[Xt2 ] = E[Z22 ](a0 +
2 ])).
a1 E[Xt−1
By placing stricter conditions on a1 , namely a1 E(Zt2d )1/d < 1, we can show that E[Xt2d ] < ∞.
The ARCH(p) model
We can generalize the above results to ARCH(p) processes (but to show existence of a solution we
need to write the ARCH(p) process as a vector process similar to the Vector AR(1) representation of
an AR(p) given in Section 2.4.1). It can be shown that under sufficient conditions on the coefficients
{aj } that the stationary, causal solution of the ARCH(p) model is
Xt2 = a0 Zt2 +
X
mt (k)
(4.8)
k≥1
where mt (k) =
X
a0
j1 ,...,jk ≥1
k
Y
r=1
ajr
k
Y
2 P
r
Zt−
s=0 js
(j0 = 0).
r=0
The above solution belongs to a general class of functions called a Volterra expansion. We note
P
that E[Xt2 ] < ∞ iff pj=1 aj < 1.
4.3
The GARCH model
A possible drawback of the ARCH(p) model is that the conditional variance only depends on finite
number of the past squared observations/log returns (in finance, the share price is often called
the return). However, when fitting the model to the data, analogous to order selection of an
autoregressive model (using, say, the AIC), often a large order p is selected. This suggests that
the conditional variance should involve a large (infinite?) number of past terms. This observation
motivated the GARCH model (first proposed in Bollerslev (1986) and Taylor (1986)), which in
many respects is analogous to the ARMA. The conditional variance of the GARCH model is a
weighted average of the squared returns, the weights decline with the lag, but never go completely
to zero. The GARCH class of models is a rather parsimonous class of models and is extremely
105
popular in finance. The GARCH(p, q) model is defined as
σt2
Xt = σt Zt
= a0 +
p
X
2
aj Xt−j
+
j=1
q
X
2
bi σt−i
(4.9)
i=1
where Zt are iid random variables where E(Zt ) = 0 and var(Zt ) = 1, a0 > 0 and for 1 ≤ j ≤ p
aj ≥ 0 and 1 ≤ i ≤ q bi ≥ 0.
Under the assumption that a causal solution with sufficient moments exist, the same properties
defined for the ARCH(p) in Section 4.2.1 also apply to the GARCH(p, q) model.
It can be shown that under suitable conditions on {bj } that Xt satisfies an ARCH(∞) representation. Formally, we can write the conditional variance σt2 (assuming that a stationarity solution
exists) as
(1 −
q
X
bi B i )σt2 = (a0 +
i=1
p
X
2
aj Xt−j
),
j=1
where B denotes the backshift notation defined in Chapter 2. Therefore if the roots of b(z) =
P
P
(1 − qi=1 bi z i ) lie outside the unit circle (which is satisfied if i bi < 1) then
σt2
=
(1 −
1
Pq
j
j=1 bj B )
(a0 +
p
X
2
aj Xt−j
)
j=1
= α0 +
∞
X
2
αj Xt−j
,
(4.10)
j=1
where a recursive equation for the derivation of αj can be found in Berkes et al. (2003). In other
words the GARCH(p, q) process can be written as a ARCH(∞) process. This is analogous to the
invertibility representation given in Definition 2.2.2. This representation is useful when estimating
the parameters of a GARCH process (see Berkes et al. (2003)) and also prediction. The expansion
in (4.10) helps explain why the GARCH(p, q) process is so popular. As we stated at the start of this
section, the conditional variance of the GARCH is a weighted average of the squared returns, the
weights decline with the lag, but never go completely to zero, a property that is highly desirable.
Example 4.3.1 (Inverting the GARCH(1, 1)) If b1 < 1, then we can write σt2 as
σt2


∞
∞
X
X
a0
2
2
=
+ a1
bj Xt−1−j
.
bj B j  · a0 + a1 Xt−1
= 
1−b
j=0
j=0
This expression is useful in both prediction and estimation.
106
In the following section we derive conditions for existence of the GARCH model and also it’s
moments.
4.3.1
Existence of a stationary solution of a GARCH(1, 1)
We will focus on the GARCH(1, 1) model as this substantially simplifies the conditions. We recall
the conditional variance of the GARCH(1, 1) can be written as
2
2
2
2
σt2 = a0 + a1 Xt−1
+ b1 σt−1
= a1 Zt−1
+ b1 σt−1
+ a0 .
(4.11)
We observe that (4.11) can be written in the stochastic recurrence relation form given in (4.1) with
2
2
At = (a1 Zt−1
+ b1 ) and Bt = a0 . Therefore, by using Theorem 4.0.1, if E[log(a1 Zt−1
+ b1 )] < 0,
then σt2 has the strictly stationary causal solution
σt2 = a0 + a0
∞ Y
k
X
2
(a1 Zt−j
+ b1 ).
(4.12)
k=1 j=1
These conditions are relatively weak and depend on the distribution of Zt . They are definitely
2
2
satisfied if a1 + b1 < 1, since E[log(a1 Zt−1
+ b1 )] ≤ log E[a1 Zt−1
+ b1 ] = log(a1 + b1 ). However
existence of a stationary solution does not require such a strong condition on the coefficients (and
there can still exist a stationary solution if a1 + b1 > 1, so long as the distribution of Zt2 is such
that E[log(a1 Zt2 + b1 )] < 0).
By taking expectations of (4.12) we can see that
E[Xt2 ]
=
E[σt2 ]
= a0 + a0
∞ Y
k
X
(a1 + b1 ) = a0 + a0
k=1 j=1
∞
X
(a1 + b1 )k .
k=1
Thus E[Xt2 ] < ∞ iff a1 + b1 < 1 (noting that a1 and b1 are both positive). Expanding on this
argument, if d > 1 we can use Minkowski inequality to show
(E[σt2d ])1/d ≤ a0 + a0
∞
X
k=1
(E[
k
Y
2
(a1 Zt−j
+ b1 )]d )1/d ≤ a0 + a0
j=1
∞ Y
k
X
2
(
E[(a1 Zt−j
+ b1 )d ])1/d .
k=1 j=1
2
Therefore, if E[(a1 Zt−j
+ b1 )d ] < 1, then E[Xt2d ] < ∞. This is an iff condition, since from the
107
definition in (4.11) we have
2
2
2
2
2
2d
E[σt2d ] = E[a0 + (a1 Zt−1
+ b1 )σt−1
]d ≥ E[(a1 Zt−1
+ b1 )σt−1
]d = E[(a1 Zt−1
+ b1 )d ]E[σt−1
],
{z
}
|
every term is positive
2
2 . We observe that by stationarity and if
since σt−1
has a causal solution, it is independent of Zt−1
2d ]. Thus the above inequality only holds if E[(a Z 2 + b )d ] < 1.
E[σt2d ] < ∞, then E[σt2d ] = E[σt−1
1 t−1
1
2 + b )d ] < 1.
Therefore, E[Xt2d ] < ∞ iff E[(a1 Zt−1
1
Indeed in order for E[Xt2d ] < ∞ a huge constraint needs to be placed on the parameter space
of a1 and b1 .
Exercise 4.1 Suppose {Zt } are standard normal random variables. Find conditions on a1 and b1
such that E[Xt4 ] < ∞.
The above results can be generalised to GARCH(p, q) model. Conditions for existence of
a stationary solution hinge on the random matrix corresponding to the SRE representation of
the GARCH model (see ?), which are nearly impossible to verify. Sufficient and necessary conditions for both a stationary (causal) solution and second order stationarity (E[Xt2 ] < ∞) is
Pp
Pq
j=1 aj +
i=1 bi < 1. However, many econometricians believe this condition places an unreasonable constraint on the parameter space of {aj } and {bj }. A large amount of research has
been done on finding consistent parameter estimators under weaker conditions. Indeed, in the very
interesting paper by Berkes et al. (2003) (see also Straumann (2005)) they derive consistent estimates of GARCH parameters on far milder set of conditions on {aj } and {bi } (which don’t require
E(Xt2 ) < ∞).
Definition 4.3.1 The IGARCH model is a GARCH model where
Xt = σt Zt
σt2
= a0 +
p
X
2
aj Xt−j
j=1
where the coefficients are such that
Pp
j=1 aj
+
Pq
i=1 bi
+
q
X
2
bi σt−i
(4.13)
i=1
= 1. This is an example of a time series
model which has a strictly stationary solution but it is not second order stationary.
Exercise 4.2 Simulate realisations of ARCH(1) and GARCH(1, 1) models. Simulate with both iid
Gaussian and t-distribution errors ({Zt } where E[Zt2 ] = 1). Remember to ‘burn-in’ each realisation.
In all cases fix a0 > 0. Then
108
(i) Simulate an ARCH(1) with a1 = 0.3 and a1 = 0.9.
(ii) Simulate a GARCH(1, 1) with a1 = 0.1 and b1 = 0.85, and a GARCH(1, 1) with a1 = 0.85
and b1 = 0.1. Compare the two behaviours.
4.3.2
Extensions of the GARCH model
One criticism of the GARCH model is that it is ‘blind’ to negative the sign of the return Xt . In
other words, the conditional variance of Xt only takes into account the magnitude of Xt and does
not depend on increases or a decreases in St (which corresponds to Xt being positive or negative).
In contrast it is largely believed that the financial markets react differently to negative or positive
Xt . The general view is that there is greater volatility/uncertainity/variation in the market when
the price goes down.
This observation has motivated extensions to the GARCH, such as the EGARCH which take
into account the sign of Xt . Deriving conditions for such a stationary solution to exist can be
difficult task, and the reader is refered to Straumann (2005) and more the details.
Other extensions to the GARCH include an Autoregressive type model with GARCH innovations.
4.3.3
R code
install.packages("tseries"), library("tseries") recently there have been a new package
developed library("fGARCH").
4.4
Bilinear models
The Bilinear model was first proposed in Subba Rao (1977) and Granger and Andersen (1978) (see
also Subba Rao (1981)). The general Bilinear (BL(p, q, r, s)) model is defined as
Xt −
p
X
j=1
φj Xt−j = εt +
q
X
θi εt−i +
r X
s
X
bk,k0 Xt−k εt−k0 ,
k=1 k0 =1
i=1
where {εt } are iid random variables with mean zero and variance σ 2 .
109
To motivate the Bilinear model let us consider the simplest version of the model BL(1, 0, 1, 1)
Xt = φ1 Xt−1 + b1,1 Xt−1 εt−1 + εt = [φ1 + b1,1 εt−1 ]Xt−1 + εt .
(4.14)
Comparing (4.16) with the conditional variane of the GARCH(1, 1) in (4.11) we see that they are
very similar, the main differences are that (a) the bilinear model does not constrain the coefficients
to be positive (whereas the conditional variance requires the coefficients to be positive) (b) the
2
2
εt−1 depends on Xt−1 , whereas in the GARCH(1, 1) Zt−1
and σt−1
are independent coefficients and
(c) the innovation in the GARCH(1, 1) model is deterministic, whereas in the innovation in the
Bilinear model is random. (b) and (c) makes the analysis of the Bilinear more complicated than
the GARCH model.
4.4.1
Features of the Bilinear model
In this section we assume a causal, stationary solution of the bilinear model exists, the appropriate
number of moments and that it is invertible in the sense that there exists a function g such that
εt = g(Xt−1 , Xt−2 , . . .).
Under the assumption that the Bilinear process is invertible we can show that
E[Xt |Xt−1 , Xt−2 , . . .] = E[(φ1 + b1,1 εt−1 )Xt−1 |Xt−1 , Xt−2 , . . .] + E[εt |Xt−1 , Xt−2 , . . .]
= (φ1 + b1,1 εt−1 )Xt−1 ,
(4.15)
thus unlike the autoregressive model the conditional expectation of the Xt given the past is a
nonlinear function of the past. It is this nonlinearity that gives rise to the spontaneous peaks that
we see a typical realisation.
To see how the bilinear model was motivated in Figure 4.7 we give a plot of
Xt = φ1 Xt−1 + b1,1 Xt−1 εt−1 + εt ,
(4.16)
where φ1 = 0.5 and b1,1 = 0, 0.35, 0.65 and −0.65. and {εt } are iid standard normal random
variables. We observe that Figure 4.7a is a realisation from an AR(1) process and the subsequent
plots are for different values of b1,1 . Figure 4.7a is quite ‘regular’, whereas the sudden bursts in
activity become more pronounced as b1,1 grows (see Figures 4.7b and 4.7c). In Figure 4.7d we give
110
a plot realisation from a model where b1,1 is negative and we see that the fluctation has changed
2
bilinear(400, 0.5, 0.3)
0
1
0
−1
−3
−2
−2
bilinear(400, 0.5, 0)
2
4
3
6
4
direction.
0
100
200
300
400
0
100
Time
200
300
400
Time
(b) φ1 = 0.5 and b1,1 = 0.35
−5
−10
bilinear(400, 0.5, −0.6)
−20
0
−15
5
bilinear(400, 0.5, 0.6)
0
10
5
(a) φ1 = 0.5 and b1,1 = 0
0
100
200
300
400
0
Time
100
200
300
400
Time
(d) φ1 = 0.5 and b1,1 = −0.65
(c) φ1 = 0.5 and b1,1 = 0.65
Figure 4.7: Realisations from different BL(1, 0, 1, 1) models
Remark 4.4.1 (Markov Bilinear model) Some authors define the BL(1, 0, 1, 1) as
Yt = φ1 Yt−1 + b1,1 Yt−1 εt + εt = [φ1 + b11 εt ]Yt−1 + εt .
The fundamental difference between this model and (4.16) is that the multiplicative innovation
(using εt rather than εt−1 ) does not depend on Yt−1 . This means that E[Yt |Yt−1 , Yt−2 , . . .] = φ1 Yt−1
and the autocovariance function is the same as the autocovariance function of an AR(1) model
with the same AR parameter. Therefore, it is unclear the advantage of using this version of the
111
model if the aim is to forecast, since the forecast of this model is the same as a forecast using the
corresponding AR(1) process Xt = φ1 Xt−1 + εt . Forecasting with this model does not take into
account its nonlinear behaviour.
4.4.2
Solution of the Bilinear model
We observe that (4.16) can be written in the stochastic recurrence relation form given in (4.1) with
At = (φ1 + b11 εt−1 ) and Bt = a0 . Therefore, by using Theorem 4.0.1, if E[log(φ1 + b11 εt−1 )] < 0
and E[εt ] < ∞, then Xt has the strictly stationary, causal solution


∞
k−1
X
Y
 (φ1 + b1,1 εt−j ) · [(φ1 + b1,1 εt−k )εt−k ] + εt .
Xt =
k=1
(4.17)
j=1
To show that it is second order stationary we require that E[Xt2 ] < ∞, which imposes additional
conditions on the parameters. To derive conditions for E[Xt2 ] we use (4.18) and the Minkowski
inequality to give
(E[Xt2 ])1/2 ≤
∞
X
k−1
Y
E 
k=1
=
2 1/2

∞ k−1
X
Y
(φ1 + b1,1 εt−j ) 
1/2
· E [(φ1 + b11 εt−k )εt−k ]2
j=1
1/2 1/2
E [(φ1 + b1,1 εt−j )]2
· E [(φ1 + b1,1 εt−k )εt−k ]2
.
(4.18)
k=1 j=1
Therefore if E[ε4t ] < ∞ and
E [(φ1 + b1,1 εt )]2 = φ2 + b211 var(εt ) < 1,
then E[Xt2 ] < ∞ (note that the above equality is due to E[εt ] = 0).
Exercise 4.3 Simulate the BL(2, 0, 1, 1) model (using the AR(2) parameters φ1 = 1.5 and φ2 =
0.75). Experiment with different parameters to give different types of behaviour.
Exercise 4.4 The random coefficient AR model is a nonlinear time series proposed by Barry Quinn
(see Nicholls and Quinn (1982)). The random coefficient AR(1) model is defined as
Xt = (φ + ηt )Xt−1 + εt
112
where {εt } and {ηt } are iid random variables which are independent of each other.
(i) State sufficient conditions which ensure that {Xt } has a strictly stationary solution.
(ii) State conditions which ensure that {Xt } is second order stationary.
(iii) Simulate from this model for different φ and var[ηt ].
4.4.3
R code
Code to simulate a BL(1, 0, 1, 1) model:
# B i l i n e a r Simulation
# B i l i n e a r ( 1 , 0 , 1 , 1 ) model , we u s e t h e f i r s t n0 o b s e r v a t i o n s a r e burn−i n
# in order to get c l o s e to the s t a t i o n a r y s o l u t i o n .
b i l i n e a r <− f u n c t i o n ( n , phi , b , n0 =400) {
y <− rnorm ( n+n0 )
w <− rnorm ( n + n0 )
f o r ( t i n 2 : ( n+n0 ) ) {
y [ t ] <− p h i ∗ y [ t −1] + b ∗ w [ t −1] ∗ y [ t −1] + w [ t ]
}
r e t u r n ( y [ ( n0 + 1 ) : ( n0+n ) ] )
}
4.5
Nonparametric time series models
Many researchers argue that fitting parametric models can lead to misspecification and argue that
it may be more realistic to fit nonparametric or semi-parametric time series models instead. There
exists several nonstationary and semi-parametric time series (see Fan and Yao (2003) and Douc
et al. (2014) for a comprehensive summary), we give a few examples below. The most general
nonparametric model is
Xt = m(Xt−1 , . . . , Xt−p , εt ),
113
but this is so general it looses all meaning, especially if the need is to predict. A slight restriction
is make the innovation term additive (see Jones (1978))
Xt = m(Xt−1 , . . . , Xt−p ) + εt ,
it is clear that for this model E[Xt |Xt−1 , . . . , Xt−p ] = m(Xt−1 , . . . , Xt−p ). However this model has
the distinct disadvantage that without placing any structure on m(·), for p > 2 nonparametric
estimators of m(·) are lousy (as the suffer from the curse of dimensionality).
Thus such a generalisation renders the model useless. Instead semi-parametric approaches have
been developed. Examples include the functional AR(p) model defined as
Xt =
p
X
φj (Xt−p )Xt−j + εt
j=1
the semi-parametric AR(1) model
Xt = φXt−1 + γ(Xt−1 ) + εt ,
the nonparametric ARCH(p)
Xt = σt Zt
σt2
= a0 +
p
X
2
aj (Xt−j
).
j=1
In the case of all these models it is not easy to establish conditions in which a stationary solution
exists. More often then not, if conditions are established they are similar in spirit to those that
are used in the parametric setting. For some details on the proof see Vogt (2013) (also here), who
considers nonparametric and nonstationary models (note the nonstationarity he considers is when
the covariance structure changes over time, not the unit root type). For example in the case of the
the semi-parametric AR(1) model, a stationary causal solution exists if |φ + γ 0 (0)| < 1.
114
Chapter 5
Prediction
Prerequisites
• The best linear predictor.
• Some idea of what a basis of a vector space is.
Objectives
• Understand that prediction using a long past can be difficult because a large matrix has to
be inverted, thus alternative, recursive method are often used to avoid direct inversion.
• Understand the derivation of the Levinson-Durbin algorithm, and why the coefficient, φt,t ,
corresponds to the partial correlation between X1 and Xt+1 .
• Understand how these predictive schemes can be used write space of sp(Xt , Xt−1 , . . . , X1 ) in
terms of an orthogonal basis sp(Xt − PXt−1 ,Xt−2 ,...,X1 (Xt ), . . . , X1 ).
• Understand how the above leads to the Wold decomposition of a second order stationary
time series.
• To understand how to approximate the prediction for an ARMA time series into a scheme
which explicitly uses the ARMA structure. And this approximation improves geometrically,
when the past is large.
One motivation behind fitting models to a time series is to forecast future unobserved observations - which would not be possible without a model. In this chapter we consider forecasting, based
on the assumption that the model and/or autocovariance structure is known.
115
5.1
Forecasting given the present and infinite past
In this section we will assume that the linear time series {Xt } is both causal and invertible, that is
Xt =
∞
X
aj εt−j =
j=0
∞
X
bi Xt−i + εt ,
(5.1)
i=1
where {εt } are iid random variables (recall Definition 2.2.2). Both these representations play an
important role in prediction. Furthermore, in order to predict Xt+k given Xt , Xt−1 , . . . we will
assume that the infinite past is observed. In later sections we consider the more realistic situation
that only the finite past is observed. We note that since Xt , Xt−1 , Xt−2 , . . . is observed that we can
obtain ετ (for τ ≤ t) by using the invertibility condition
∞
X
ετ = Xτ −
bi Xτ −i .
i=1
Now we consider the prediction of Xt+k given {Xτ ; τ ≤ t}. Using the MA(∞) presentation
(since the time series is causal) of Xt+k we have
Xt+k =
∞
X
k−1
X
+
aj+k εt−j
|
,
aj εt+k−j
j=0
j=0
{z
}
innovations are ‘observed’
|
{z
}
future innovations impossible to predict
P
Pk−1
since E[ k−1
j=0 aj εt+k−j |Xt , Xt−1 , . . .] = E[ j=0 aj εt+k−j ] = 0. Therefore, the best linear predictor
of Xt+k given Xt , Xt−1 , . . ., which we denote as Xt (k) is
Xt (k) =
∞
X
aj+k εt−j =
j=0
∞
X
j=0
aj+k (Xt−j −
∞
X
bi Xt−i−j ).
(5.2)
i=1
Xt (k) is called the k-step ahead predictor and it is straightforward to see that it’s mean squared
error is

2
k−1
k
X
X
E [Xt+k − Xt (k)]2 = E 
aj εt+k−j  = var[εt ]
a2j ,
j=0
(5.3)
j=0
where the last line is due to the uncorrelatedness and zero mean of the innovations.
Often we would like to obtain the k-step ahead predictor for k = 1, . . . , n where n is some
116
time in the future. We now explain how Xt (k) can be evaluated recursively using the invertibility
assumption.
Step 1 Use invertibility in (5.1) to give
Xt (1) =
∞
X
bi Xt+1−i ,
i=1
and E [Xt+1 − Xt (1)]2 = var[εt ]
Step 2 To obtain the 2-step ahead predictor we note that
Xt+2 =
=
∞
X
i=2
∞
X
bi Xt+2−i + b1 Xt+1 + εt+2
bi Xt+2−i + b1 [Xt (1) + εt+1 ] + εt+2 ,
i=2
thus it is clear that
Xt (2) =
∞
X
bi Xt+2−i + b1 Xt (1)
i=2
and E [Xt+2 − Xt (2)]2 = var[εt ] b21 + 1 = var[εt ] a22 + a21 .
Step 3 To obtain the 3-step ahead predictor we note that
Xt+3 =
=
∞
X
i=3
∞
X
bi Xt+2−i + b2 Xt+1 + b1 Xt+2 + εt+3
bi Xt+2−i + b2 (Xt (1) + εt+1 ) + b1 (Xt (2) + b1 εt+1 + εt+2 ) + εt+3 .
i=3
Thus
Xt (3) =
∞
X
bi Xt+2−i + b2 Xt (1) + b1 Xt (2)
i=3
and E [Xt+3 − Xt (3)]2 = var[εt ] (b2 + b21 )2 + b21 + 1 = var[εt ] a23 + a22 + a21 .
117
Step k Using the arguments above it is easily seen that
Xt (k) =
∞
X
bi Xt+k−i +
k−1
X
bi Xt (k − i).
i=1
i=k
Thus the k-step ahead predictor can be recursively estimated.
We note that the predictor given above it based on the assumption that the infinite past is
observed. In practice this is not a realistic assumption. However, in the special case that time
series is an autoregressive process of order p (with AR parameters {φj }pj=1 ) and Xt , . . . , Xt−m is
observed where m ≥ p − 1, then the above scheme can be used for forecasting. More precisely,
Xt (1) =
p
X
φj Xt+1−j
j=1
Xt (k) =
Xt (k) =
p
X
j=k
p
X
φj Xt+k−j +
k−1
X
φj Xt (k − j) for 2 ≤ k ≤ p
j=1
φj Xt (k − j) for k > p.
(5.4)
j=1
However, in the general case more sophisticated algorithms are required when only the finite
past is known.
Example: Forecasting yearly temperatures
We now fit an autoregressive model to the yearly temperatures from 1880-2008 and use this model
to forecast the temperatures from 2009-2013. In Figure 5.1 we give a plot of the temperature time
series together with it’s ACF. It is clear there is some trend in the temperature data, therefore we
have taken second differences, a plot of the second difference and it’s ACF is given in Figure 5.2.
We now use the command ar.yule(res1,order.max=10) (we will discuss in Chapter 7 how this
function estimates the AR parameters) to estimate the the AR parameters. The function ar.yule
uses the AIC to select the order of the AR model. When fitting the second differences from (from
1880-2008 - a data set of length of 127) the AIC chooses the AR(7) model
Xt = −1.1472Xt−1 − 1.1565Xt−2 − 1.0784Xt−3 − 0.7745Xt−4 − 0.6132Xt−5 − 0.3515Xt−6 − 0.1575Xt−7 + εt ,
118
ACF
−0.2
−0.5
0.0
0.2
0.0
0.4
temp
0.6
0.5
0.8
1.0
Series global.mean
1880
1900
1920
1940
1960
1980
2000
0
5
10
Time
15
20
25
30
25
30
Lag
Figure 5.1: Yearly temperature from 1880-2013 and the ACF.
1.0
0.5
ACF
0.2
0.0
0.0
−0.6
−0.5
−0.4
−0.2
second.differences
0.4
0.6
Series diff2
1880
1900
1920
1940
1960
1980
2000
0
Time
5
10
15
20
Lag
Figure 5.2: Second differences of yearly temperature from 1880-2013 and its ACF.
with var[εt ] = σ 2 = 0.02294. An ACF plot after fitting this model and then estimating the residuals
{εt } is given in Figure 5.3. We observe that the ACF of the residuals ‘appears’ to be uncorrelated,
which suggests that the AR(7) model fitted the data well (there is a formal test for this called the
Ljung-Box test which we cover later). By using the sequence of equations
119
0.4
−0.2
0.0
0.2
ACF
0.6
0.8
1.0
Series residuals
0
5
10
15
20
Lag
Figure 5.3: An ACF plot of the estimated residuals {εbt }.
ˆ 127 (1) = −1.1472X127 − 1.1565X126 − 1.0784X125 − 0.7745X124 − 0.6132X123
X
−0.3515X122 − 0.1575X121
ˆ 127 (2) = −1.1472X
ˆ 127 (1) − 1.1565X127 − 1.0784X126 − 0.7745X125 − 0.6132X124
X
−0.3515X123 − 0.1575X122
ˆ 127 (3) = −1.1472X
ˆ 127 (2) − 1.1565X
ˆ 127 (1) − 1.0784X127 − 0.7745X126 − 0.6132X125
X
−0.3515X124 − 0.1575X123
ˆ 127 (4) = −1.1472X
ˆ 127 (3) − 1.1565X
ˆ 127 (2) − 1.0784X
ˆ 127 (1) − 0.7745X127 − 0.6132X126
X
−0.3515X125 − 0.1575X124
ˆ 127 (5) = −1.1472X
ˆ 127 (4) − 1.1565X
ˆ 127 (3) − 1.0784X
ˆ 127 (2) − 0.7745X
ˆ 127 (1) − 0.6132X127
X
−0.3515X126 − 0.1575X125 .
ˆ 127 (1), . . . , X
ˆ 127 (5) as forecasts of X128 , . . . , X132 (we recall are the second differences),
We can use X
which we then use to construct forecasts of the temperatures. A plot of the second difference
forecasts together with the true values are given in Figure 5.4. We note that (5.3) can be used to
120
give the mse error. For example
ˆ 127 (1)]2 = σ 2
E[X128 − X
t
ˆ 127 (1)]2 = (1 + φ21 )σt2
E[X128 − X
If we believe the residuals are Gaussian we can use the mean squared error to construct confidence
intervals for the predictions.
0.3
●
●
= forecast
●
0.2
●
= true value
●
0.1
●
0.0
●
●
●
●
−0.1
second difference
●
−0.2
●
−0.3
●
2000
2002
2004
●
●
2006
2008
2010
2012
year
Figure 5.4: Forecasts of second differences.
A small criticism of our approach is that we have fitted a rather large AR(7) model to time
series of length of 127. It may be more appropriate to fit an ARMA model to this time series.
Exercise 5.1 In this exercise we analyze the Sunspot data found on the course website. In the data
analysis below only use the data from 1700 - 2003 (the remaining data we will use for prediction).
In this section you will need to use the function ar.yw in R.
(i) Fit the following models to the data and study the residuals (using the ACF). Using this
121
decide which model
Xt = µ + A cos(ωt) + B sin(ωt) + εt
|{z}
or
AR
Xt = µ + εt
|{z}
AR
is more appropriate (take into account the number of parameters estimated overall).
(ii) Use these models to forecast the sunspot numbers from 2004-2013.
d i f f 1 = g l o b a l . mean [ c ( 2 : 1 3 4 ) ] − g l o b a l . mean [ c ( 1 : 1 3 3 ) ]
diff2 = diff1 [ c (2:133)] − diff1 [ c (1:132)]
res1 = d i f f 2 [ c (1:127)]
r e s i d u a l s a r 7 <− a r . yw( r e s 1 , o r d e r . max=10) $ r e s i d
r e s i d u a l s <− r e s i d u a l s a r 7 [− c ( 1 : 7 ) ]
# F o r e c a s t u s i n g t h e above model
r e s = c ( res1 , rep ( 0 , 5 ) )
r e s [ 1 2 8 ] = −1.1472∗ r e s [ 1 2 7 ]
−0.6132∗ r e s [ 1 2 3 ]
−0.3515∗ r e s [ 1 2 2 ]
r e s [ 1 2 9 ] = −1.1472∗ r e s [ 1 2 8 ]
−0.6132∗ r e s [ 1 2 4 ]
r e s [ 1 3 1 ] = −1.1472∗ r e s [ 1 3 0 ]
−0.6132∗ r e s [ 1 2 6 ]
r e s [ 1 3 2 ] = −1.1472∗ r e s [ 1 3 1 ]
−0.6132∗ r e s [ 1 2 7 ]
−0.7745∗ r e s [ 1 2 5 ]
−1.0784∗ r e s [ 1 2 7 ]
−0.7745∗ r e s [ 1 2 6 ]
−1.0784∗ r e s [ 1 2 8 ]
−0.7745∗ r e s [ 1 2 7 ]
−0.1575∗ r e s [ 1 2 4 ]
−1.1565∗ r e s [ 1 3 0 ]
−0.3515∗ r e s [ 1 2 6 ]
−1.0784∗ r e s [ 1 2 6 ]
−0.1575∗ r e s [ 1 2 3 ]
−1.1565∗ r e s [ 1 2 9 ]
−0.3515∗ r e s [ 1 2 5 ]
−0.7745∗ r e s [ 1 2 4 ]
−0.1575∗ r e s [ 1 2 2 ]
−1.1565∗ r e s [ 1 2 8 ]
−0.3515∗ r e s [ 1 2 4 ]
−1.0784∗ r e s [ 1 2 5 ]
−0.1575∗ r e s [ 1 2 1 ]
−1.1565∗ r e s [ 1 2 7 ]
−0.3515∗ r e s [ 1 2 3 ]
r e s [ 1 3 0 ] = −1.1472∗ r e s [ 1 2 9 ]
−0.6132∗ r e s [ 1 2 5 ]
−1.1565∗ r e s [ 1 2 6 ]
−1.0784∗ r e s [ 1 2 9 ]
−0.1575∗ r e s [ 1 2 5 ]
122
−0.7745∗ r e s [ 1 2 8 ]
5.2
Review of vector spaces
In next few sections we will consider prediction/forecasting for stationary time series. In particular
to find the best linear predictor of Xt+1 given the finite past Xt , . . . , X1 . Setting up notation our
aim is to find
Xt+1|t = PX1 ,...,Xt (Xt+1 ) = Xt+1|t,...,1 =
t
X
φt,j Xt+1−j ,
j=1
where {φt,j } are chosen to minimise the mean squared error minφ E(Xt+1 −
t
Pt
2
j=1 φt,j Xt+1−j ) .
Basic results from multiple regression show that


φt,1


 .. 
 .  = Σ−1
t rt ,


φt,t
where (Σt )i,j = E(Xi Xj ) and (rt )i = E(Xt−i Xt+1 ). Given the covariances this can easily be done.
However, if t is large a brute force method would require O(t3 ) computing operations to calculate
(5.7). Our aim is to exploit stationarity to reduce the number of operations. To do this, we will
briefly discuss the notion of projections on a space, which help in our derivation of computationally
more efficient methods.
Before we continue we first discuss briefly the idea of a a vector space, inner product spaces,
Hilbert spaces, spans and basis. A more complete review is given in Brockwell and Davis (1998),
Chapter 2.
First a brief definition of a vector space. X is called an vector space if for every x, y ∈ X and
a, b ∈ R (this can be generalised to C), then ax + by ∈ X . An inner product space is a vector
space which comes with an inner product, in other words for every element x, y ∈ X we can defined
an innerproduct hx, yi, where h·, ·i satisfies all the conditions of an inner product. Thus for every
element x ∈ X we can define its norm as kxk = hx, xi. If the inner product space is complete
(meaning the limit of every sequence in the space is also in the space) then the innerproduct space
is a Hilbert space (see wiki).
(i) The classical example of a Hilbert space is the Euclidean space Rn where
P
the innerproduct between two elements is simply the scalar product, hx, yi = ni=1 xi yi .
Example 5.2.1
123
(ii) The subset of the probability space (Ω, F, P ), where all the random variables defined on Ω
R
have a finite second moment, ie. E(X 2 ) = Ω X(ω)2 dP (ω) < ∞. This space is denoted as
L2 (Ω, F, P ). In this case, the inner product is hX, Y i = E(XY ).
(iii) The function space L2 [R, µ], where f ∈ L2 [R, µ] if f is mu-measureable and
Z
|f (x)|2 dµ(x) < ∞,
R
is a Hilbert space. For this space, the inner product is defined as
Z
hf, gi =
f (x)g(x)dµ(x).
R
In this chapter we will not use this function space, but it will be used in Chapter ?? (when
we prove the Spectral representation theorem).
It is straightforward to generalize the above to complex random variables and functions defined
on C. We simply need to remember to take conjugates when defining the innerproduct, ie. hX, Y i =
R
E(XY ) and hf, gi = C f (z)g(z)dµ(z).
In this chapter our focus will be on certain spaces of random variables which have a finite variance.
Basis
The random variables {Xt , Xt−1 , . . . , X1 } span the space Xt1 (denoted as sp(Xt , Xt−1 , . . . , X1 )), if
for every Y ∈ Xt1 , there exists coefficients {aj ∈ R} such that
Y =
t
X
aj Xt+1−j .
(5.5)
j=1
Moreover, sp(Xt , Xt−1 , . . . , X1 ) = Xt1 if for every {aj ∈ R},
Pt
j=1 aj Xt+1−j
∈ Xt1 . We now
define the basis of a vector space, which is closely related to the span. The random variables
{Xt , . . . , X1 } form a basis of the space Xt1 , if for every Y ∈ Xt1 we have a representation (5.5) and
this representation is unique. More precisely, there does not exist another set of coefficients {bj }
P
such that Y = tj=1 bj Xt+1−j . For this reason, one can consider a basis as the minimal span, that
is the smallest set of elements which can span a space.
Definition 5.2.1 (Projections) The projection of the random variable Y onto the space spanned
124
by sp(Xt , Xt−1 , . . . , X1 ) (often denoted as PXt ,Xt−1 ,...,X1 (Y)) is defined as PXt ,Xt−1 ,...,X1 (Y) =
Pt
j=1 cj Xt+1−j ,
where {cj } is chosen such that the difference Y −P( Xt ,Xt−1 ,...,X1 ) (Yt ) is uncorrelated (orthogonal/perpendicular) to any element in sp(Xt , Xt−1 , . . . , X1 ). In other words, PXt ,Xt−1 ,...,X1 (Yt ) is the best
linear predictor of Y given Xt , . . . , X1 .
Orthogonal basis
An orthogonal basis is a basis, where every element in the basis is orthogonal to every other element
in the basis. It is straightforward to orthogonalize any given basis using the method of projections.
To simplify notation let Xt|t−1 = PXt−1 ,...,X1 (Xt ). By definition, Xt − Xt|t−1 is orthogonal to
the space sp(Xt−1 , Xt−1 , . . . , X1 ). In other words Xt − Xt|t−1 and Xs (1 ≤ s ≤ t) are orthogonal
(cov(Xs , (Xt − Xt|t−1 )), and by a similar argument Xt − Xt|t−1 and Xs − Xs|s−1 are orthogonal.
Thus by using projections we have created an orthogonal basis X1 , (X2 −X2|1 ), . . . , (Xt −Xt|t−1 )
of the space sp(X1 , (X2 − X2|1 ), . . . , (Xt − Xt|t−1 )). By construction it clear that sp(X1 , (X2 −
X2|1 ), . . . , (Xt − Xt|t−1 )) is a subspace of sp(Xt , . . . , X1 ). We now show that
sp(X1 , (X2 − X2|1 ), . . . , (Xt − Xt|t−1 )) = sp(Xt , . . . , X1 ).
To do this we define the sum of spaces. If U and V are two orthogonal vector spaces (which
share the same innerproduct), then y ∈ U ⊕ V , if there exists a u ∈ U and v ∈ V such that
1 .
y = u + v. By the definition of Xt1 , it is clear that (Xt − Xt|t−1 ) ∈ Xt1 , but (Xt − Xt|t−1 ) ∈
/ Xt−1
1 . Continuing this argument we see that X 1 = sp(X
Hence Xt1 = sp(X
¯ t − Xt|t−1 ) ⊕ Xt−1
¯ t − Xt|t−1 ) ⊕
t
sp(X
¯ t−1 − Xt−1|t−2 )⊕, . . . , ⊕sp(X
¯ 1 ). Hence sp(X
¯ t , . . . , X1 ) = sp(X
¯ t − Xt|t−1 , . . . , X2 − X2|1 , X1 ).
Pt
Therefore for every PXt ,...,X1 (Y ) = j=1 aj Xt+1−j , there exists coefficients {bj } such that
PXt ,...,X1 (Y ) = PXt −Xt|t−1 ,...,X2 −X2|1 ,X1 (Y ) =
t
X
PXt+1−j −Xt+1−j|t−j (Y ) =
j=1
t−1
X
bj (Xt+1−j − Xt+1−j|t−j ) + bt X1 ,
j=1
where bj = E(Y (Xj − Xj|j−1 ))/E(Xj − Xj|j−1 ))2 . A useful application of orthogonal basis is the
ease of obtaining the coefficients bj , which avoids the inversion of a matrix. This is the underlying
idea behind the innovations algorithm proposed in Brockwell and Davis (1998), Chapter 5.
5.2.1
Spaces spanned by infinite number of elements
The notions above can be generalised to spaces which have an infinite number of elements in their
basis (and are useful to prove Wold’s decomposition theorem). Let now construct the space spanned
125
by infinite number random variables {Xt , Xt−1 , . . .}. As with anything that involves ∞ we need to
define precisely what we mean by an infinite basis. To do this we construct a sequence of subspaces,
each defined with a finite number of elements in the basis. We increase the number of elements in
the subspace and consider the limit of this space. Let Xt−n = sp(Xt , . . . , X−n ), clearly if m > n,
−n
−∞
then Xt−n ⊂ Xt−m . We define Xt−∞ , as Xt−∞ = ∪∞
, then there
n=1 Xt , in other words if Y ∈ Xt
exists an n such that Y ∈ Xt−n . However, we also need to ensure that the limits of all the sequences
lie in this infinite dimensional space, therefore we close the space by defining defining a new space
which includes the old space and also includes all the limits. To make this precise suppose the
sequence of random variables is such that Ys ∈ Xt−s , and E(Ys1 − Ys2 )2 → 0 as s1 , s2 → ∞. Since
the sequence {Ys } is a Cauchy sequence there exists a limit. More precisely, there exists a random
−n
variable Y , such that E(Ys − Y )2 → 0 as s → ∞. Since the closure of the space, X t , contains the
set Xt−n and all the limits of the Cauchy sequences in this set, then Y ∈ Xt−∞ . We let
Xt−∞ sp(Xt , Xt−1 , . . .),
(5.6)
The orthogonal basis of sp(Xt , Xt−1 , . . .)
An orthogonal basis of sp(Xt , Xt−1 , . . .) can be constructed using the same method used to orthogonalize sp(Xt , Xt−1 , . . . , X1 ). The main difference is how to deal with the initial value, which in the
case of sp(Xt , Xt−1 , . . . , X1 ) is X1 . The analogous version of the initial value in infinite dimension
space sp(Xt , Xt−1 , . . .) is X−∞ , but this it not a well defined quantity (again we have to be careful
with these pesky infinities).
Let Xt−1 (1) denote the best linear predictor of Xt given Xt−1 , Xt−2 , . . .. As in Section 5.2 it is
−∞
,
clear that (Xt −Xt−1 (1)) and Xs for s ≤ t−1 are uncorrelated and Xt−∞ = sp(Xt −Xt−1 (1))⊕Xt−1
where Xt−∞ = sp(Xt , Xt−1 , . . .). Thus we can construct the orthogonal basis (Xt −Xt−1 (1)), (Xt−1 −
Xt−2 (1)), . . . and the corresponding space sp((Xt − Xt−1 (1)), (Xt−1 − Xt−2 (1)), . . .). It is clear that
sp((Xt −Xt−1 (1)), (Xt−1 −Xt−2 (1)), . . .) ⊂ sp(Xt , Xt−1 , . . .). However, unlike the finite dimensional
case it is not clear that they are equal, roughly speaking this is because sp((Xt − Xt−1 (1)), (Xt−1 −
Xt−2 (1)), . . .) lacks the inital value X−∞ . Of course the time −∞ in the past is not really a well
defined quantity. Instead, the way we overcome this issue is that we define the initial starting
−∞
random variable as the intersection of the subspaces, more precisely let X−∞ = ∩∞
.
n=−∞ Xt
Furthermore, we note that since Xn − Xn−1 (1) and Xs (for any s ≤ n) are orthogonal, then
sp((Xt − Xt−1 (1)), (Xt−1 − Xt−2 (1)), . . .) and X−∞ are orthogonal spaces. Using X−∞ , we have
126
⊕tj=0 sp((Xt−j − Xt−j−1 (1)) ⊕ X−∞ = sp(Xt , Xt−1 , . . .).
We will use this result when we prove the Wold decomposition theorem (in Section 5.7).
5.3
Levinson-Durbin algorithm
We recall that in prediction the aim is to predict Xt+1 given Xt , Xt−1 , . . . , X1 . The best linear
predictor is
Xt+1|t = PX1 ,...,Xt (Xt+1 ) = Xt+1|t,...,1 =
t
X
φt,j Xt+1−j ,
(5.7)
j=1
where {φt,j } are chosen to minimise the mean squared error, and are the solution of the equation


φt,1


 .. 
 .  = Σ−1
t rt ,


φt,t
(5.8)
where (Σt )i,j = E(Xi Xj ) and (rt )i = E(Xt−i Xt+1 ). Using standard methods, such as Gauss-Jordan
elimination, to solve this system of equations requires O(t3 ) operations. However, we recall that
{Xt } is a stationary time series, thus Σt is a Toeplitz matrix, by using this information in the 1940s
Norman Levinson proposed an algorithm which reduced the number of operations to O(t2 ). In the
1960s, Jim Durbin adapted the algorithm to time series and improved it.
We first outline the algorithm. We recall that the best linear predictor of Xt+1 given Xt , . . . , X1
is
Xt+1|t =
t
X
φt,j Xt+1−j .
(5.9)
j=1
The mean squared error is r(t + 1) = E[Xt+1 − Xt+1|t ]2 . Given that the second order stationary
covariance structure, the idea of the Levinson-Durbin algorithm is to recursively estimate {φt,j ; j =
1, . . . , t} given {φt−1,j ; j = 1, . . . , t − 1} (which are the coefficients of the best linear predictor of Xt
given Xt−1 , . . . , X1 ). Let us suppose that the autocovariance function c(k) = cov[X0 , Xk ] is known.
The Levinson-Durbin algorithm is calculated using the following recursion.
Step 1 φ1,1 = c(1)/c(0) and r(2) = E[X2 − X2|1 ]2 = E[X2 − φ1,1 X1 ]2 = 2c(0) − 2φ1,1 c(1).
127
Step 2 For j = t
φt,t =
φt,j
c(t) −
= φt−1,j
Pt−1
j=1 φt−1,j c(t
r(t)
− φt,t φt−1,t−j
− j)
1 ≤ j ≤ t − 1,
and r(t + 1) = r(t)(1 − φ2t,t ).
We give two proofs of the above recursion.
Exercise 5.2 Suppose Xt = φXt−1 + εt (where |φ| < 1). Use the Levinson-Durbin algorithm (and
possibly Maple/Matlab), deduce an expression for φt,j for (1 ≤ j ≤ t). (recall that you already have
an analytic expression for φt,t ).
5.3.1
A proof based on projections
Let us suppose {Xt } is a zero mean stationary time series and c(k) = E(Xk X0 ). Let PXt ,...,X2 (X1 )
denote the best linear predictor of X1 given Xt , . . . , X2 and PXt ,...,X2 (Xt+1 ) denote the best linear
predictor of Xt+1 given Xt , . . . , X2 . Stationarity means that the following predictors share the same
coefficients
Xt|t−1 =
t−1
X
φt−1,j Xt−j
PXt ,...,X2 (Xt+1 ) =
j=1
PXt ,...,X2 (X1 ) =
t−1
X
t−1
X
φt−1,j Xt+1−j
(5.10)
j=1
φt−1,j Xj+1 .
j=1
The last line is because stationarity means that flipping a time series round has the same correlation
structure. These three relations are an important component of the proof.
Recall our objective is to derive the coefficients of the best linear predictor of PXt ,...,X1 (Xt+1 )
based on the coefficients of the best linear predictor PXt−1 ,...,X1 (Xt ). To do this we partition the
space sp(Xt , . . . , X2 , X1 ) into two orthogonal spaces sp(Xt , . . . , X2 , X1 ) = sp(Xt , . . . , X2 , X1 ) ⊕
128
sp(X1 − PXt ,...,X2 (X1 )). Therefore by uncorrelatedness we have the partition
Xt+1|t = PXt ,...,X2 (Xt+1 ) + PX1 −PXt ,...,X2 (X1 ) (Xt+1 )
=
t−1
X
φt−1,j Xt+1−j + φtt (X1 − PXt ,...,X2 (X1 ))
|
{z
}
j=1
{z
} by projection onto one variable
|
by (5.10)


=
t−1
X
φt−1,j Xt+1−j
j=1


t−1


X


+ φt,t X1 −
φt−1,j Xj+1  .


j=1


{z
}
|
(5.11)
by (5.10)
We start by evaluating an expression for φt,t (which in turn will give the expression for the other
coefficients). It is straightforward to see that
φt,t =
=
=
E(Xt+1 (X1 − PXt ,...,X2 (X1 )))
E(X1 − PXt ,...,X2 (X1 ))2
E[(Xt+1 − PXt ,...,X2 (Xt+1 ) + PXt ,...,X2 (Xt+1 ))(X1 − PXt ,...,X2 (X1 ))]
E(X1 − PXt ,...,X2 (X1 ))2
E[(Xt+1 − PXt ,...,X2 (Xt+1 ))(X1 − PXt ,...,X2 (X1 ))]
E(X1 − PXt ,...,X2 (X1 ))2
(5.12)
Therefore we see that the numerator of φt,t is the partial covariance between Xt+1 and X1 (see
Section 3.2.2), furthermore the denominator of φt,t is the mean squared prediction error, since by
stationarity
E(X1 − PXt ,...,X2 (X1 ))2 = E(Xt − PXt−1 ,...,X1 (Xt ))2 = r(t)
(5.13)
Returning to (5.12), expanding out the expectation in the numerator and using (5.13) we have
φt,t =
c(0) −
c(0) − E[Xt+1 PXt ,...,X2 (X1 ))]
E(Xt+1 (X1 − PXt ,...,X2 (X1 )))
=
=
r(t)
r(t)
Pt−1
j=1 φt−1,j c(t
− j)
r(t)
,
(5.14)
which immediately gives us the first equation in Step 2 of the Levinson-Durbin algorithm. To
129
obtain the recursion for φt,j we use (5.11) to give
Xt+1|t =
t
X
φt,j Xt+1−j
j=1
=
t−1
X

φt−1,j Xt+1−j + φt,t X1 −
j=1
t−1
X

φt−1,j Xj+1  .
j=1
To obtain the recursion we simply compare coefficients to give
φt,j = φt−1,j − φt,t φt−1,t−j
1 ≤ j ≤ t − 1.
This gives the middle equation in Step 2. To obtain the recursion for the mean squared prediction
error we note that by orthogonality of {Xt , . . . , X2 } and X1 − PXt ,...,X2 (X1 ) we use (5.11) to give
r(t + 1) = E(Xt+1 − Xt+1|t )2 = E[Xt+1 − PXt ,...,X2 (Xt+1 ) − φt,t (X1 − PXt ,...,X2 (X1 )]2
= E[Xt+1 − PX2 ,...,Xt (Xt+1 )]2 + φ2t,t E[X1 − PXt ,...,X2 (X1 )]2
−2φt,t E[(Xt+1 − PXt ,...,X2 (Xt+1 ))(X1 − PXt ,...,X2 (X1 ))]
= r(t) + φ2t,t r(t) − 2φt,t E[Xt+1 (X1 − PXt ,...,X2 (X1 ))]
{z
}
|
=r(t)φt,t by (5.14)
= r(t)[1 −
φ2tt ].
This gives the final part of the equation in Step 2 of the Levinson-Durbin algorithm.
Further references: Brockwell and Davis (1998), Chapter 5 and Fuller (1995), pages 82.
5.3.2
A proof based on symmetric Toeplitz matrices
We now give an alternative proof which is based on properties of the (symmetric) Toeplitz matrix.
We use (5.8), which is a matrix equation where


φt,1


 . 
Σt  ..  = rt ,


φt,t
130
(5.15)
with




Σt = 



c(2) . . . c(t − 1)

c(1) . . . c(t − 2)
..
..
..
.
.
.
..
..
c(t − 1) c(t − 2)
.
.
c(0)







c(0)
c(1)
..
.
c(1)
c(0)
..
.

c(1)





 c(2) 

rt = 
 ..  .
 . 


c(t)
and
The proof is based on embedding rt−1 and Σt−1 into Σt−1 and using that Σt−1 φt−1 = rt−1 .
To do this, we define the (t − 1) × (t − 1) matrix Et−1 which basically swops round all the
elements in a vector

Et−1



=



0 0 0 ... 0 1



0 0 0 ... 1 0 
,
.. .. .. .. ..


. . . . .

..
1 0 . 0 0 0
(recall we came across this swopping matrix in Section 3.2.2). Using the above notation, we have
the interesting block matrix structure

Σt = 
and rt =
Σt−1
Et−1 rt−1
r0t−1 Et−1
(r0t−1 , c(t))0 .


c(0)
Returning to the matrix equations in (5.15) and substituting the above into (5.15) we have

Σt φt = r t ,
⇒

Σt−1
Et−1 rt−1
r0t−1 Et−1
c(0)


φt−1,t
φt,t


=
rt−1
c(t)

,
where φ0t−1,t = (φ1,t , . . . , φt−1,t ). This leads to the two equations
Σt−1 φt−1,t + Et−1 rt−1 φt,t = rt−1
(5.16)
r0t−1 Et−1 φt−1,t + c(0)φt,t = c(t).
(5.17)
We first show that equation (5.16) corresponds to the second equation in the Levinson-Durbin
131
algorithm. Multiplying (5.16) by Σ−1
t−1 , and rearranging the equation we have
rt−1 − Σ−1 Et−1 rt−1 φt,t .
φt−1,t = Σ−1
| t−1
{z } | t−1 {z
}
=φt−1
=Et−1 φt−1
Thus we have
φt−1,t = φt−1 − φt,t Et−1 φt−1 .
(5.18)
This proves the second equation in Step 2 of the Levinson-Durbin algorithm.
We now use (5.17) to obtain an expression for φt,t , which is the first equation in Step 1.
Substituting (5.18) into φt−1,t of (5.17) gives
r0t−1 Et−1 φt−1 − φt,t Et−1 φt−1 + c(0)φt,t = c(t).
(5.19)
Thus solving for φt,t we have
φt,t =
c(t) − c0t−1 Et−1 φt−1
c(0) − c0t−1 φ0t−1
.
(5.20)
Noting that r(t) = c(0) − c0t−1 φ0t−1 . (5.20) is the first equation of Step 2 in the Levinson-Durbin
equation.
Note from this proof it does not appear that we need that the (symmetric) Toeplitz matrix is
positive semi-definite.
5.3.3
Using the Durbin-Levinson to obtain the Cholesky decomposition of the precision matrix
We recall from Section 3.2.1 that by sequentially projecting the elements of random vector on the
past elements in the vector gives rise to Cholesky decomposition of the inverse of the variance/covariance (precision) matrix. This is exactly what was done in when we make the Durbin-Levinson
132
algorithm. In other words,





var 





√X1
r(1)
X1 −φ1,1 X2
√
r(2)




 = In




..
.
P
Xn − n−1
j=1 φn−1,j Xn−j
√
r(n)
0
Therefore, if Σn = var[X n ], where X n = (X1 , . . . , Xn ), then Σ−1
n = Ln Dn Ln , where






Ln = 




1
0
...
−φ1,1
1
0
−φ2,1
..
.
−φ2,2
..
.
1
..
.
... ... 0



... ... 0 


0 ... 0 

. . . . .. 
. . 
.

(5.21)
−φn−1,n−1 −φn−1,n−2 −φn−1,n−3 . . . . . . 1
and Dn = diag(r1−1 , r2−1 , . . . , rn−1 ).
5.4
Forecasting for ARMA processes
Given the autocovariance of any stationary process the Levinson-Durbin algorithm allows us to
systematically obtain one-step predictors of second order stationary time series without directly
inverting a matrix.
In this section we consider forecasting for a special case of stationary processes, the ARMA
process. We will assume throughout this section that the parameters of the model are known.
We showed in Section 5.1 that if {Xt } has an AR(p) representation and t > p, then the best
linear predictor can easily be obtained using (5.4). Therefore, when t > p, there is no real gain
in using the Levinson-Durbin for prediction of AR(p) processes. However, we do show in Chapter
?? we can apply the Levinson-Durbin algorithm for obtaining estimators of the autoregressive
parameters.
Similarly if {Xt } satisfies an ARMA(p, q) representation, then the prediction scheme can be
greatly simplified. Unlike the AR(p) process, which is p-Markovian, PXt ,Xt−1 ,...,X1 (Xt+1 ) does
involve all regressors Xt , . . . , X1 . However, some simplifications can be made in the scheme. To
133
explain how, let us suppose that Xt satisfies the representation
Xt −
p
X
φi Xt−j = εt +
j=1
q
X
θi εt−i ,
i=1
where {εt } are iid zero mean random variables and the roots of φ(z) and θ(z) lie outside the
unit circle. For the analysis below, let Wt = Xt for 1 ≤ t ≤ p and for t > max(p, q) let Wt =
P
P
εt + qi=1 θi εt−i (which is the MA(q) part of the process). Since Xp+1 = pj=1 φj Xt+1−j + Wp+1
and so forth it is clear that sp(X1 , . . . , Xt ) = sp(W1 , . . . , Wt ) (ie, they are linear combinations of
each other). We will show for t > max(p, q) that
Xt+1|t = PXt ,...,X1 (Xt+1 ) =
p
X
φj Xt+1−j +
j=1
q
X
θt,i (Xt+1−i − Xt+1−i|t−i ).
(5.22)
i=1
for some θt,i which can be evaluated from the autocovariance structure. To prove the result we use
the following steps:
PXt ,...,X1 (Xt+1 ) =
p
X
j=1
=
=
=
p
X
φj PXt ,...,X1 (Xt+1−j ) +
{z
}
|
Xt+1−j
φj Xt+1−j +
j=1
i=1
p
X
q
X
φj Xt+1−j +
j=1
i=1
p
X
q
X
φj Xt+1−j +
j=1
=
=
q
X
p
X
q
X
θi PXt ,...,X1 (εt+1−i )
i=1
θi PXt −Xt|t−1 ,...,X2 −X2|1 ,X1 (εt+1−i )
{z
}
|
=PWt −Wt|t−1 ,...,W2 −W2|1 ,W1 (εt+1−i )
θi PWt+1−i −Wt+1−i|t−i ,...,Wt −Wt|t−1 (εt+1−i )
θi
i=1
i−1
X
PWt+1−i+s −Wt+1−i+s|t−i+s (εt+1−i )
{z
}
s=0 |
since εt+1−1 is independent of the past
q
X
j=1
θt,i (Wt+1−i − Wt+1−i|t−i )
|
{z
}
i=1
p
X
q
X
j=1
φj Xt+1−j +
=Xt+1−i −Xt+1−i|t−i
φj Xt+1−j +
θt,i (Xt+1−i − Xt+1−i|t−i ),
(5.23)
i=1
this gives the desired result. Thus given the parameters {θt,i } is straightforward to construct the
predictor Xt+1|t . It can be shown that θt,i → θi as t → ∞ (see Brockwell and Davis (1998)),
Chapter 5.
Remark 5.4.1 In terms of notation we can understand the above result for the MA(q) case. In
134
this case,the above result reduces to
bt+1|t =
X
q
X
bt+1−i|t−i .
θt,i Xt+1−i − X
i=1
We now state a few results which will be useful later.
Lemma 5.4.1 Suppose {Xt } is a stationary time series with spectral density f (ω). Let X t =
(X1 , . . . , Xt ) and Σt = var(X t ).
(i) If the spectral density function is bounded away from zero (there is some γ > 0 such that
inf ω f (ω) > 0), then for all t, λmin (Σt ) ≥ γ (where λmin and λmax denote the smallest and
largest absolute eigenvalues of the matrix).
−1
(ii) Further, λmax (Σ−1
t )≤γ .
(Since for symmetric matrices the spectral norm and the largest eigenvalue are the same, then
−1
kΣ−1
t kspec ≤ γ ).
(iii) Analogously, supω f (ω) ≤ M < ∞, then λmax (Σt ) ≤ M (hence kΣt kspec ≤ M ).
PROOF. See Chapter ??.
Remark 5.4.2 Suppose {Xt } is an ARMA process, where the roots φ(z) and and θ(z) have absolute value greater than 1 + δ1 and less than δ2 , then the spectral density f (ω) is bounded by
(1− δ1 )2p
var(εt ) (1−(
2
1
)2p
1+δ1
≤ f (ω) ≤ var(εt )
1
)2p
(1−( 1+δ
1
(1− δ1 )2p
. Therefore, from Lemma 5.4.1 we have that λmax (Σt )
2
and λmax (Σ−1
t ) is bounded uniformly over t.
The prediction can be simplified if we make a simple approximation (which works well if t is
bt+1|t = Xt and for t > max(p, q) we define the
relatively large). For 1 ≤ t ≤ max(p, q), set X
recursion
bt+1|t =
X
p
X
j=1
φj Xt+1−j +
q
X
bt+1−i|t−i ).
θi (Xt+1−i − X
(5.24)
i=1
This approximation seems reasonable, since in the exact predictor (5.23), θt,i → θi .
In the following proposition we show that the best linear predictor of Xt+1 given X1 , . . . , Xt ,
bt+1|t and the best linear predictor given the infinite past,
Xt+1|t , the approximating predictor X
135
bt+1|t
Xt (1) are asymptotically equivalent. To do this we obtain expressions for Xt (1) and X
∞
X
Xt (1) =
bj Xt+1−j ( since Xt+1 =
j=1
∞
X
bj Xt+1−j + εt+1 ).
j=1
Furthermore, by iterating (5.24) backwards we can show that
t−max(p,q)
bt+1|t =
X
X
max(p,q)
bj Xt+1−j +
j=1
X
gammaj Xj
(5.25)
j=1
where |γj |Cρt , with 1/(1 + δ) < ρ < 1 and the roots of θ(z) are outside (1 + δ). We give a proof in
the remark below.
Remark 5.4.3 We prove (5.25) for the ARMA(1, 2). We first recall that the AR(1) part in the
ARMA(1, 1) model does not play any role since sp(X1 , Xt , . . . , Xt ) = sp(W1 , W2 , . . . , Wt ), where
W1 = X1 and for t ≥ 2 we define the corresponding MA(2) process Wt = θ1 εt−1 + θ2 εt−2 + εt . The
c2|1 = W1 , W
c3|2 = W2 and for t > 3
corresponding approximating predictor is defined as W
ct|t−1 = θ1 [Wt−1 − W
ct−1|t−2 ] + θ2 [Wt−2 − W
ct−2|t−3 ].
W
Using this and rearranging (5.24) gives
bt+1|t − φ1 Xt = θ1 [Xt − X
bt−1|t−2 ],
bt|t−1 ] +θ2 [Xt−1 − X
X
|
|
{z
}
|
{z
}
{z
}
ct|t−1 )
=(Wt −W
ct+1|t
W
ct−1|t−2 )
=(Wt−1 −W
By subtracting the above from Wt+1 we have
ct|t−1 ) − θ2 (Wt−1 − W
ct−1|t−2 ) + Wt+1 .
ct+1|t = −θ1 (Wt − W
Wt+1 − W
(5.26)
ct+1|t as the matrix difference equation
It is straightforward to rewrite Wt+1 − W


|
ct+1|t
Wt+1 − W
ct|t−1
Wt − W
{z
=b
εt+1


 = −
}
θ1
θ2
−1 0
|
{z
=Q

ct|t−1
Wt − W


+
ct−1|t−2
Wt−1 − W
}|
{z
} |

=b
εt
Wt+1
0
{z
W t+1


}
ct+1|t lead to the same difference equation except for some
We now show that εt+1 and Wt+1 − W
136
initial conditions, it is this that will give us the result. To do this we
write εt as function of {Wt } (the irreducible condition). We first note that εt can be written as
the matrix difference equation

εt+1

|
εt
{z


 = −
θ2
−1 0
|
{z
}
=εt+1
θ1
Q

εt



+
εt−1
} | {z }
εt
|
Wt+1
0
{z


W t+1
(5.27)
}
Thus iterating backwards we can write
εt+1 =
∞
∞
X
X
˜bj Wt+1−j ,
(−1)j [Qj ](1,1) Wt+1−j =
j=0
j=0
where ˜bj = (−1)j [Qj ](1,1) (noting that ˜b0 = 1) denotes the (1, 1)th element of the matrix Qj (note
we did something similar in Section 2.4.1). Furthermore the same iteration shows that
εt+1 =
t−3
X
(−1)j [Qj ](1,1) Wt+1−j + (−1)t−2 [Qt−2 ](1,1) ε3
j=0
=
t−3
X
˜bj Wt+1−j + (−1)t−2 [Qt−2 ](1,1) ε3 .
(5.28)
j=0
Therefore, by comparison we see that
εt+1 −
t−3
X
˜bj Wt+1−j = (−1)t−2 [Qt−2 ε ]1 =
3
∞
X
˜bj Wt+1−j .
j=t−2
j=0
We now return to the approximation prediction in (5.26). Comparing (5.27) and (5.27) we see
that they are almost the same difference equations. The only difference is the point at which the
algorithm starts. εt goes all the way back to the start of time. Whereas we have set initial values
c2|1 = W1 , W
c3|2 = W2 , thus b
for W
ε03 = (W3 − W2 , W2 − W1 ).Therefore, by iterating both (5.27) and
(5.27) backwards, focusing on the first element of the vector and using (5.28) we have
ε3 ]1
εt+1 − εbt+1 = (−1)t−2 [Qt−2 ε3 ]1 +(−1)t−2 [Qt−2b
|
{z
}
P
˜
= ∞
j=t−2 bj Wt+1−j
We recall that εt+1 = Wt+1 +
P∞ ˜
ct+1|t . Substituting this into
bt+1 = Wt+1 − W
j=1 bj Wt+1−j and that ε
137
the above gives
ct+1|t −
W
∞
X
˜bj Wt+1−j =
j=1
∞
X
˜bj Wt+1−j + (−1)t−2 [Qt−2b
ε3 ]1 .
j=t−2
Replacing Wt with Xt − φ1 Xt−1 gives (5.25), where the bj can be easily deduced from ˜bj and φ1 .
Proposition 5.4.1 Suppose {Xt } is an ARMA process where the roots of φ(z) and θ(z) have roots
bt+1|t and Xt (1) be defined as in (5.23),
which are greater in absolute value than 1 + δ. Let Xt+1|t , X
(5.24) and (5.2) respectively. Then
for any
1
1+δ
bt+1|t ]2 ≤ Kρt ,
E[Xt+1|t − X
(5.29)
bt+1|t − Xt (1)]2 ≤ Kρt
E[X
(5.30)
E[Xt+1 − Xt+1|t ]2 − σ 2 ≤ Kρt
(5.31)
< ρ < 1 and var(εt ) = σ 2 .
PROOF. The proof of (5.29) becomes clear when we use the expansion Xt+1 =
P∞
j=1 bj Xt+1−j
+
εt+1 , noting that by Lemma 2.5.1(iii), |bj | ≤ Cρj .
Evaluating the best linear predictor of Xt+1 given Xt , . . . , X1 , using the autoregressive expansion
gives
Xt+1|t =
∞
X
j=1
bj PXt ,...,X1 (Xt+1−j ) + PXt ,...,X1 (εt+1 )
|
{z
}
=0
t−max(p,q)
X
=
bj Xt+1−j +
j=1
|
∞
X
bj PXt ,...,X1 (Xt−j+1 ).
j=t−max(p,q)
{z
}
bt+1|t −Pmax(p,q) γj Xj
X
j=1
bt+1|t is
Therefore by using (5.25) we see that the difference between the best linear predictor and X
bt+1|t =
Xt+1|t − X
∞
X
max(p,q)
bt+j PXt ,...,X1 (X−j+1 ) +
X
j=1
j=− max(p,q)
138
γj Xj = I + II.
By using (5.25), it is straightforward to show that the second term E[II 2 ] = E[
Pmax(p,q)
j=1
γj Xt−j ]2 ≤
Cρt , therefore what remains is to show that E[II 2 ] attains a similar bound. Heuristically, this seems
reasonable, since bt+j ≤ Kρt+j , the main obstacle is to show that E[PXt ,...,X1 (X−j+1 )2 ] and does
not grow with t. To obtain a bound, we first obtain a bound for E[PXt ,...,X1 (X−j+1 )]2 . Basic results
in linear regression shows that
PXt ,...,X1 (X−j+1 ) = β 0j,t X t ,
(5.32)
0
0
0
where β j,t = Σ−1
t r t,j , with β j,t = (β1,j,t , . . . , βt,j,t ), X t = (X1 , . . . , Xt ), Σt = E(X t X t ) and r t,j =
E(X t Xj ). Substituting (5.32) into I gives
∞
X
∞
X
bt+j PXt ,...,X1 (X−j+1 ) =
j=− max(p,q)
∞
X
bt+j β 0j,t X t =
j=− max(p,q)
bj r 0t,j Σ−1
t X t . (5.33)
j=t−max(p,q)
Therefore the mean squared error of I is

E[I 2 ] = 
∞
X


∞
X

bt+j r 0t,j  Σ−1
t
j=− max(p,q)

bt+j r t,j  .
j=− max(p,q)
To bound the above we use the Cauchy schwarz inequality (kaBbk1 ≤ kak2 kBbk2 ), the specP
tral norm inequality (kak2 kBbk2 ≤ kak2 kBkspec kbk2 ) and Minkowiski’s inequality (k nj=1 aj k2 ≤
Pn
j=1 kaj k2 ) we have
∞
∞
X
2
X
2
2
2
E I2 ≤ bt+j r 0t,j 2 kΣ−1
k
≤
|bt+j | · kr t,j k2 kΣ−1
spec
t
t kspec .
j=1
(5.34)
j=1
We now bound each of the terms above. We note that for all t, using Remark 5.4.2 that kΣ−1
t kspec ≤
K (for some constant K). We now consider r 0t,j = (E(X1 X−j ), . . . , E(Xt X−j )) = (c(1−j), . . . , c(t−
j)). By using (3.2) we have |c(k)| ≤ Cρk , therefore
kr t,j k2 ≤ K(
t
X
ρ2(j+r) )1/2 ≤ K
r=1
ρj
.
(1 − ρ2 )2
Substituting these bounds into (5.34) gives E I 2 ≤ Kρt . Altogether the bounds for I and II give
bt+1|t )2 ≤ K
E(Xt+1|t − X
139
ρj
.
(1 − ρ2 )2
Thus proving (5.29).
To prove (5.30) we note that

∞
X
bt+1|t ]2 = E 
E[Xt (1) − X
bt+j X−j +
j=0
t
X
2
bj Yt−j  .
j=t−max(p,q)
Using the above and that bt+j ≤ Kρt+j , it is straightforward to prove the result.
Finally to prove (5.31), we note that by Minkowski’s inequality we have
h
2 i1/2
E Xt+1 − Xt+1|t
≤
h
i1/2 2 1/2 2 1/2
2
b
b
+ E Xt (1) − Xt+1|t
+ E Xt+1|t − Xt+1|t
.
E (Xt − Xt (1))
{z
} |
|
{z
} |
{z
}
=σ
≤Kρt/2 by (5.30)
≤Kρt/2 by (5.29)
Thus giving the desired result.
5.5
Forecasting for nonlinear models
In this section we consider forecasting for nonlinear models. The forecasts we construct, may not
necessarily/formally be the best linear predictor, because the best linear predictor is based on
minimising the mean squared error, which we recall from Chapter 4 requires the existence of the
higher order moments. Instead our forecast will be the conditional expection of Xt+1 given the past
(note that we can think of it as the best linear predictor). Furthermore, with the exception of the
ARCH model we will derive approximation of the conditional expectation/best linear predictor,
bt+1|t (given in (5.24)).
analogous to the forecasting approximation for the ARMA model, X
5.5.1
Forecasting volatility using an ARCH(p) model
We recall the ARCH(p) model defined in Section 4.2
Xt = σt Zt
σt2 = a0 +
p
X
j=1
140
2
aj Xt−j
.
Using a similar calculation to those given in Section 4.2.1, we see that
E[Xt+1 |Xt , Xt−1 , . . . , Xt−p+1 ] = E(Zt+1 σt+1 |Xt , Xt−1 , . . . , Xt−p+1 ) = σt+1 E(Zt+1 |Xt , Xt−1 , . . . , Xt−p+1 )
{z
}
|
σt+1 function of Xt ,...,Xt−p+1
= σt+1 E(Zt+1 ) = 0 · σt+1 = 0.
| {z }
by causality
In other words, past values of Xt have no influence on the expected value of Xt+1 . On the other
hand, in Section 4.2.1 we showed that
2
2
2
2
2
2
E(Xt+1
|Xt , Xt−1 , . . . , Xt−p+1 ) = E(Zt+1
σt+1
|Xt , Xt−2 , . . . , Xt−p+1 ) = σt+1
E[Zt+1
] = σt+1
=
p
X
2
aj Xt+1−j
,
j=1
thus Xt has an influence on the conditional mean squared/variance. Therefore, if we let Xt+k|t
denote the conditional variance of Xt+k given Xt , . . . , Xt−p+1 , it can be derived using the following
recursion
2
Xt+1|t
=
p
X
2
aj Xt+1−j
j=1
2
Xt+k|t
=
2
Xt+k|t
=
p
X
j=k
p
X
2
+
aj Xt+k−j
k−1
X
2
aj Xt+k−j|k
2≤k≤p
j=1
2
aj Xt+k−j|t
k > p.
j=1
5.5.2
Forecasting volatility using a GARCH(1, 1) model
We recall the GARCH(1, 1) model defined in Section 4.3
2
2
2
2
+ a0 .
σt2 = a0 + a1 Xt−1
+ b1 σt−1
= a1 Zt−1
+ b1 σt−1
Similar to the ARCH model it is straightforward to show that E[Xt+1 |Xt , Xt−1 , . . .] = 0 (where we
use the notation Xt , Xt−1 , . . . to denote the infinite past or more precisely conditioned on the sigma
algebra Ft = σ(Xt , Xt−1 , . . .)). Therefore, like the ARCH process, our aim is to predict Xt2 .
We recall from Example 4.3.1 that if the GARCH the process is invertible (satisfied if b < 1),
141
then
∞
2
2
2
2
E[Xt+1
|Xt , Xt−1 , . . .] = σt+1
= a0 + a1 Xt−1
+ b1 σt−1
=
X
a0
2
+ a1
bj Xt−j
.
1−b
(5.35)
j=0
Of course, in reality we only observe the finite past Xt , Xt−1 , . . . , X1 . We can approximate
2 |X , X
2 = 0, then for t ≥ 1 let
E[Xt+1
b1|0
t
t−1 , . . . , X1 ] using the following recursion, set σ
2
2
= a0 + a1 Xt2 + b1 σ
bt|t−1
σ
bt+1|t
(noting that this is similar in spirit to the recursive approximate one-step ahead predictor defined
in (5.25)). It is straightforward to show that
t−1
2
σ
bt+1|t
=
X
a0 (1 − bt+1 )
2
+ a1
bj Xt−j
,
1−b
j=0
2 |X , . . . , X ] (if the mean square error existed
taking note that this is not the same as E[Xt+1
t
1
2 |X , . . . , X ] would give a smaller mean square error), but just like the ARMA process it will
E[Xt+1
t
1
2
closely approximate it. Furthermore, from (5.35) it can be seen that σ
bt+1|t
closely approximates
2
σt+1
Exercise 5.3 To answer this question you need R install.package("tseries") then remember
library("garch").
(i) You will find the Nasdaq data from 4th January 2010 - 15th October 2014.
(ii) By taking log differences fit a GARCH(1,1) model to the daily closing data (ignore the adjusted
closing value) from 4th January 2010 - 30th September 2014 (use the function garch(x,
order = c(1, 1)) fit the GARCH(1, 1) model).
(iii) Using the fitted GARCH(1, 1) model, forecast the volatility σt2 from October 1st-15th (not2 . Evaluate
ing that no trading is done during the weekends). Denote these forecasts as σt|0
P11 2
t=1 σt|0
(iv) Compare this to the actual volatility
P11
2
t=1 Xt
142
(where Xt are the log differences).
5.5.3
Forecasting using a BL(1, 0, 1, 1) model
We recall the Bilinear(1, 0, 1, 1) model defined in Section 4.4
Xt = φ1 Xt−1 + b1,1 Xt−1 εt−1 + εt .
Assuming invertibility (give condition) it can be shown that
Xt (1) = E[Xt+1 |Xt , Xt−1 , . . .] = φ1 Xt + b1,1 Xt εt .
However, just as in the ARMA and GARCH case we can obtain an approximation, by setting
b1|0 = 0 and for t ≥ 1 defining the recursion
X
bt+1|t = φ1 Xt + b1,1 Xt Xt − X
bt|t−1 .
X
See ? and ? for further details.
bt+1|t approximate Xt (1)?) We now derive conditions for X
bt+1|t
Remark 5.5.1 (How well does X
to be a close approximation of Xt (1) when t is large. We use a similar technique to that used in
Remark 5.4.3.
We note that Xt+1 − Xt (1) = εt+1 (since a future innovation, εt+1 , cannot be predicted). We
bt+1|t is ‘close’ to εt+1 . Subtracting X
bt+1|t from Xt+1 gives the recursion
will show that Xt+1 − X
bt+1|t = −b1,1 (Xt − X
bt|t−1 )Xt + (bεt Xt + εt+1 ) .
Xt+1 − X
(5.36)
We will compare the above recursion to the recursion based on εt+1 . Rearranging the bilinear
equation gives
εt+1 = −bεt Xt + (Xt+1 − φ1 Xt ) .
{z
}
|
(5.37)
=bεt Xt +εt+1
We observe that (5.36) and (5.37) are almost the same difference equation, the only difference is
b1|0 . This gives the difference between the two equations as
that an initial value is set for X
bt+1|t ] = (−1)t bt X1
εt+1 − [Xt+1 − X
t
Y
j=1
143
b1|0 ]
εj + (−1)t bt [X1 − X
t
Y
j=1
εj .
P
bt+1|t →
→ 0 as t → ∞, then X
Xt (1) as t → ∞. We now show that if
Q
Q
a.s.
E[log |εt | < − log |b|, then bt tj=1 εj → 0. Since bt tj=1 εj is a product, it seems appropriate to
Thus if bt
Qt
j=1 εj
a.s.
take logarithms to transform it into a sum. To ensure that it is positive, we take absolutes and
t-roots
t
log |b
t
Y
t
1/t
εj |
1X
log |εj |
t
j=1
|
{z
}
= log |b| +
j=1
.
average of iid random variables
Therefore by using the law of large numbers we have
log |bt
t
Y
t
εj |1/t = log |b| +
j=1
j=1
Thus we see that |bt
1/t a.s.
→
j=1 εj |
Qt
1X
P
log |εj | → log |b| + E log |ε0 | = γ.
t
exp(γ). In other words, |bt
Qt
j=1 εj |
≈ exp(tγ), which will only
converge to zero if E[log |εt | < − log |b|.
5.6
Nonparametric prediction
In this section we briefly consider how prediction can be achieved in the nonparametric world. Let
us assume that {Xt } is a stationary time series. Our objective is to predict Xt+1 given the past.
However, we don’t want to make any assumptions about the nature of {Xt }. Instead we want to
obtain a predictor of Xt+1 given Xt which minimises the means squared error, E[Xt+1 − g(Xt )]2 . It
is well known that this is conditional expectation E[Xt+1 |Xt ]. (since E[Xt+1 − g(Xt )]2 = E[Xt+1 −
E(Xt+1 |Xt )]2 + E[g(Xt ) − E(Xt+1 |Xt )]2 ). Therefore, one can estimate
E[Xt+1 |Xt = x] = m(x)
nonparametrically. A classical estimator of m(x) is the Nadaraya-Watson estimator
Pn−1
m
b n (x) =
x−Xt
t=1 Xt+1 K( b )
,
Pn−1
x−Xt
t=1 K( b )
where K : R → R is a kernel function (see Fan and Yao (2003), Chapter 5 and 6). Under some
‘regularity conditions’ it can be shown that m
b n (x) is a consistent estimator of m(x) and converges
to m(x) in mean square (with the typical mean squared rate O(b4 + (bn)−1 )). The advantage of
144
going the non-parametric route is that we have not imposed any form of structure on the process
(such as linear/(G)ARCH/Bilinear). Therefore, we do not run the risk of misspecifying the model
A disadvantage is that nonparametric estimators tend to be a lot worse than parametric estimators
(in Chapter ?? we show that parametric estimators have O(n−1/2 ) convergence which is faster than
the nonparametric rate O(b2 + (bn)−1/2 )). Another possible disavantage is that if we wanted to
include more past values in the predictor, ie. m(x1 , . . . , xd ) = E[Xt+1 |Xt = x1 , . . . , Xt−p = xd ] then
the estimator will have an extremely poor rate of convergence (due to the curse of dimensionality).
A possible solution to the problem is to assume some structure on the nonparametric model,
and define a semi-parametric time series model. We state some examples below:
(i) An additive structure of the type
Xt =
p
X
gj (Xt−j ) + εt
j=1
where {εt } are iid random variables.
(ii) A functional autoregressive type structure
Xt =
p
X
gj (Xt−d )Xt−j + εt .
j=1
(iii) The semi-parametric GARCH(1, 1)
Xt = σt Zt ,
2
σt2 = bσt−1
+ m(Xt−1 ).
However, once a structure has been imposed, conditions need to be derived in order that the model
has a stationary solution (just as we did with the fully-parametric models).
See ?, ?, ?, ?, ? etc.
5.7
The Wold Decomposition
Section 5.2.1 nicely leads to the Wold decomposition, which we now state and prove. The Wold
decomposition theorem, states that any stationary process, has something that appears close to
an MA(∞) representation (though it is not). We state the theorem below and use some of the
145
notation introduced in Section 5.2.1.
Theorem 5.7.1 Suppose that {Xt } is a second order stationary time series with a finite variance
(we shall assume that it has mean zero, though this is not necessary). Then Xt can be uniquely
expressed as
Xt =
∞
X
ψj Zj + Vt ,
(5.38)
j=0
where {Zt } are uncorrelated random variables, with var(Zt ) = E(Xt −Xt−1 (1))2 (noting that Xt−1 (1)
−∞ , where X −∞
is the best linear predictor of Xt given Xt−1 , Xt−2 , . . .) and Vt ∈ X−∞ = ∩−∞
n
n=−∞ Xn
is defined in (5.6).
PROOF. First let is consider the one-step ahead prediction of Xt given the infinite past, denoted
P
Xt−1 (1). Since {Xt } is a second order stationary process it is clear that Xt−1 (1) = ∞
j=1 bj Xt−j ,
where the coefficients {bj } do not vary with t. For this reason {Xt−1 (1)} and {Xt − Xt−1 (1)} are
second order stationary random variables. Furthermore, since {Xt − Xt−1 (1)} is uncorrelated with
Xs for any s ≤ t, then {Xs − Xs−1 (1); s ∈ R} are uncorrelated random variables. Define Zs = Xs −
Xs−1 (1), and observe that Zs is the one-step ahead prediction error. We recall from Section 5.2.1
that Xt ∈ sp((Xt − Xt−1 (1)), (Xt−1 − Xt−2 (1)), . . .) ⊕ sp(X
¯ −∞ ) = ⊕∞
¯ −∞ ). Since
j=0 sp(Zt−j ) ⊕ sp(X
∞
the spaces ⊕∞
j=0 sp(Zt−j ) and sp(X−∞ ) are orthogonal, we shall first project Xt onto ⊕j=0 sp(Zt−j ),
due to orthogonality the difference between Xt and its projection will be in sp(X−∞ ). This will
lead to the Wold decomposition.
First we consider the projection of Xt onto the space ⊕∞
j=0 sp(Zt−j ), which is
PZt ,Zt−1 ,... (Xt ) =
∞
X
ψj Zt−j ,
j=0
where due to orthogonality ψj = cov(Xt , (Xt−j − Xt−j−1 (1)))/var(Xt−j − Xt−j−1 (1)). Since Xt ∈
⊕∞
¯ −∞ ), the difference Xt − PZt ,Zt−1 ,... Xt is orthogonal to {Zt } and belongs in
j=0 sp(Zt−j ) ⊕ sp(X
sp(X
¯ −∞ ). Hence we have
Xt =
∞
X
ψj Zt−j + Vt ,
j=0
146
where Vt = Xt −
P∞
j=0 ψj Zt−j
and is uncorrelated to {Zt }. Hence we have shown (5.38). To show
that the representation is unique we note that Zt , Zt−1 , . . . are an orthogonal basis of sp(Zt , Zt−1 , . . .),
which pretty much leads to uniqueness.
Exercise 5.4 Consider the process Xt = A cos(Bt + U ) where A, B and U are random variables
such that A, B and U are independent and U is uniformly distributed on (0, 2π).
(i) Show that Xt is second order stationary (actually it’s stationary) and obtain it’s means and
covariance function.
(ii) Show that the distribution of A and B can be chosen is such a way that {Xt } has the same
covariance function as the MA(1) process Yt = εt + φεt (where |φ| < 1) (quite amazing).
(iii) Suppose A and B have the same distribution found in (ii).
(a) What is the best predictor of Xt+1 given Xt , Xt−1 , . . .?
(b) What is the best linear predictor of Xt+1 given Xt , Xt−1 , . . .?
It is worth noting that variants on the proof can be found in Brockwell and Davis (1998),
Section 5.7 and Fuller (1995), page 94.
Remark 5.7.1 Notice that the representation in (5.38) looks like an MA(∞) process. There is,
however, a significant difference. The random variables {Zt } of an MA(∞) process are iid random
variables and not just uncorrelated.
We recall that we have already come across the Wold decomposition of some time series. In
Section 3.3 we showed that a non-causal linear time series could be represented as a causal ‘linear
time series’ with uncorrelated but dependent innovations. Another example is in Chapter 4, where
we explored ARCH/GARCH process which have an AR and ARMA type representation. Using this
representation we can represent ARCH and GARCH processes as the weighted sum of {(Zt2 − 1)σt2 }
which are uncorrelated random variables.
Remark 5.7.2 (Variation on the Wold decomposition) In many technical proofs involving
time series, we often use a results related to the Wold decomposition. More precisely, we often
decompose the time series in terms of an infinite sum of martingale differences. In particular,
we define the sigma-algebra Ft = σ(Xt , Xt−1 , . . .), and suppose that E(Xt |F−∞ ) = µ. Then by
147
telescoping we can formally write Xt as
Xt − µ =
∞
X
Zt,j
j=0
where Zt,j = E(Xt |Ft−j ) − E(Xt |Ft−j−1 ). It is straightforward to see that Zt,j are martingale
differences, and under certain conditions (mixing, physical dependence, your favourite dependence
P
flavour etc) it can be shown that ∞
j=0 kZt,j kp < ∞ (where k · kp is the pth moment). This means
the above representation holds almost surely. Thus in several proofs we can replace Xt − µ by
P∞
j=0 Zt,j . This decomposition allows us to use martingale theorems to prove results.
148
Chapter 6
Estimation of the mean and
covariance
Prerequisites
• Some idea of what a cumulant is.
Objectives
• To derive the sample autocovariance of a time series, and show that this is a positive definite
sequence.
• To show that the variance of the sample covariance involves fourth order cumulants, which
can be unwielding to estimate in practice. But under linearity the expression for the variance
greatly simplifies.
• To show that under linearity the correlation does not involve the fourth order cumulant. This
is the Bartlett formula.
• To use the above results to construct a test for uncorrelatedness of a time series (the Portmanteau test). And understand how this test may be useful for testing for independence in
various different setting. Also understand situations where the test may fail.
149
6.1
An estimator of the mean
Suppose we observe {Yt }nt=1 , where
Yt = µ + Xt ,
where µ is the finite mean, {Xt } is a zero mean stationary time series with absolutely summable
P
covariances ( k |cov(X0 , Xk )| < ∞). Our aim is to estimate the mean µ. The most obvious
P
estimator is the sample mean, that is Y¯n = n−1 nt=1 Yt as an estimator of µ.
6.1.1
The sampling properties of the sample mean
We recall from Example 1.5.1 that we obtained an expression for the sample mean. We showed
that
n
¯ =
var(X)
2 X n − k
1
var(X0 ) +
c(k).
n
n
n
k=1
Furthermore, if
P
k
|c(k)| < ∞, then in Example 1.5.1 we showed that
∞
¯ =
var(X)
1
2X
1
var(X0 ) +
c(k) + o( ).
n
n
n
k=1
Thus if the time series has sufficient decay in it’s correlation structure a mean squared consistent
estimator of the sample mean can be achieved. However, one drawback is that the dependency
means that one observation will influence the next, and if the influence is positive (seen by a positive
covariance), the resulting estimator may have a (much) larger variance than the iid case.
The above result does not require any more conditions on the process, besides second order
stationarity and summability of it’s covariance. However, to obtain confidence intervals we require
a stronger result, namely a central limit theorem for the sample mean. The above conditions are
P
not enough to give a central limit theorem. To obtain a CLT for sums of the form nt=1 Xt we
need to main ingredients:
(i) The variance needs to be finite.
(ii) The dependence between Xt decreases the further apart in time the observations. However,
this is more than just the correlation, it really means the dependence.
150
The above conditions are satisfied by linear time series, but a similar result for nonlinear time series
(for example the (G)ARCH and Bilinear model described in Chapter 4) also holds.
We now state the asymptotic normality result for linear models.
P
Theorem 6.1.1 Suppose that Xt is a linear time series, of the form Xt = ∞
j=−∞ ψj εt−j , where εt
P∞
P
are iid random variables with mean zero and variance one, j=−∞ |ψj | < ∞ and ∞
j=−∞ ψj 6= 0.
Let Yt = µ + Xt , then we have
√
where σ 2 = var(X0 ) + 2
n Y¯n − µ = N (0, σ 2 )
P∞
k=1 c(k).
PROOF. We will give precise details on how to prove asymptotic normality of several different type
of estimators in time series. However, we give a small flavour here by showing asymptotic normality
of Y¯n in the special case that {Xt }nt=1 satisfy an MA(q) model.
The main idea of the proof is to transform/approximate the average into a quantity that we
know is asymptotic normal. We know if {t }nt=1 are iid random variables with mean µ and variance
one then
√
D
n(¯
n − µ) → N (0, 1).
(6.1)
We aim to use this result to prove the theorem. Returning to Y¯n by a change of variables (s = t − j)
we can show that
n
1X
Yt
n
=
t=1
=
=
q
n
n
1X
1 XX
µ+
Xt = µ +
ψj εt−j
n
n
t=1
t=1 j=0






n−q
q
q
0
n
n−s
X
X
X
X
X
X
1
µ+
εs 
ψj  +
εs 
ψj  +
εs 
ψj 
n
s=1
s=−q+1
s=n−q+1
j=0
j=q−s
j=0






q
n−q
q
0
n
n−s
X
X
X
n − q X  1 X
1 X
1
µ+
ψj
εs +
εs 
ψj  +
εs 
ψj 
n
n−q
n
n
j=0
:= µ +
s=1
s=−q+1
Ψ(n − q)
ε¯n−q + E1 + E2 ,
n
j=q−s
s=n−q+1
j=0
(6.2)
Pq
It is straightforward to show that E|E1 | ≤ Cn−1 and E|E2 | ≤ Cn−1 .
P
Finally we examine Ψ(n−q)
ε¯n−q . We note that if the assumptions are not satisfied and qj=0 ψj =
n
where Ψ =
j=0 ψj .
151
0 (for example the process Xt = εt − εt−1 ), then
n
1
1X
Yt = µ +
n
n
t=1
0
X

εs 
s=−q+1
q
X
j=q−s

ψj  +
1
n
n
X

εs 
s=n−q+1
n−s
X

ψj  .
j=0
This is a degenerate case, since E1 and E2 only consist of a finite number of terms and thus if εt are
non-Gaussian these terms will never be asymptotically normal. Therefore, in this case we simply
P
have that n1 nt=1 Yt = µ + O( n1 ) (this is why in the assumptions it was stated that Ψ 6= 0).
On the other hand, if Ψ 6= 0, then the dominating term in Y¯n is ε¯n−q . From (6.1) it is
p
√
P
P
clear that n − q ε¯n−q → N (0, 1) as n → ∞. However, for finite q, (n − q)/n → 1, therefore
√
P
n¯
εn−q → N (0, 1). Altogether, substituting E|E1 | ≤ Cn−1 and E|E2 | ≤ Cn−1 into (6.2) gives
√
√
1 P
n Y¯n − µ = Ψ n¯
εn−q + Op ( ) → N 0, Ψ2 .
n
With a little work, it can be shown that Ψ2 = σ 2 .
Observe that the proof simply approximated the sum by a sum of iid random variables. In the
case that the process is an MA(∞) or linear time series, a similar method is used. The idea is to
approximate the infinite sum by a finite number of terms, q and use a similar method.
Estimation of the so called long run variance (given in Theorem 6.1.1) can be difficult. There
are various methods that can be used, such as estimating the spectral density function (which we
define in Chapter ??) at zero. An interesting approach advocated by Xiaofeng Shao is to use the
method of so called self-normalization which circumvents the need to estimate the long run mean,
see Shao (2010).
6.2
An estimator of the covariance
Suppose we observe {Yt }nt=1 , to estimate the covariance we can estimate the covariance c(k) =
E(X0 Xk ) from the the observations. A plausible estimator is
cˆn (k) =
n−|k|
1 X
(Yt − Y¯n )(Yt+|k| − Y¯n ),
n
t=1
152
(6.3)
since E[(Yt − Y¯n )(Yt+|k| − Y¯n )] ≈ c(k). Of course if the mean of Yt were zero (Yt = Xt ), then the
covariance estimator is
n−|k|
1 X
Xt Xt+|k| .
cˆn (k) =
n
(6.4)
t=1
1 Pn−|k|
The eagle-eyed amongst you may wonder why we don’t use n−|k|
ˆn (k)
t=1 Xt Xt+|k| , and that c
P
n−|k|
1
is more biased than n−|k|
ˆn (k) has some very nice properties which we
t=1 Xt Xt+|k| . However c
discuss in the lemma below.
Lemma 6.2.1 Suppose we define the empirical covariances
cˆn (k) =



1
n
Pn−k
t=1
Xt Xt−k |k| ≤ n − 1
0
otherwise
then {ˆ
cn (k)} is a positive definite sequence. Therefore, using Lemma 1.6.1 there exists a stationary
time series {Zt } which has the covariance cˆn (k).
PROOF. There are various ways to show that {ˆ
cn (k)} is a positive definite sequence. One method
uses that the spectral density corresponding to this sequence is non-negative, we give this proof in
Section ??.
Here we give an alternative proof. We recall a sequence is positive definite if for any vector
a = (a1 , . . . , ar )0 we have
r
X
bra ≥ 0
ak1 ck2 cˆn (k1 − k2 ) = a0 Σ
k1 ,k2 =1
where




b
Σr = 



noting that cˆn (k) =
1
n
cˆn (0)
cˆn (1)
cˆn (2) . . . cˆn (n − 1)



cˆn (1)
cˆn (0)
cˆn (1) . . . cˆn (n − 2) 
,
..
..
..
..

..

.
.
.
.
.

..
..
cˆn (n − 1) cˆn (n − 2)
.
.
cˆn (0)
Pn−k
t=1
Xt Xt−k . However, cˆn (k) =
1
n
Pn−k
t=1
Xt Xt−k has a very interesting
b r = Xn X0 , where Xn is a
construction, it can be shown that the above convariance matrix is Σ
n
153
n × 2r matrix with




Xn = 



0
0
...
0
X1
X2
0
..
.
0
..
.
...
..
.
X1
..
.
X2
..
.
. . . Xn−1
..
..
.
.
X1 X2 . . . Xn−1 Xn
0
...
...
Xn−1 Xn
Xn
..
.
0
..
.
...
0








Using the above we have
b r a = a0 X0 Xn a = kXak2 ≥ 0.
a0 Σ
n
This this proves that {ˆ
cn (k)} is a positive definite sequence.
Finally, by using Theorem 1.6.1, there exists a stochastic process with {ˆ
cn (k)} as it’s autocovariance function.
6.2.1
Asymptotic properties of the covariance estimator
The main reason we construct an estimator is either for testing or constructing a confidence interval
for the parameter of interest. To do this we need the variance and distribution of the estimator. It
is impossible to derive the finite sample distribution, thus we look at their asymptotic distribution.
Besides showing asymptotic normality, it is important to derive an expression for the variance.
In an ideal world the variance will be simple and will not involve unknown parameters. Usually
in time series this will not be the case, and the variance will involve several (often an infinite)
number of parameters which are not straightforward to estimate. Later in this section we show
that the variance of the sample covariance can be extremely complicated. However, a substantial
simplification can arise if we consider only the sample correlation (not variance) and assume linearity
of the time series. This result is known as Bartlett’s formula (you may have come across Maurice
Bartlett before, besides his fundamental contributions in time series he is well known for proposing
the famous Bartlett correction). This example demonstrates, how the assumption of linearity can
really simplify problems in time series analysis and also how we can circumvent certain problems
in which arise by making slight modifications of the estimator (such as going from covariance to
correlation).
The following theorem gives the asymptotic sampling properties of the covariance estimator
(6.3). One proof of the result can be found in Brockwell and Davis (1998), Chapter 8, Fuller
154
(1995), but it goes back to Bartlett (indeed its called Bartlett’s formula). We prove the result in
Section 6.2.2.
Theorem 6.2.1 Suppose {Xt } is a linear stationary time series where
∞
X
Xt = µ +
ψj εt−j ,
j=−∞
where
P
j
|ψj | < ∞, {εt } are iid random variables with E(ε4t ) < ∞. Suppose we observe {Xt :
t = 1, . . . , n} and use (6.3) as an estimator of the covariance c(k) = cov(X0 , Xk ). Define ρˆn (r) =
cˆn (r)/ˆ
cn (0) as the sample correlation. Then for each h ∈ {1, . . . , n}
√
D
n(ˆ
ρn (h) − ρ(h)) → N (0, Wh )
(6.5)
where ρˆn (h) = (ˆ
ρn (1), . . . , ρˆn (h)), ρ(h) = (ρ(1), . . . , ρ(h)) and
(Wh )ij =
∞
X
{ρ(k + i) + ρ(k − i) − 2ρ(i)ρ(k)}{ρ(k + j) + ρ(k − j) − 2ρ(j)ρ(k)}.
(6.6)
k=−∞
Equation (6.6) is known as Bartlett’s formula.
In Section 6.3 we apply the method for checking for correlation in a time series. We first show
how the expression for the asymptotic variance is obtained.
6.2.2
Proof of Bartlett’s formula
The variance of the sample covariance in the case of strict stationarity
We first derive an expression for cˆn (r) under the assumption that {Xt } is a strictly stationary time
P
P
series with finite fourth order moment, k |c(k)| < ∞ and for all r1 , r2 ∈ Z, k |κ4 (r1 , k, k + r2 )| <
∞ where κ4 (k1 , k2 , k3 ) = cum(X0 , Xk1 , Xk2 , Xk3 ) (note the strictly stationary assumption can be
relaxed to forth order stationary).
A simply expansion shows that
var[ˆ
cn (r)]
=
n−|r|
1 X
cov(Xt Xt+r , Xτ Xτ +r ).
n2
t,τ =1
155
cov(Xt Xt+r , Xτ Xτ +r ) is the covariance of the product of random variables. This belong to the
general class of cumulants of products of random variables. We now use standard results on cumulants, which show that cov[XY, U V ] = cov[X, U ]cov[Y, V ] + cov[X, V ]cov[Y, U ] + cum(X, Y, U, V )
(note this result can be generalized to higher order cumulants, see ?). Using this result we have
var[ˆ
cn (r)]
=
n−|r|
1 X
cov(X
,
X
)cov(X
,
X
)
+
cov(X
,
X
)cov(X
,
X
)
+
cum(X
,
X
,
X
,
X
)
t
τ
t+r
τ
+r
t
τ
+r
t+r
τ
t
t+r
τ
τ
+r
n2
t,τ =1
=
n−|r|
1 X c(t − τ )2 + c(t − τ − r)c(t + r − τ ) + k4 (r, τ − t, τ + r − t)
2
n
t,τ =1
:= I + II + III,
where the above is due to strict stationarity of the time series. We analyse the above term by term.
By changing variables, we have
1
I=
n
n−|r|
X
k=−(n−|r|)
n − |k|
n
c(k)2 .
Pn−|r|
P
For all k, (1−|k|/n)c(k)2 → c(k)2 and | k=−(n−|r|) (1−|k|/n)c(k)2 | ≤ k c(k)2 , thus by dominated
P
P
2
convergence (see Chapter A) nk=−(n−|r|) (1 − |k|/n)c(k)2 → ∞
k=−∞ c(k) . This gives
I=
∞
1 X
1
c(k)2 + o( ).
n
n
k=−∞
Using a similar argument we can show that
II =
∞
1
1 X
c(k)c(k − r) + o( ).
n
n
k=−∞
To derive the limit of III, again we use a change of variables to give
1
III =
n
n−|r|
X
k=−(n−|r|)
n − |k|
n
k4 (r, k, k + r).
Pn−|r|
To bound we note that for all k, (1 − |k|/n)k4 (r, k, k + r) → k4 (r, k, k + r) and | k=−(n−|r|) (1 −
P
P
|k|/n)k4 (r, k, k + r)| ≤ k |k4 (r, k, k + r)|, thus by dominated convergence we have nk=−(n−|r|) (1 −
156
P∞
|k|/n)k4 (r, k, k + r) →
k=−∞ k4 (r, k, k
III =
+ r). This gives
1X
1
κ4 (r, k, k + r) + o( ).
n
n
k
Therefore altogether we have
nvar[ˆ
cn (r)] =
∞
X
c(k)2 +
k=−∞
∞
X
c(k)c(k − r) +
k=−∞
∞
X
κ4 (r, k, k + r) + o(1).
k=−∞
Using similar arguments we obtain
∞
X
ncov[ˆ
cn (r1 ), cˆn (r2 )] =
c(k + r1 − r2 )2 +
k=−∞
∞
X
∞
X
c(k − r1 )c(k + r2 ) +
k=−∞
κ4 (r1 , k, k + r2 ) + o(1)
k=−∞
We observe that the covariance of the covariance estimator contains both covariance and cumulants terms. Thus if we need to estimate them, for example to construct confidence intervals, this
can be extremely difficult. However, we show below that under linearity the above fourth order
cumulant term has a simpler form.
The covariance of the sample covariance under linearity
We recall that
∞
X
c(k + r1 − r2 )2 +
k=−∞
∞
X
c(k − r1 )c(k + r2 ) +
k=−∞
∞
X
κ4 (r1 , k, k + r2 ) + o(1) = T1 + T2 + T3 + o(1).
k=−∞
We now show that under linearity, T3 has a much simpler form. Let us suppose that the time series
is linear
Xt =
∞
X
ψj εt−j
j=−∞
where
P
j
|ψj | < ∞, {εt } are iid, E(εt ) = 0, var(εt ) = 1 and κ4 = cum4 (εt ). Then T3 is
T3 =
∞
X
k=−∞
=
∞
X


X
cum 
j1 =−∞
X
ψj1 ε−j1 ,
X
ψj2 εr1 −j2 ,
j2 =−∞
X
j3 =−∞
ψj3 εk−j3 ,
X
j4 =−∞
ψj1 ψj2 ψj3 ψj4 cum (ε−j1 , εr1 −j2 , εk−j3 , εk+r2 −j1 ) .
k=−∞ j1 ,...,j4 =−∞
157
ψj4 εk+r2 −j1 
Standard results in cumulants (which can be proved using the characteristic function), show that
cum[Y1 , Y2 , . . . , Yn ] = 0, if any of these variables is independent of all the others. Applying this
result to cum (ε−j1 , εr1 −j2 , εk−j3 , εk+r2 −j1 ) reduces T3 to
∞
X
T3 = κ4
∞
X
ψj ψj−r1 ψj−k ψj−r2 −k .
k=−∞ j=−∞
Using a change of variables j1 = j and j2 = j − k we have
∞
X
κ4
j1 =−∞
ψj ψj−r1
∞
X
ψj2 ψj2 −r2 = κ4 c(r1 )c(r2 ),
j2 =−∞
recalling the covariance of a linear process in Lemma 3.1.1.
Altogether this gives
ncov[ˆ
cn (r1 ), cˆn (r2 )] =
∞
X
c(k)c(k + r1 − r2 ) +
k=−∞
∞
X
c(k − r1 )c(k + r2 ) + κ4 c(r1 )c(r2 ) + o(1). (6.7)
k=−∞
Thus in the case of linearity our expression for the variance is simpler, and the only difficult
parameter to estimate of κ4 , which can be done using various methods.
The variance of the sample correlation under linearity
A suprisingly twist in the story is that (6.7) can be reduced further, if we are interested in estimating
the correlation rather than the covariance. We recall the sample correlation is
ρˆn (r) =
cˆn (r)
,
cˆn (0)
which is an estimator of ρ(r) = c(r)/c(0).
Lemma 6.2.2 (Bartlett’s formula) Suppose {Xt } is a linear time series, where
P
Then the variance of the distribution of ρˆn (r) is
∞
X
{ρ(k + r) + ρ(k − r) − 2ρ(r)ρ(k)}{ρ(k + r) + ρ(k − r) − 2ρ(r)ρ(k)}
k=−∞
158
j
|ψ(j)| < ∞.
PROOF. By making a Taylor expansion of cˆn (0)−1 about c(0)−1 we have
ρˆn (r) − ρ(r)
=
=
=
cˆn (r) c(r)
−
cˆn (0) c(0)
[ˆ
cn (r) − c(r)]
cˆn (r)
c¯n (r)
− [ˆ
cn (0) − c(0)]
+ 2 [ˆ
cn (0) − c(0)]2
c(0)
c(0)2
cˆn (0)3
c(r)
cˆn (r)
[ˆ
cn (r) − c(r)]
[ˆ
cn (r) − c(r)]
− [ˆ
cn (0) − c(0)]
+ 2 [ˆ
cn (0) − c(0)]2
− [ˆ
cn (0) − c(0)]
2
3
c(0)
c(0)
c¯n (0)
c(0)2
|
{z
}
O(n−1 )
1
:= An + Op ( ),
n
where c¯n (0) lies between cˆn (0) and c(0). We observe that the last two terms of the above are
of order O(n−1 ) (by (6.7) and that c(0) is bounded away from zero) and the dominating term is
An which is of order O(n−1/2 ) (again by (6.7)). Thus the limiting distribution of ρˆn (r) − ρ(r) is
determined by An and the variance of the limiting distribution is also determined by An . It is
straightforward to show that
nvar[An ] = n
c(r)2
c(r)2
var[ˆ
cn (r)]
−
2ncov[ˆ
c
(r),
c
ˆ
(0)]
+
nvar[ˆ
c
(0)]
.
n
n
n
c(0)2
c(0)3
c(0)4
(6.8)
By using (6.7) we have

nvar 
cˆn (r)
cˆn (0)



 P
P∞
P∞
∞
2
2+
c(k)c(k
−
r)
+
κ
c(r)c(0)
c(k)c(k
−
r)
+
κ
c(r)
2
c(k)
4
4
k=−∞
k=−∞
k=−∞

= 
P∞
P∞
P
∞
2
2
2 k=−∞ c(k)c(k − r) + κ4 c(r)c(0)
k=−∞ c(k) +
k=−∞ c(k)c(k − r) + κ4 c(0)
= +o(1).
Substituting the above into (6.8) gives us
nvar[An ] =
∞
X
c(k)2 +
k=−∞
∞
X
2 2
k=−∞
∞
X
!
c(k)c(k − r) + κ4 c(r)2
k=−∞
!
c(k)c(k − r) + κ4 c(r)c(0)
c(k)2 +
k=−∞
∞
X
∞
X
c(r)2
+
c(0)3
!
c(k)c(k − r) + κ4 c(0)2
k=−∞
159
1
−
c(0)2
c(r)2
+ o(1).
c(0)4
Focusing on the fourth order cumulant terms, we see that these cancel, which gives the result. To prove Theorem 6.2.1, we simply use the Lemma 6.2.2 to obtain an asymptotic expression
for the variance, then we use An to show asymptotic normality of cˆn (r) (under linearity).
Exercise 6.1 Under the assumption that {Xt } are iid random variables show that cˆn (r) is asymptotically normal.
Remark 6.2.1 The above would appear to be a nice trick, but there are two major factors that
lead to the cancellation of the fourth order cumulant term
• Linearity of the time series
• Ratio between cˆn (r) and cˆn (0).
Indeed this is not a chance result, in fact there is a logical reason why this result is true (and is
true for many statistics, which have a similar form - commonly called ratio statistics). It is easiest
explained in the Fourier domain. If the estimator can be written as
1
n
Pn
k=1 φ(ωk )In (ωk )
,
1 Pn
k=1 In (ωk )
n
where In (ω) is the periodogram, and {Xt } is a linear time series, then we will show later that the
asymptotic distribution of the above has a variance which is only in terms of the covariances not
higher order cumulants. We prove this result in Section ??.
6.3
Using Bartlett’s formula for checking for correlation
Bartlett’s formula if commonly used to check by ‘eye; whether a time series is uncorrelated (there
are more sensitive tests, but this one is often used to construct CI in for the sample autocovariances
in several statistical packages). This is an important problem, for many reasons:
• Given a data set, we need to check whether there is dependence, if there is we need to analyse
it in a different way.
160
• Suppose we fit a linear regression, we may need to check whether the residuals are actually
uncorrelated, else the standard errors based on the assumption of uncorrelatedness would be
unreliable.
• We need to check whether a time series model is the appropriate model. To do this we fit
the model, estimate the residuals, if the residuals appear to be uncorrelated it would seem
likely that the model is correct. If they are correlated, then the model is in appropriate.
For example, we may fit an AR(1) to the data, estimate the residuals εt , if there is still
correlation in the residuals, then the AR(1) was not the correct model, since Xt − ρXt−1 still
has information about other residuals.
We now apply Theorem 6.2.1 to the case that the time series are iid random variables. Suppose {Xt }
are iid random variables, then it is clear that it is trivial example of a (not necessarily Gaussian)
linear process. We use (6.3) as an estimator of the autocovariances.
To derive the asymptotic variance of {ˆ
cn (r)}, we recall that if {Xt } are iid then ρ(k) = 0 for
k 6=, substituting this into (6.6) we see that
√
D
n(ˆ
ρn (h) − ρ(h)) → N (0, Wh )
where

 1 i=j
(Wh )ij =
 0 i=
6 j
In other words
√
D
n(ˆ
ρn (h)−ρ(h)) → N (0, Ih ). Hence the sample autocovariances at different lags are
asymptotically uncorrelated and have variance one. This allows us to easily construct confidence
intervals for the autocovariances under the assumption of independence of the. If the vast majority
of the sample autocovariance lie inside the confidence there is not enough evidence to suggest that
the data is a realisation of a iid random variables (often called a white noise process). An example
of the empirical ACF and error bars (confidence interval, constructed under the assumption of
independence) is given in Figure 6.1. We see that the empirical autocorrelations of the realisation
from iid random variables all lie within the CI. In contrast in Figure 6.2 we give a plot of the sample
ACF of an AR(2). We observe that a large number of the sample autocorrelations lie outside the
error bars.
161
ACF
0.0
0.2
0.4
0.6
0.8
1.0
Series iid
0
5
10
15
20
Lag
Figure 6.1: The sample ACF of an iid sample with error bars (sample size n = 200).
−0.4 0.0 0.4 0.8
ACF
Series ar2
0
5
10
15
20
−0.4 0.0 0.4 0.8
acf
Lag
5
10
15
20
lag
Figure 6.2: Top: The sample ACF of the AR(2) process Xt = 1.5Xt−1 + 0.75Xt−2 + εt with
error bars n = 200. Bottom: The true ACF.
162
Of course, by simply checking by eye means that we risk misconstruing a sample coefficient that
lies outside the error bars as meaning that the time series is correlated, whereas this could simply
by a false positive (due to multiple testing). To counter this problem, we construct a test statistic
√
D
for testing uncorrelatedness. Since under the null n(ˆ
ρn (h) − ρ(h)) → N (0, I), one method of
testing is to use the square correlations
Sh = n
h
X
|ˆ
ρn (r)|2 ,
(6.9)
r=1
under the null it will asymptotically have a χ2 -distribution with h degrees of freedom, under the
alternative it will be a non-central (generalised) chi-squared. The non-centrality is what makes us
reject the null if the alternative of correlatedness is true. This is known as the Box-Pierce test (a
test which gives better finite sample results is the Ljung-Box test).
6.4
Long range dependence versus changes in the mean
A process is said to have long range dependence if the autocovariances are not absolutely summable,
P
ie.
k |c(k)| = ∞.
From a practical point of view data is said to exhibit long range dependence if the autocovariances do not decay very fast to zero as the lag increases. Returning to the Yahoo data considered
in Section 4.1.1 we recall that the ACF plot of the absolute log differences, given again in Figure
6.3 appears to exhibit this type of behaviour. However, it has been argued by several authors that
0.0
0.2
0.4
ACF
0.6
0.8
1.0
Series abs(yahoo.log.diff)
0
5
10
15
20
25
30
35
Lag
Figure 6.3: ACF plot of the absolute of the log differences.
the appearance of long memory is really because a time-dependent mean has not been corrected
163
for. Could this be the reason we see the ‘memory’ in the log differences?
We now demonstrate that one must be careful when diagnosing long range dependence, because
a slow/none decay of the autocovariance could also imply a time-dependent mean that has not been
corrected for. This was shown in Bhattacharya et al. (1983), and applied to econometric data in
Mikosch and St˘
aric˘
a (2000) and Mikosch and St˘aric˘a (2003). A test for distinguishing between long
range dependence and change points is proposed in Berkes et al. (2006).
Suppose that Yt satisfies
Yt = µt + εt ,
where {εt } are iid random variables and the mean µt depends on t. We observe {Yt } but do not
know the mean is changing. We want to evaluate the autocovariance function, hence estimate the
autocovariance at lag k using
cˆn (k) =
n−|k|
1 X
(Yt − Y¯n )(Yt+|k| − Y¯n ).
n
t=1
Observe that Y¯n is not really estimating the mean but the average mean! If we plotted the empirical
ACF {ˆ
cn (k)} we would see that the covariances do not decay with time. However the true ACF
would be zero and at all lags but zero. The reason the empirical ACF does not decay to zero is
because we have corrected for the correct mean. Indeed it can be shown that
cˆn (k) =
n−|k|
1 X
(Yt − µt + µt − Y¯n )(Yt+|k| − µt+k + µt+k − Y¯n )
n
t=1
n−|k|
≈
n−|k|
1 X
1 X
(Yt − µt )(Yt+|k| − µt+k ) +
(µt − Y¯n )(µt+k − Y¯n )
n
n
t=1
t=1
n−|k|
≈
c(k)
|{z}
true autocovariance=0
+
1 X
(µt − Y¯n )(µt+k − Y¯n )
n
| t=1
{z
}
additional term due to time-dependent mean
Expanding the second term and assuming that k << n and µt ≈ µ(t/n) (and is thus smooth) we
164
have
≈
=
n−|k|
1 X
(µt − Y¯n )(µt+k − Y¯n )
n
t=1
!2
n
n
1X 2
1X
µt −
µt + op (1)
n
n
t=1
t=1
!2
n
n
n
1 XX 2
1X
µt + op (1)
µt −
n2
n
s=1 t=1
=
t=1
n
n
n
n
n
n
1 XX
1 XX
1 XX
2
µ
(µ
−
µ
)
=
(µ
−
µ
)
+
µs (µt − µs )
t
t
s
t
s
n2
n2
n2
s=1 t=1
s=1 t=1
| s=1 t=1 {z
}
=−
1
n2
Pn
s=1
Pn
t=1
µt (µt −µs )
Therefore
n−|k|
n
n
1 X
1 XX
(µt − Y¯n )(µt+k − Y¯n ) ≈ 2
(µt − µs )2 .
n
2n
t=1
s=1 t=1
Thus we observe that the sample covariances are positive and don’t tend to zero for large lags.
This gives the false impression of long memory.
It should be noted if you study a realisation of a time series with a large amount of dependence,
it is unclear whether what you see is actually a stochastic time series or an underlying trend. This
makes disentangling a trend from data with a large amount of correlation extremely difficult.
165
Chapter 7
Parameter estimation
Prerequisites
• The Gaussian likelihood.
Objectives
• To be able to derive the Yule-Walker and least squares estimator of the AR parameters.
• To understand what the quasi-Gaussian likelihood for the estimation of ARMA models is,
and how the Durbin-Levinson algorithm is useful in obtaining this likelihood (in practice).
Also how we can approximate it by using approximations of the predictions.
• Understand that there exists alternative methods for estimating the ARMA parameters,
which exploit the fact that the ARMA can be written as an AR(∞).
We will consider various methods for estimating the parameters in a stationary time series.
We first consider estimation parameters of an AR and ARMA process. It is worth noting that we
will look at maximum likelihood estimators for the AR and ARMA parameters. The maximum
likelihood will be constructed as if the observations were Gaussian. However, these estimators
‘work’ both when the process is Gaussian is also non-Gaussian. In the non-Gaussian case, the
likelihood simply acts as a contrast function (and is commonly called the quasi-likelihood). In time
series, often the distribution of the random variables is unknown and the notion of ‘likelihood’ has
little meaning. Instead we seek methods that give good estimators of the parameters, meaning that
they are consistent and as close to efficiency as possible without placing too many assumption on
166
the distribution. We need to ‘free’ ourselves from the notion of likelihood acting as a likelihood
(and attaining the Cr´
amer-Rao lower bound).
7.1
Estimation for Autoregressive models
Let us suppose that {Xt } is a zero mean stationary time series which satisfies the AR(p) representation
Xt =
p
X
φj Xt−j + εt ,
j=1
where E(εt ) = 0 and var(εt ) = σ 2 and the roots of the characteristic polynomial 1 −
Pp
j=1 φj z
j
lie outside the unit circle. We will assume that the AR(p) is causal (the techniques discussed
here will not consistently estimate the parameters in the case that the process is non-causal, they
were will only consistently estimate the corresponding causal model). Our aim in this section is to
construct estimator of the AR parameters {φj }. We will show that in the case that {Xt } has an
AR(p) representation the estimation is relatively straightforward, and the estimation methods all
have properties which are asymptotically equivalent to the Gaussian maximum estimator.
The Yule-Walker estimator is based on the Yule-Walker equations derived in (3.4) (Section
3.1.4).
7.1.1
The Yule-Walker estimator
We recall that the Yule-Walker equation state that if an AR process is causal, then for i > 0 we
have
E(Xt Xt−i ) =
p
X
φj E(Xt−j Xt−i ), ⇒ c(i) =
j=1
p
X
φj c(i − j).
(7.1)
j=1
Putting the cases 1 ≤ i ≤ p together we can write the above as
rp = Σp φp ,
(7.2)
where (Σp )i,j = c(i − j), (rp )i = c(i) and φ0p = (φ1 , . . . , φp ). Thus the autoregressive parameters
solve these equations.
167
The Yule-Walker equations inspire the method of moments estimator called the Yule-Walker
ˆ p are estimators of
estimator. We use (7.2) as the basis of the estimator. It is clear that ˆrp and Σ
ˆ p )i,j = cˆn (i − j) and (ˆrp )i = cˆn (i). Therefore we can use
rp and Σp where (Σ
ˆ =Σ
ˆ −1
φ
rp ,
p ˆ
p
(7.3)
as an estimator of the AR parameters φ0p = (φ1 , . . . , φp ). We observe that if p is large this involves
ˆ by
inverting a large matrix. However, we can use the Durbin-Levinson algorithm to calculate φ
p
recursively fitting lower order AR processes to the observations and increasing the order. This way
an explicit inversion can be avoided. We detail how the Durbin-Levinson algorithm can be used to
estimate the AR parameters below.
Step 1 Set φˆ1,1 = cˆn (1)/ˆ
cn (0) and rˆn (2) = 2ˆ
cn (0) − 2φˆ1,1 cˆn (1).
Step 2 For 2 ≤ t ≤ p, we define the recursion
φˆt,t =
φˆt,j
cˆn (t) −
= φˆt−1,j
Pt−1 ˆ
ˆn (t − j)
j=1 φt−1,j c
rˆn (t)
− φˆt,t φˆt−1,t−j
1 ≤ j ≤ t − 1,
and rˆn (t + 1) = rˆn (t)(1 − φˆ2t,t ).
Step 3 We recall from (5.12) that φt,t is the partial correlation between Xt+1 and X1 , therefore φˆtt
are estimators of the partial correlation.
As mentioned in Step 3, the Yule-Walker estimators have the useful property that the partial
correlations can easily be evaluated within the procedure. This is useful when trying to determine
the order of the model to fit to the data. In Figure 7.1 we give the partial correlation plot corresponding to Figure 6.1. We only the first two terms are outside the error bars. This rightly
suggests the time series comes from an autoregressive process of order two. Assuming that φˆt,t is
asymptotically normal the error bars (confidence interval) is determined by the variance of φˆt,t .
This can be determined by deriving the variance of φt .
The Yule-Walker estimator has the useful property that the parameter estimates {φˆj ; j =
ˆ
1, . . . , p} correspond to a causal AR(p), in other words, the roots corresponding to φ(z)
= 1−
Pp ˆ j
cn (r)} form a positive definite
j=1 φj z lie outside the unit circle. This is because the covariances {ˆ
ˆ p+1 )i,j =
sequence, thus there exists a random vector Z p+1 = (Z1 , . . . , Zp+1 ) where var[Z]p+1 = (Σ
168
0.0
−0.5
Partial ACF
0.5
Series ar2
5
10
15
20
Lag
Figure 7.1: Top: The sample partial autocorrelation plot of the AR(2) process Xt =
1.5Xt−1 + 0.75Xt−2 + εt with error bars n = 200.
cˆn (i−j), using this and the following result it follows that {φˆj ; j = 1, . . . , p} corresponds to a causal
AR process.
Lemma 7.1.1 Let us suppose Z p+1 = (Z1 , . . . , Zp+1 ) is a random vector, where var[Z]p+1 =
(Σp+1 )i,j = cn (i − j). Let Zp+1|p be the best linear predictor of Zp+1 given Zp , . . . , Z1 , where
φp = (φ1 , . . . , φp ) are the coefficients corresponding to the best linear predictor. Then the roots of
P
the corresponding characteristic polynomial φ(z) = 1 − pj=1 φj z j lie outside the unit circle.
PROOF. We first note that by definition of the best linear predictor, for any coefficients {aj } we
have the inequality
E[Zp+1 −
p
X
2
2
φj Zp+1−j ] = E[φ(B)Zp+1 ] ≤ E[Yp+1 −
j=1
p
X
aj Zp+1−j ]2 = E[a(B)Zp+1 ]2 .
(7.4)
j=1
We prove the result by contradiction. Let us suppose that there exists at least one root of φ(z),
which lies inside the unit circle. We denote this root as λ−1 (|λ| > 1) and factorize φ(z) as
φ(z) = (1 − λz)R(z), where R(z) contains the other remaining roots and can be either inside or
outside the unit circle.
Define the two new random variables, Yp+1 = R(B)Zp+1 and Yp = R(B)Zp (where B acts
as the backshift operator), which we note is a linear combination of Zp+1 , . . . , Z2 and Zp , . . . , Z1
P
Pp−1
respectively ie. Yp+1 = p−1
0=1 Ri Zp+1−i and Yp =
i=0 Ri Zp−i . The most important observation
in this construction is that the matrix Σp+1 is Toeplitz (ie. {Zt } is a ‘stationary’ vector), therefore
169
Yp+1 and Yp have the same covariance structure, in particular they have the same variance. Let
ρ=
cov[Yp+1 , Yp ]
cov[Yp+1 , Yp ]
= cor(Yp+1 , Yp ).
=q
var[Yp ]
var[Yp ]var[Yp+1 ]
|
{z
}
by stationarity
We recall that ρYp is the best linear predictor of Yp+1 given Yp and |ρ| ≤ 1.
We now define a new polynomial φ1 (z) = (1 − ρz)R(z), we will show that the mean squared
error of E[φ1 (B)Zp+1 ]2 ≤ E[φ(B)Zp+1 ]2 , which by (7.4) leads to a contradiction. Evaluating
E[φ1 (B)Zp+1 ]2 we have
E[φ1 (B)Zp+1 ]2 = E[(1 − ρB)R(B)Zp+1 ]2 = E[Yp+1 − ρYp ]2 ≤ E[Yp+1 − λYp ]2 = E[φ(B)Zp+1 ]2 .
From (7.4), the above can only be true if λ = ρ, and thus the root lies outside the unit circle. Thus
we see that all the roots of φ(z) can only lie outside the unit circle. Thus giving the required result.
The desirable property that the coefficients correspond to a causal autoregressive model hold
true for the Yule-Walker equations, however below we define the intuitively obvious least squares
estimator, which does not necessarily have this property. The least squares estimator is based can
either be defined in it’s own right or be considered as the conditional Gaussian likelihood. We start
by defining the Gaussian likelihood.
7.1.2
The Gaussian maximum likelihood
Our object here is to obtain the maximum likelihood estimator of the AR(p) parameters. We recall
that the maximum likelihood estimator is the parameter which maximises the joint density of the
observations. Since the log-likelihood often has a simpler form, we will focus on the log-likelihood.
We note that the Gaussian MLE is constructed as if the observations {Xt } were Gaussian, though it
is not necessary that {Xt } is Gaussian when doing the estimation. In the case that the innovations
are not Gaussian, estimator will be less efficient (will not obtain the Cramer-Rao lower bound)
then the likelihood constructed as if the distribution were known.
Suppose we observe {Xt ; t = 1, . . . , n} where Xt are observations from an AR(p) process. Let
us suppose for the moment that the innovations of the AR process are Gaussian, this implies that
170
X n = (X1 , . . . , Xn ) is a n-dimension Gaussian random vector, with the corresponding log-likelihood
Ln (a) = − log |Σn (a)| − X0n Σn (a)−1 Xn ,
(7.5)
where Σt (a) the variance covariance matrix of Xn constructed as if Xn came from an AR process
with parameters a. Of course, in practice in the likelihood in the form given above is impossible to
maximise. Therefore we need to rewrite the likelihood in a more tractable form.
We now derive a tractable form of the likelihood under the assumption that the innovations
come from an arbitrary distribution. To construct the likelihood, we use the method of conditioning,
to write the likelihood as the product of conditional likelihoods. In order to do this, we derive the
conditional distribution of Xt+1 given Xt−1 , . . . , X1 . We first note that the AR(p) process is pMarkovian, therefore if t ≥ p all the information about Xt+1 is contained in the past p observations,
therefore
P(Xt+1 ≤ x|Xt , Xt−1 , . . . , X1 ) = P(Xt+1 ≤ x|Xt , Xt−1 , . . . , Xt−p+1 ).
(7.6)
Since the Markov property applies to the distribution function it also applied to the density
f (Xt+1 |Xt , . . . , X1 ) = f (Xt+1 |Xt , . . . , Xt−p+1 ).
By using the (7.6) we have
P(Xt+1 ≤ x|Xt , . . . , X1 ) = P(Xt+1 ≤ x|Xt , . . . , X1 ) = P ( ≤ x −
p
X
aj Xt+1−j ),
(7.7)
j=1
where P denotes the distribution of the innovation. Differentiating P with respect to Xt+1 gives
f (Xt+1 |Xt , . . . , Xt−p+1 ) =
∂P ( ≤ Xt+1 −
Pp
j=1 aj Xt+1−j )
∂Xt+1

= f Xt+1 −
p
X

aj Xt+1−j  . (7.8)
j=1
Example 7.1.1 (AR(1)) To understand why (7.6) is true consider the simple case that p = 1
171
(AR(1)). Studying the conditional probability gives
P(Xt+1 ≤ xt+1 |Xt = xt , . . . , X1 = x1 ) = P(
aX + t ≤ xt+1
}
| t {z
|Xt = xt , . . . , X1 = x1 )
all information contained in Xt
= P (t ≤ xt+1 − axt ) = P(Xt+1 ≤ xt+1 |Xt = xt ),
where Pε denotes the distribution function of the innovation ε.
Using (7.8) we can derive the joint density of {Xt }nt=1 . By using conditioning we obtain
f(X1 , X2 , . . . , Xn ) = f (X1 , . . . , Xp )
n−1
Y
f(Xt+1 |Xt , . . . , X1 )
(by repeated conditioning)
t=p
= f(X1 , . . . , Xp )
n−1
Y
f(Xt+1 |Xt , . . . , Xt−p+1 )
(by the Markov property)
t=p
= f(X1 , . . . , Xp )
n−1
Y
f (Xt+1 −
t=p
p
X
aj Xt+1−j )
(by (7.8)).
j=1
Therefore the log likelihood is
p
n−1
X
X
log f (Xt+1 −
aj Xt+1−j ) .
log f(X1 , X2 , . . . , Xn ) = log f(X1 , . . . , Xp ) +
|
{z
}
|
{z
} t=p
j=1
Full log-likelihood Ln (a;X n )
initial observations
|
{z
}
conditional log-likelihood=Ln (a;X n )
In the case that the sample sizes are large n >> p, the contribution of initial observations
log f(X1 , . . . , Xp ) is minimal and the conditional log-likelihood and full log-likelihood are asymptotically equivalent.
So far we have not specified the distribution of ε. From now on we shall assume that it is
Gaussian. In the case that ε is Gaussian, log f(X1 , . . . , Xp ) is multivariate normal with mean zero
(since we are assuming, for convenience, that the time series has zero mean) and variance Σp . We
recall that Σp (a) is a Toeplitz matrix whose covariance is determined by the AR parameters a, see
(3.7). As can be seen from (3.7), the coefficients are ‘buried’ within the covariance (which is in
terms of the roots of the characteristic), this makes it quite an unpleasant part of the likelihood to
maximise. On the other hand the conditional log-likelihood has a far simpler form
Ln (a; X) = −n log σ 2 −
1
σ2
n−1
X

Xt+1 −
t=p
172
p
X
j=1
2
aj Xt+1−j  .
The maximum likelihood estimator is
b = arg max − log |Σp (a)| − X 0 Σp (a)−1 X + Ln (a; X) .
φ
p
p
n
a∈Θ
(7.9)
By constraining the parameter space, we an ensure the estimator correspond to a causal AR process.
However, it is clear that despite having the advantage that it attains the Cr´amer-Rao lower bound
in the case that the innovations are Gaussian, it not simple to evaluate. A far simpler estimator can
be obtained, by simply focusing on the conditiona log likelihood Ln (a; X). An explicit expression
for it’s maximum can easily be obtained (as long as we do not constrain the parameter space). It
˜ = arg max Ln (a; X) and that
is simply the least squares estimator, in other words, φ
p
˜ =Σ
˜ −1
φ
rp ,
p ˜
p
˜ p )i,j =
where (Σ
1
n−p
Pn
t=p+1 Xt−i Xt−j
and (˜rn )i =
1
n−p
Pn
t=p+1 Xt Xt−i .
Remark 7.1.1 (A comparison of the Yule-Walker and least squares estimators) Comparing
˜ = Σ
ˆ = Σ
˜ −1 ˜rp with the Yule-Walker estimator φ
ˆ −1 ˆrp we see that
the least squares estimator φ
p
p
p
p
˜ p and Σ
ˆ p (and the corresponding ˜rp and ˆrp ). We see
they are very similar. The difference lies in Σ
ˆ p is a Toeplitz matrix, defined entirely by the positive definite sequence cˆn (r). On the other
that Σ
˜ p is not a Toeplitz matrix, the estimator of c(r) changes subtly at each row. This means
hand, Σ
that the proof given in Lemma 7.1.1 cannot be applied to the least squares estimator as it relies
on the matrix Σp+1 (which is a combination of Σp and rp ) being Toeplitz (thus stationary). Thus
the characteristic polynomial corresponding to the least squares estimator will not necessarily have
roots which lie outside the unit circle.
Example 7.1.2 (Toy Example) To illustrate the difference between the Yule-Walker and least
squares estimator (at least for example samples) consider the rather artifical example that the time
series consists of two observations X1 and X2 (we will assume the mean is zero). We fit an AR(1)
model to the data, the least squares estimator of the AR(1) parameter is
X1 X2
φbLS =
X22
173
whereas the Yule-Walker estimator of the AR(1) parameter is
φbY W =
X1 X2
.
X12 + X22
It is clear that φbLS < 1 only if X2 < X1 . On the other hand φbY W < 1. Indeed since (X1 −X2 )2 > 0,
we see that φbY W ≤ 1/2.
7.2
Estimation for ARMA models
Let us suppose that {Xt } satisfies the ARMA representation
Xt −
p
X
φi Xt−i = εt +
i=1
q
X
θj εt−j ,
j=1
and θ = (θ1 , . . . , θq ), φ = (φ1 , . . . , φp ) and σ 2 = var(εt ). We will suppose for now that p and
q are known. The objective in this section is to consider various methods for estimating these
parameters.
7.2.1
The Gaussian maximum likelihood estimator
We now derive the Gaussian maximum likelihood estimator (GMLE) to estimate the parameters θ
and φ. Let X 0n = (X1 , . . . , Xn ). The criterion (the GMLE) is constructed as if {Xt } were Gaussian,
but this need not be the case. The likelihood is similar to the likelihood given in (7.5), but just as
in the autoregressive case it can be not directly maximised, ie.
Ln (φ, θ, σ) = − log |Σn (φ, θ, σ)| − X 0n Σn (φ, θ, σ)−1 X n ,
where Σn (φ, θ, σ) the variance covariance matrix of X n . However, in Section 5.3.3, equation (5.21)
the Cholesky decomposition of Σn is given and using this we can show that
X 0n Σn (φ, θ, σ)−1 X n
P
n−1
X (Xt+1 − tj=1 φt,j (θ)Xt+1−j )2
X12
+
,
=
r(1; θ)
r(t + 1; θ)
t=1
174
0
−1
2
where θ = (φ, θ, σ 2 ). Furthermore, since Σ−1
n = Ln Dn Ln , then det(Σn ) = det(Ln ) det(Dn ) =
Pn
Qn
−1
t=1 log r(t; θ). Thus the log-likelihood is
t=1 r(t) , this implies log |Σn (φ, θ, σ)| =
Ln (φ, θ, σ) = −
n
X
t=1
P
n−1
X (Xt+1 − tj=1 φt,j (θ)Xt+1−j )2
X12
−
.
log r(t; θ) −
r(1; θ)
r(t + 1; θ)
t=1
We recall from (5.24) that best linear predictor,
Pt
j=1 φt,j (θ)Xt+1−j ,
can be simplified by taking
into account the ARMA structure
X 0n Σn (φ, θ, σ)−1 X n
=
X12
+
r(1; θ)
n−1
X
max(p,q)
X
(Xt+1 −
Pt
2
j=1 φt,j (θ)Xt+1−j )
+
r(t + 1; θ)
P
P
(Xt+1 − pj=1 φj Xt+1−j − qi=1 θi (Xt+1−i − Xt+1−i|t−i (θ))2
t=1
r(t + 1; θ)
max(p,q)
.
Substituting this into Ln (θ) gives
P
max(p,q)
X (Xt+1 − tj=1 φt,j (θ)Xt+1−j )2
X12
Ln (φ, θ, σ) = −
log r(t; θ) −
−
−
r(1; θ)
r(t + 1; θ)
t=1
t=1
P
P
n−1
X (Xt+1 − pj=1 φj Xt+1−j − qi=1 θi (Xt+1−i − Xt+1−i|t−i (θ))2
.
r(t + 1; θ)
n
X
max(p,q)
ˆ , σ 2 which maximises Ln (θ).
The maximum likelihood estimator are the parameters ˆθn , φ
n
We can also use an approximation to the log-likelihood which can simplify the estimation
ˆ t+1|t . This motivates the approxischeme. We recall in Section 5.4 we approximated Xt+1|t with X
ˆ t+1|t , where X
ˆ t+1|t is defined in (5.25) to give the
mation where we replace in Ln (θ) Xt+1|t with X
approximate Gaussian log-likelihood
P
max(p,q)
2
X (Xt+1 − tj=1 φt,j (θ)Xt+1−j )2
X
1
b n (φ, θ, σ) = −
log r(t; θ) −
−
−
L
r(1; θ)
r(t + 1; θ)
t=1
t=1
P
P
n−1
ˆ t+1−i|t−i (θ))2
X (Xt+1 − pj=1 φj Xt+1−j − qi=1 θi (Xt+1−i − X
.
r(t + 1; θ)
n
X
max(p,q)
Theorem 7.2.1 Let us suppose that Xt has a causal and invertible ARMA representation
Xt −
p
X
φj Xt−j = +εi +
j=1
q
X
i=1
175
θi εt−i
where {εt } are iid random variables with mean zero and var[εt ] = σ 2 . Then the the (quasi)-Gaussian
√
D
ˆ − φ, ψ
ˆ − ψ) →
n(φ
N (0, σ 2 Λ−1 ),
n
n
with

Λ=
E(Ut U0t ) E(Vt U0t )

E(Ut Vt0 ) E(Vt Vt0 )

and Ut = (Ut , . . . , Ut−p+1 ) and Vt = (Vt , . . . , Vt−q+1 ), where {Ut } and {Vt } are autoregressive
processes which satisfy φ(B)Ut = εt and θ(B)Vt = εt .
7.2.2
The Hannan-Rissanen AR(∞) expansion method
The methods detailed above require good initial values in order to begin the maximisation (in order
to prevent convergence to a local maximum).
We now describe a simple method first propose in Hannan and Rissanen (1982) and An et al.
(1982). It is worth bearing in mind that currently the ‘large p small n problem’ is a hot topic.
These are generally regression problems where the sample size n is quite small but the number of
regressors p is quite large (usually model selection is of importance in this context). The methods
proposed by Hannan involves expanding the ARMA process (assuming invertibility) as an AR(∞)
process and estimating the parameters of the AR(∞) process. In some sense this can be considered
as a regression problem with an infinite number of regressors. Hence there are some parallels
between the estimation described below and the ‘large p, small n problem’.
As we mentioned in Lemma 2.5.1, if an ARMA process is invertible it is can be represented as
Xt =
∞
X
bj Xt−j + εt .
(7.10)
j=1
The idea behind Hannan’s method is to estimate the parameters {bj }, then estimate the innovations
εt , and use the estimated innovations to construct a multiple linear regression estimator of the
ARMA paramters {θi } and {φj }. Of course in practice we cannot estimate all parameters {bj } as
there are an infinite number of them. So instead we do a type of sieve estimation where we only
estimate a finite number and let the number of parameters to be estimated grow as the sample size
increases. We describe the estimation steps below:
176
n
(i) Suppose we observe {Xt }nt=1 . Recalling (7.10), will estimate {bj }pj=1
parameters. We will
suppose that pn → ∞ as n → ∞ and pn << n (we will state the rate below).
n
We use least squares to estimate {bj }pj=1
and define
ˆb = R
ˆ −1 ˆrn ,
n
n
where
ˆn =
R
n
X
X t−1 X 0t−1
T
X
ˆrn =
t=pn +1
Xt X t−1
t=pn +1
and X 0t−1 = (Xt−1 , . . . , Xt−pn ).
n
coefficients we estimate the residuals with
(ii) Having estimated the first {bj }pj=1
ε˜t = Xt −
pn
X
ˆbj,n Xt−j .
j=1
˜ , ˜θ where
(iii) Now use as estimates of φ0 and θ0 φ
n n
n
X
˜ , ˜θ = arg min
φ
n n
(Xt −
t=pn +1
p
X
φj Xt−j −
j=1
q
X
θi ε˜t−i )2 .
i=1
We note that the above can easily be minimised. In fact
˜ , ˜θ ) = R
˜ −1 ˜sn
(φ
n
n n
where
˜n = 1
R
n
n
X
˜t
Y˜ t Y
t=max(p,q)
1
and ˜sn =
n
0
Y˜ t = (Xt−1 , . . . , Xt−p , ε˜t−1 , . . . , ε˜t−q ).
177
n
X
t=max(p,q)
Y˜ t Xt ,
7.3
The quasi-maximum likelihood for ARCH processes
In this section we consider an estimator of the parameters a0 = {aj : j = 0, . . . , p} given the
observations {Xt : t = 1, . . . , N }, where {Xt } is a ARCH(p) process. We use the conditional logP
likelihood to construct the estimator. We will assume throughout that E(Zt2 ) = 1 and pj=1 αj =
ρ < 1.
We now construct an estimator of the ARCH parameters based on Zt ∼ N (0, 1). It is worth
mentioning that despite the criterion being constructed under this condition it is not necessary
that the innovations Zt are normally distributed. In fact in the case that the innovations are not
normally distributed but have a finite fourth moment the estimator is still good. This is why it
is called the quasi-maximum likelihood , rather than the maximum likelihood (similar to the how
the GMLE estimates the parameters of an ARMA model regardless of whether the innovations are
Gaussian or not).
q
P
2 , E(X |X
Let us suppose that Zt is Gaussian. Since Zt = Xt / a0 + pj=1 aj Xt−j
t
t−1 , . . . , Xt−p ) =
Pp
2 , then the log density of X given X
0 and var(Xt |Xt−1 , . . . , Xt−p ) = a0 + j=1 aj Xt−j
t
t−1 , . . . , Xt−p
is
log(a0 +
p
X
2
aj Xt−j
)+
j=1
a0 +
X2
Pp t
2 .
j=1 aj Xt−j
Therefore the conditional log density of Xp+1 , Xp+2 , . . . , Xn given X1 , . . . , Xp is
n X
log(a0 +
t=p+1
p
X
2
aj Xt−j
)
+
j=1
Xt2
a0 +
Pp
2
j=1 aj Xt−j
.
This inspires the the conditional log-likelihood
p
n X
X
1
Xt2
2
P
Ln (α) =
log(α0 +
αj Xt−j ) +
.
2
n−p
α0 + pj=1 αj Xt−j
t=p+1
j=1
To obtain the estimator we define the parameter space
Θ = {α = (α0 , . . . , αp ) :
p
X
αj ≤ 1, 0 < c1 ≤ α0 ≤ c2 < ∞, c1 ≤ αj }
j=1
178
and assume the true parameters lie in its interior a = (a0 , . . . , ap ) ∈ Int(Θ). We let
a
ˆn = arg min Ln (α).
α∈Θ
179
(7.11)
Appendix A
Background: some definition and
inequalities
• Some norm definitions.
The norm of an object, is a postive numbers which measure the ‘magnitude’ of that object.
P
P
Suppose x = (x1 , . . . , xn ) ∈ Rn , then we define kxk1 = nj=1 |xj | and kxk2 = ( nj=1 |x2j )1/2
(this is known as the Euclidean norm). There are various norms for matrices, the most
popular is the spectral norm k · kspec : let A be a matrix, then kAkspec = λmax (AA0 ), where
λmax denotes the largest eigenvalue.
• Z denotes the set of a integers {. . . , −1, 0, 1, 2, . . .}. R denotes the real line (−∞, ∞).
• Complex variables.
√
i = −1 and the complex variable z = x + iy, where x and y are real.
Often the radians representation of a complex variable is useful. If z = x + iy, then it can
p
also be written as r exp(iθ), where r = x2 + y 2 and θ = tan−1 (y/x).
If z = x + iy, its complex conjugate is z¯ = x − iy.
• The roots of a rth order polynomial a(z), are those values λ1 , . . . , λr where a(λi ) = 0 for
i = 1, . . . , r.
• Let λ(A) denote the spectral radius of the the matrix A (the largest eigenvalue in absolute
terms). Then for any matrix norm kAk we have limj→∞ kAj k1/j = λ(A) (see Gelfand’s
180
formula). Suppose λ(A) < 1, then Gelfand’s formula implies that for any λ(A) < ρ < 1,
there exists a constant, C, (which only depends A and ρ), such that kAj k ≤ CA,ρ ρj .
• The mean value theorem.
This basically states that if the partial derivative of the function f (x1 , x2 , . . . , xn ) has a
bounded in the domiain Ω, then for x = (x1 , . . . , xn ) and y = (y1 , . . . , yn )
n
X
∂f
cx=x∗
f (x1 , x2 , . . . , xn ) − f (y1 , y2 , . . . , yn ) =
(xi − yi )
∂xi
i=1
where x∗ lies somewhere between x and y.
• The Taylor series expansion.
This is closely related to the mean value theorem and a second order expansion is
f (x1 , x2 , . . . , xn ) − f (y1 , y2 , . . . , yn ) =
n
X
(xi − yi )
n
X
∂f 2
∂f
+
(xi − yi )(xj − yj )
cx=x∗
∂xi
∂xi ∂xj
i,j=1
i=1
• Partial Fractions.
We use the following result mainly for obtaining the MA(∞) expansion of an AR process.
Suppose that |gi | > 1 for 1 ≤ i ≤ n. Then if g(z) =
Qn
i=1 (1
− z/gi )ri , the inverse of g(z)
satisfies
n
r
i=1
j=1
i
XX
gi,j 1
=
,
g(z)
(1 − gzi )j
where gi,j = ..... Now we can make a polynomial series expansion of (1 − gzi )−j which is valid
for all |z| ≤ 1.
• Dominated convergence.
Suppose a sequence of functions fn (x) is such that pointwise fn (x) → f (x) and for all n and
R
R
x, |fn (x)| ≤ g(x), then fn (x)dx → f (x)dx as n → ∞.
We use this result all over the place to exchange infinite sums and expectations. For example,
181
if
P∞
j=1 |aj |E(|Zj |)
< ∞, then by using dominated convergence we have
∞
∞
X
X
E(
aj Zj ) =
aj E(Zj ).
j=1
j=1
• Cauchy Schwarz inequality.
In terms of sequences it is
|
∞
X
∞
∞
X
X
2 1/2
aj bj | ≤ (
aj ) (
b2j )1/2
j=1
j=1
j=1
. For integrals and expectations it is
E|XY | ≤ E(X 2 )1/2 E(Y 2 )1/2
• Holder’s inequality.
This is a generalisation of the Cauchy Schwarz inequality. It states that if 1 ≤ p, q ≤ ∞ and
p + q = 1, then
E|XY | ≤ E(|X|p )1/p E(|Y |q )1/q
. A similar results is true for sequences too.
• Martingale differences. Let Ft be a sigma-algebra, where Xt , Xt−1 , . . . ∈ Ft . Then {Xt } is a
sequence of martingale differences if E(Xt |Ft−1 ) = 0.
• Minkowski’s inequality.
If 1 < p < ∞, then
n
n
X
X
(E(
Xi )p )1/p ≤
(E(|Xi |p ))1/p .
i=1
i=1
• Doob’s inequality.
This inequality concerns martingale differences. Let Sn =
2
E( sup |Sn |2 ) ≤ E(SN
).
n≤N
182
Pn
t=1 Xt ,
then
• Burkh¨
older’s inequality.
Suppose that {Xt } are martingale differences and define Sn =
Pn
k=1 Xt .
For any p ≥ 2 we
have
{E(Snp )}1/p ≤ 2p
n
X
E(Xkp )2/p
1/2
.
k=1
An application, is to the case that {Xt } are identically distributed random variables, then
we have the bound E(Snp ) ≤ E(X0p )2 (2p)p/2 np/2 .
It is worthing noting that the Burkh¨older inequality can also be defined for p < 2 (see
Davidson (1994), pages 242). It can also be generalised to random variables {Xt } which are
not necessarily martingale differences (see Dedecker and Doukhan (2003)).
• Riemann-Stieltjes Integrals.
R
In basic calculus we often use the basic definition of the Riemann integral, g(x)f (x)dx, and if
R
R
the function F (x) is continuous and F 0 (x) = f (x), we can write g(x)f (x)dx = g(x)dF (x).
There are several instances where we need to broaden this definition to include functions F
which are not continuous everywhere. To do this we define the Riemann-Stieltjes integral,
which coincides with the Riemann integral in the case that F (x) is continuous.
R
R
g(x)dF (x) is defined in a slightly different way to the Riemann integral g(x)f (x)dx.
P
Let us first consider the case that F (x) is the step function F (x) = ni=1 ai I[xi−1 ,xi ] , then
R
R
Pn
g(x)dF (x) is defined as g(x)dF (x) =
i=1 (ai − ai−1 )g(xi ) (with a−1 = 0). Already
we see the advantage of this definition, since the derivative of the step function is not
well defined at the jumps. As most functions can be written as the limit of step funcR
P k
tions (F (x) = limk∞ Fk (x), where Fk (x) = ni=1
ai,nk I[xi −1,x ] ), we define g(x)dF (x) =
ik
k−1
Pnk
limk→∞ i=1 (ai,nk − ai−1,nk )g(xik ).
In statistics the function F will usually non-decreasing and bounded. We call such functions
distributions.
Theorem A.0.1 (Helly’s Theorem) Suppose that {Fn } are a sequence of distributions with
Fn (−∞) = 0 and supn Fn (∞) ≤ M < ∞. There exists a distribution F , and a subsequence
Fnk such that for each x ∈ R Fnk → F and F is right continuous.
183
Remark A.0.1 Martingales arise all the time. Its useful to know if the true distributional is
used, the gradient of the conditional log likelihood evaluated at the true parameter is the sum of
P
martingale differences. We show why this is true now. Let BT = Tt=2 log fθ (Xt |Xt−1 , . . . , X1 ) be
the conditonal log likelihood and CT (θ) its derivative, where
CT (θ) =
T
X
∂ log fθ (Xt |Xt−1 , . . . , X1 )
∂θ
t=2
.
We want to show that CT (θ0 ) is the sum of martingale differences. By definition if CT (θ0 ) is the
sum of martingale differences then
E
∂ log fθ (Xt |Xt−1 , . . . , X1 )
cθ=θ0 Xt−1 , Xt−2 , . . . , X1 = 0,
∂θ
we will show this. Rewriting the above in terms of integrals and exchanging derivative with integral
we have
∂ log fθ (Xt |Xt−1 , . . . , X1 )
cθ=θ0 Xt−1 , Xt−2 , . . . , X1
∂θ
∂ log fθ (xt |Xt−1 , . . . , X1 )
cθ=θ0 fθ0 (xt |Xt−1 , . . . , X1 )dxt
∂θ
∂fθ (xt |Xt−1 , . . . , X1 )
1
cθ=θ0 fθ0 (xt |Xt−1 , . . . , X1 )dxt
fθ0 (xt |Xt−1 , . . . , X1 )
∂θ
Z
fθ (xt |Xt−1 , . . . , X1 )dxt cθ=θ0 = 0.
E
Z
=
Z
=
=
∂
∂θ
|Xt−1 ,...,X1 )
Therefore { ∂ log fθ (Xt∂θ
cθ=θ0 }t are a sequence of martingale differences and Ct (θ0 ) is the
sum of martingale differences (hence it is a martingale).
184
Bibliography
Hong-Zhi An, Zhao-Guo. Chen, and E.J. Hannan. Autocorrelation, autoregression and autoregressive approximation. Ann. Statist., 10:926–936, 1982.
I. Berkes, L. Horv´
ath, and P. Kokoskza. GARCH processes: Structure and estimation. Bernoulli,
9:2001–2007, 2003.
I. Berkes, L. Horvath, P. Kokoszka, and Q. Shao. On discriminating between long range dependence
and changes in mean. Ann. Statist., 34:1140–1165, 2006.
R.N. Bhattacharya, V.K. Gupta, and E. Waymire. The hurst effect under trend. J. Appl. Probab.,
20:649–662, 1983.
T Bollerslev. Generalized autoregressive conditional heteroscedasticity. J. Econometrics, 31:301–
327, 1986.
G. E. P. Box and G. M. Jenkins. Time Series Analysis, Forecasting and Control. Cambridge
University Press, Oakland, 1970.
A. Brandt. The stochastic equation Yn+1 = An Yn + Bn with stationary coefficients. Adv. in Appl.
Probab., 18:211–220, 1986.
P. Brockwell and R. Davis. Time Series: Theory and Methods. Springer, New York, 1998.
J Davidson. Stochastic Limit Theory. Oxford University Press, Oxford, 1994.
J. Dedecker and P. Doukhan. A new covariance inequality. Stochastic Processes and their applications, 106:63–80, 2003.
R. Douc, E. Moulines, and D. Stoffer. Nonlinear Time Series: Theory, Methods and Applications
with R Examples. Chapman and Hall, 2014.
185
R. Engle. Autoregressive conditional heteroscedasticity with estimates of the variance of the United
Kingdom inflation. Econometrica, 50:987–1006, 1982.
J. Fan and Q. Yao. Nonlinear time series: Nonparametric and parametric methods. Springer,
Berlin, 2003.
W. Fuller. Introduction to Statistical Time Series. Wiley, New York, 1995.
C. W. J. Granger and A. P. Andersen. An introduction to Bilinear Time Series models. Vandenhoek
and Ruprecht, G¨
ottingen, 1978.
E.J. Hannan and Rissanen. Recursive estimation of ARMA order. Biometrika, 69:81–94, 1982.
J. Hart. Kernel regression estimation with time series errors. Journal of the Royal Statistical
Society, 53:173–187, 1991.
D. A. Jones. Nonlinear autoregressive processes. Proceedings of the Royal Society (A), 360:71–95,
1978.
Rosenblatt. M. and U. Grenander. Statistical Analysis of Stationary Time Series. Chelsea Publishing Co, 1997.
T. Mikosch and C. St˘
aric˘
a. Is it really long memory we see in financial returns? In P. Embrechts,
editor, Extremes and Integrated Risk Management, pages 149–168. Risk Books, London, 2000.
T. Mikosch and C. St˘
aric˘
a. Long-range dependence effects and arch modelling. In P. Doukhan,
G. Oppenheim, and M.S. Taqqu, editors, Theory and Applications of Long Range Dependence,
pages 439–459. Birkh¨
auser, Boston, 2003.
D.F. Nicholls and B.G. Quinn.
Random Coefficient Autoregressive Models, An Introduction.
Springer-Verlag, New York, 1982.
M. Pourahmadi. Foundations of Time Series Analysis and Prediction Theory. Wiley, 2001.
M. B. Priestley. Spectral Analysis and Time Series: Volumes I and II. Academic Press, London,
1983.
B.G. Quinn and E.J. Hannan. The Estimation and Tracking of Frequency. Cambridge University
Press, 2001.
186
X. Shao. A self-normalized approacj to confidence interval construction in time series. Journal of
the Royal Statistical Society (B), 72:343–366, 2010.
R. Shumway and D. Stoffer. Time Series Analysis and Its applications: With R examples. Springer,
New York, 2006.
D. Straumann. Estimation in Conditionally Heteroscedastic Time Series Models. Springer, Berlin,
2005.
T. Subba Rao. On the estimation of bilinear time series models. In Bull. Inst. Internat. Statist.
(paper presented at 41st session of ISI, New Delhi, India), volume 41, 1977.
T. Subba Rao. On the theory of bilinear time series models. Journal of the Royal Statistical
Society(B), 43:244–255, 1981.
T. Subba Rao and M. M. Gabr. An Introduction to Bispectral Analysis and Bilinear Time Series
Models. Lecture Notes in Statistics (24). Springer, New York, 1984.
S. C. Taylor. Modelling Financial Time Series. John Wiley and Sons, Chichester, 1986.
Gy. Terdik. Bilinear Stochastic Models and Related Problems of Nonlinear Time Series Analysis;
A Frequency Domain Approach, volume 142 of Lecture Notes in Statistics. Springer Verlag, New
York, 1999.
M. Vogt. Nonparametric regression for locally stationary time series. Annals of Statistics, 40:
2601–2633, 2013.
A. M. Walker. On the estimation of a harmonic component in a time series with stationary
independent residuals. Biometrika, 58:21–36, 1971.
187