1. science
2. mathematics
3. statistics

Vol. 50 No. 1 (MAY, 2013), 133–161
How Informative are In-sample
Information Criteria to Forecasting?
The Case of Chilean GDP*
Carlos A. Medel**
This paper compares out-of-sample performance, using the Chilean GDP
dataset, of a large number of autoregressive integrated moving average
(ARIMA) models with some variations to identify how to achieve the
smallest root mean squared forecast error with models based on information
criteria—Akaike, Schwarz, and Hannan-Quinn. The analysis also addresses
the role of seasonal adjustment and the Easter ef fect. The results show
that Akaike and Schwarz are better criteria for forecasting when using
actual series and Schwarz and Hannan-Quinn are better with seasonally
adjusted data. Accounting for the Easter ef fect improves forecast accuracy
for actual and seasonally adjusted data.
JEL classification: C13, C22, C52, C53
Keywords: Data mining, forecasting, ARIMA, seasonal adjustment,
Easter ef fect
1.
Introduction
An accurate forecast is a key element of successful economic decisionmaking, but there is no standard procedure for selecting a model that
systematically outperforms the others at all horizons and with any
dataset. A common way to proceed is to choose the best specification
within a family of models based on a fitting criterion and then to
generate a forecast. In this paper, I compare and evaluate the outof-sample performance of a large number of autoregressive integrated
moving average models (ARIMA) chosen according to three commonly
used information criteria for model-building: Akaike, Bayesian or
Schwarz, and Hannan-Quinn (henceforth AIC, BIC, and HQ),1 on a
* I would especially like to thank Pablo Pincheira for his comments, support and insights. I am also
grateful to Carlos Alvarado, Yan Carrière-Swallow, Francis X. Diebold, Michael Pedersen, Sergio
Salgado, Raimundo Soto (editor), Consuelo Edwards, two anonymous referees, and the participants in
seminars at Universidad Santo Tomás and the Central Bank of Chile. This is a revised and modified
version of Central Bank of Chile Working Paper 657. Any errors or omissions are my own responsibility. The views expressed in this paper do not necessarily represent the opinion of the Central Bank of
Chile or its authorities.
** Economic Research Department, Central Bank of Chile. Agustinas 1180, Oficina 451C, Santiago,
Chile. Phone: +56 2 3882256. Email: [email protected].
1. See Akaike (1974), Shibata (1976), Rissasen (1978), Schwarz (1978), Hannan and Quinn (1979),
Lütkepohl (1985), and Koehler and Murphree (1988) for details.
doi 10.7764/LAJE.50.1.133
134
LATIN AMERICAN JOURNAL OF ECONOMICS | Vol. 50 No. 1 (May, 2013), 133–161
publicly available Chilean GDP dataset without revisions, estimated
with a rolling window sample for 1- to 4-steps-ahead forecasts. I also
examine the role of seasonal adjustment and the Easter ef fect on outof-sample performance, whose results are not comparable across all
kinds of series (Ghysels, Osborn, and Rodrigues, 2006). The forecast
evaluation is based on the root mean squared forecast error (RMSFE)
and the Giacomini and White (2006) test. I consider five dif ferent
aggregation blocks that compound the GDP. I perform this exercise
to determine which information criterion leads the forecaster to select
the most accurate model.
The goals of this paper are: (i) to investigate the use of the traditional
Box-Jenkins approach to find an adequate out-of-sample benchmark
for forecasting studies, going beyond the typical naive model;2 (ii) to
provide an informed opinion on the use of three popular information
criteria for forecasting in the Chilean economy and with dif ferent
aggregations; and (iii) to investigate the accuracy achieved by using
seasonally adjusted data and considering the impact of the Easter
ef fect on RMSFE as a parallel exercise. I consider the specific case of
the Chilean GDP dataset as a representative case of an economy that
has experimented dif ferent dynamics throughout a (publicly available)
long-sample span.3
The forecasts may come from models that explain more than what
actually is explained by the data-generating processes, known as
overfitting. An overfitted model may have poor predictive performance,
as it can exaggerate minor fluctuations in the data. In the context of
this paper, the only mechanism to prevent overfitting is the penalty
term imposed by each information criterion to the number of regressors
included in the model. Thus, I perform the Giacomini and White
(2006) test to determine if the mentioned dif ference in selected model
dynamics has a significant impact on RMSFE.
As pointed out by Granger and Jeon (2004), most results from comparing
the performance between information criteria are theoretical, and the
empirical evidence with actual data on relative forecast accuracy in
using each information criterion has not been thoroughly examined.
2. See Box and Jenkins (1970), and Box, Jenkins, and Reinsel (2008) for details about ARMA modeling.
3. Also, the Chilean GDP dataset fulfills other desirable characteristics for model evaluation, as (i) it
does not have induced breaks or level shifts due to methodological changes, (ii) it is prepared and released
on a quarterly basis, (iii) it is compounded by a number of sectors with dif ferent statistical behavior,
and (iv) it does not have any missing values, mismatches, or any other shortcoming.
C.A. Medel | How Informative are In-sample Information Criteria to Forecasting?
135
Granger and Jeon (2001), by comparing the number of regressors
chosen according to each information criterion, found that AIC tends
to select more dynamic models than the BIC. Further, Clark (2004)
proved that the number of (auto)regressors of a model is inversely
related to out-of-sample forecast capacity, highlighting the cost of
overfitting; this finding was shared with Hansen (2009). This leads
us to expect BIC to beat AIC in out-of-sample exercises. Moreover,
Granger and Jeon (2004) find that an equally weighted combination
between the forecasts delivered by each information criterion increases
ef ficiency. It is intuitive to think of HQ as a combination of AIC and
BIC by including a penalization of the number of regressors as well
as the asymptotic behavior, suggesting that it may be worth learning
about its out-of-sample, model-based features. Notwithstanding, and
regardless of some theoretical findings, I treat the matter of which
information criterion outperforms the others as an empirical question,
the answer to which depends to some extent on the characteristics
of the dataset at hand. For example, in the case of Chilean inflation,
Cobb (2009) used multivariate, time-series models based on AIC and
BIC and found that AIC consistently provided more predictive power.
Using the proposed extended SARIMA models, Pincheira and García
(2010) also pointed out that AIC gives better predictive results than
those chosen with BIC.
This paper also relates to seasonal adjustment and treatments of the
Easter ef fect. I replicate the exercise of choosing the best ARMA model
to generate multi-horizon forecasts in the same setting, considering
data that has been seasonally adjusted using: (i) the X12-ARIMA
methodology, and (ii) explicit regressors (seasonal ARMA models,
SARMA).4 As pointed out by Granger (1979), since seasonality
can contribute substantially to the variance of a series while being
economically unimportant, its absence helps to build more parsimonious
models with a better fit. More recently, Capistrán, Constandse, and
Ramos-Francia (2010) estimated the portion of the Mexican CPI
components that was due to seasonality, finding that it can account
for almost 80% of the observed variation. Bell and Sotiris (2010) also
showed that certain fundamental choices about seasonal adjustment
af fect subsequent forecasts by improving their accuracy. I study the
impact of the Easter ef fect—a special feature of seasonal components—
4. The version of X12-ARIMA used in this paper is 0.2.7. A detailed description of the program can be
found in Findley et al. (1998) and U.S. Census Bureau (2007).
136
LATIN AMERICAN JOURNAL OF ECONOMICS | Vol. 50 No. 1 (May, 2013), 133–161
on RMSFE in two ways: (i) including the regressors proposed by Bell
and Hillmer (1983) in an ARMA environment, and (ii) isolating the
seasonal adjustment process prior to modeling, by first including and
later excluding the Easter ef fect using X12-ARIMA. The fundamental
reason to make a statement about the out-of-sample impact of the
Easter ef fect, particularly in the Chilean GDP series, lies in the findings
of Cobb and Medel (2010) that its exclusion can generate seasonally
adjusted series with very dif ferent dynamics from those that include
it, and because it is information that is known with high confidence
several quarters after its realization. Both elements are subjects of
interest to an out-of-sample evaluation, and also make the results
uncomparable between the dif ferent kinds of data.
After the estimation of more than 20 million equations, the results
show the following. On average, with actual series, the information
criteria that show lower RMSFE across all horizons are AIC and
BIC. While AIC forecasts better 1 and 2 steps ahead, BIC does so at
the remaining horizons. In the case of seasonally adjusted data, BIC
shows better performance 1 and 4 steps ahead, while HQ is the best
at the intermediate horizons. In all cases, with and without seasonal
adjustment, the dif ferences are not statistically significant.
There is a great deal of literature related to issues raised in this paper,
especially on ARMA (unstructured) modeling, conjunctural forecasting
(short-term and multi-horizon), seasonal adjustment, Easter-ef fect
treatment, and information criteria for model-building.5 Nevertheless,
this work follows closely in the spirit of Granger and Jeon (2004)
by attempting to reveal the empirical out-of-sample performance of
in-sample information criteria using actual data, and in doing so to
determine if the behavior is slightly dif ferent. Stock and Watson (2007)
also studied the performance of many forecasting techniques with 215
macro series from the United States, using ARMA models chosen with
information criteria and comparing those with nonlinear models. The
results were mixed: Seasonality was found to favor nonlinear models
while both AIC and BIC performed better, depending on the type of
series employed.6
5. A complete recent survey of ARMA modeling and its variants was provided by Holan, Lund, and
Davis (2010).
6. A topic not treated in this work is in regard to the dif ferent asymptotic properties of each information
criterion used in this work. It is important to note that while BIC and HQ are consistent, AIC is not.
In short, this implies that as the sample size increases, AIC may change its selected model, converging
to the true model. See Nishii (1984) and Canova (2007) for details.
C.A. Medel | How Informative are In-sample Information Criteria to Forecasting?
137
For the case of Chile, no similar systematic analysis exists on the
relative ef ficiency of information criteria using the same dataset.
A related work is Medel and Urrutia (2010), which evaluated the
forecasting procedure contained in the X12-ARIMA, but using another
criterion (Ljung-Box Q-statistic). The advantage of this automatic
procedure is that it filters for outliers and the Easter ef fect, reducing
the variance of the actual series. In this work I also try this procedure
to model the seasonally adjusted data, but this filtering makes the
results incomparable with unfiltered series.
The paper is organized as follows. In Section 2 I describe the Chilean
GDP dataset, the transformations applied to achieve stationarity, the
aggregations, and my treatment of seasonality and the Easter ef fect. In
Section 3 I explain the setting for model estimation and in Section 4
I analyze the results for each series by providing a recommendation
regarding the information criterion that is most accurate for each of
the four horizons considered and for the two kinds of data, based on
the minimization of RMSFE. Finally, Section 5 concludes.
2.
Data
I use the Central Bank of Chile’s Quarterly National Accounts (QNA)
starting with GDP as the most aggregated, and with three levels of
disaggregation on the demand and supply sides. The original series are
in levels, denominated in millions of 2003 Chilean pesos. I use the first
release of the QNA that includes 2010.IV, leaving a real-time analysis for
further research.7 The initial estimation sample covers 1986.I to 1995.IV
(40 observations), and the size of the rolling window is kept fixed. The
remaining sample for evaluation covers 1996.I to 2010.III (59 observations).
The series compound the Chilean GDP by demand and supply side.
A scheme of demand-side aggregations of all series and the acronyms
used in this paper are shown in Table 1, and those of supply-side in
Table 2. There is a total of five dif ferent aggregations plus the GDP
by itself (“aggregation 6”). All five aggregation blocks as well as
aggregation 6 deliver a forecast of GDP, calculated as a weighted sum
of the forecasts of every corresponding component. The base year for
calculating each share is 2003.8
7. A framework for a real-time approach that can be used to this end was provided by Clements and
Galvão (2010).
8. Table A1 of the appendix displays the shares of every sector on the real GDP at 2003 prices.
g
Changes in inventories (*)
Government consumption expenditure
Exports of goods
Exports of services
Imports of goods (**)
Imports of services (**)
ci
g
xg
xs
mg
ms
ed External demand
(x - m)
Investment (meq + cw)
Exports (xg + xs)
Government consumption expenditure (g)
Changes in inventories (*)
id Internal demand
(c + i + g)
Source: Central Bank of Chile.
(*) Not considered in analysis. (**) Imports are subtracted.
Aggregation 3
Household consumption expenditure
(cn + cd)
Aggregation 2
m Imports (**) (mg + ms)
x
ci
Construction and works
i
cw
Household consumption expenditure:
durables
cd
c
meq Machinery and equipment
Household consumption expenditure:
nondurables
cn
Aggregation 1
gdp=id+ed=c+i+g+(x-m)=(cn+cd)+(meq+cw+ci)+g+(xg+xs-mg-ms)
Table 1. Chilean GDP in levels by demand side
gdp Gross domestic
product (id + ed)
Aggregation 6
138
LATIN AMERICAN JOURNAL OF ECONOMICS | Vol. 50 No. 1 (May, 2013), 133–161
GDP Natural resources (egw + caf + min) egw
Aggregation 4
others
Other sectors (-dut + vat + cif)
Wholesale and retail trade, hotels and restaurants
Construction
Agriculture and forestry
Transportation and communications
Financial intermediation and business services
Personal and social services
Owner-occupied dwellings
Public administration
Duties + taxes on goods and services (*)
Non-deductible VAT
Imports CIF
con
agr
tra
fin
per
ood
pub
dut
vat
cif
man Manufacturing
Mining
com
Capture fishery
Electricity, gas and water
Aggregation 5
min
gdp nnr GDP Non-natural resources (com + man caf
+ con + agr + tra + fin + per + ood + pub)
Gross domestic product
gdp nr
(gdp nr + gdp nnr + others)
(*) DUT are subtracted. Source: Central Bank of Chile.
gdp
Aggregation 6
gdp=gdp nr+gdp nnr+others=(egw+caf+min)+(com+man+con+agr+tra+fin+per+ood+pub)+(vat+cif-dut)
Table 2. Chilean GDP in levels by supply side
C.A. Medel | How Informative are In-sample Information Criteria to Forecasting?
139
140
LATIN AMERICAN JOURNAL OF ECONOMICS | Vol. 50 No. 1 (May, 2013), 133–161
To achieve stationarity I consider the following five transformations:9
(i - iv ) : ∆d yt = ∆d  log(Yt ) - log(Yt -1 )  , d = { 0,..., 3 } ,
(v ) : yt = (Yt / Yt -4 ) ⋅ 100 - 100,
(1)
where Yt is the variable in levels. Hereafter, these transformations
are denominated d1, d2, d3, d4, and %. All of these are stationary
transformations of the series in levels. By looking at these expressions,
econometricians may be apprehensive about the overdif ferencing
issue. But, this is of minor concern since overdif ferencing is mainly
a hazard in testing economic theory rather than forecasting (Dickey
and Pantula, 1987).
Besides using actual data, the complete exercise is carried out with two
kinds of seasonally adjusted data: (i) with special regressors to control for
seasonality (SARMA models), and (ii) with the X12 ARIMA procedure.
In both cases the Easter ef fect is considered but in dif ferent ways.
This ef fect relates to the fact that the composition of a month or
quarter in terms of number of working days af fects the dynamics of a
series. Typical examples, in the Chilean case, are: retail in September
(September 18 is Chilean Independence Day) and December (Christmas
and New Year), manufacturing in the summer holiday month of
February, and consumption in March (when Chileans must make
several one-time annual payments, such as car registration). The Easter
ef fect is considered exogenous as are the variables in ARMAX and
SARMAX specifications when the dependent variable is not seasonally
adjusted.10 The row vector Xt contains a set of six series, defined as
the number of working days within a quarter minus the number of
Sundays within the same period, Dayt = {#Dayt - #Sundayt}. Bell
and Hillmer’s (1983) regressors are thus:
Xt =  Mondayt

...
Saturdayt  .

(2)
The other case with seasonally adjusted data consists of comparisons
including and excluding the use of X12-ARIMA, such that the impact
of the Easter ef fect on forecast performance can be isolated. As
9. Table B1 of the appendix shows the typical statistics of all these transformations for all series (panel  1:
demand side; panel 2: supply side).
10. In this context, exogenous means that there is new information being used to model the dependent
variable, instead of the case in which it is considered within the seasonal adjustment with X12-ARIMA.
In that case it is considered endogenous.
C.A. Medel | How Informative are In-sample Information Criteria to Forecasting?
141
X12-ARIMA changes the autocorrelation structure of the series, the
results when using this treatment are not comparable with those with
regressors on actual series.
The treatment of the Easter ef fect by X12-ARIMA is as follows.11 First,
the program is an automatic procedure based on moving averages to
seasonally adjust economic time series—that is, a separation of a series
into a trend-cycle component (TCt), a seasonal component (St), and
an irregular component (It). This decomposition is typically additive
or multiplicative, depending on It (whether it is really irregular) which
is based on a battery of tests contained in the routine. So, assuming
an additive decomposition, the series can be split as:
Yt = TCt + St + It,
(3)
where the seasonally adjusted series corresponds to Yt - St. Second,
before the application of filters (moving averages) that identify the
abovementioned components, X12-ARIMA applies a whitening based
on a special module: regARIMA (regression with ARIMA noise), which
automatically filters for the Easter ef fect, leap years, level shifts, additive
outliers, transitory changes and ramp, among other considerations.12
By doing so, the whitening series added to the seasonally-adjusted
series is Z^t , obtained from:
Yt =
∑ bi tC i t
i
+ Zt ,
Z t = Yt - ∑ b i tC i t ,
(4)
i
where Ci|t is the control i, and b^i|t is the estimated coef ficient associated
to the control i. This implies that the predicted series is not the actual
one, but rather an automatic filtered version. Third, the regressors
applied by X12-ARIMA to control for the Easter ef fect consist of the
same Bell and Hillmer regressors used in ARMAX and SARMAX
specifications in this work. After identification of all ef fects, the share
excluded by filtering is added to the irregular component to preserve
the equality Yt = TCt + St + It.
11. A complete description can be found in U.S. Census Bureau (2007).
12. See Table 4.1, pp. 36-38, of U.S. Census Bureau (2007) for formal definitions.
142
3.
LATIN AMERICAN JOURNAL OF ECONOMICS | Vol. 50 No. 1 (May, 2013), 133–161
Setting up the estimations
3.1.Models
I estimate six families of models (with intercepts) for each of the five
stationarity transformations mentioned above:
Actual series
2
(p, q ) ⊆ { 0, 1, 2, 3, 4 } ,
(i ) : ARMAX(p, q )
(ii ) : SARMAX(s, p, q )
(iii ) : ARMA(p, q )
2
(p, q ) ⊆ { 0, 1, 2, 3, 4 } , s = { 4 } ,
2
(p, q ) ⊆ { 0, 1, 2, 3, 4 } ,
(iv ) : SARMA(s, p, q )
(5)
2
(p, q ) ⊆ { 0, 1, 2, 3, 4 } , s = { 4 } ,
Seasonally-adjusted series:
2
(v ): ARMA(p, q ) (p, q ) ⊆ { 0, 1, 2, 3, 4 }
(X 12 - ARIMA sa with Easter effect),
2
(vi ): ARMA(p, q ) (p, q ) ⊆ { 0, 1, 2, 3, 4 }
(X 12 - ARIMA sa without Easter effect).
(6)
I consider the fourth order for frequency considerations; hence, monthly
cases are a bit more complicated to compute.13 All the estimations
were programmed using the ARIMASel add-in for Eviews 7.1. Each
model is re-estimated as the rolling window is moved one observation
ahead. Each family of models produces a total of 25 (without seasonal
regressors) or 36 (with seasonal regressors) models by combining
non-skipped AR(p), MA(q), SAR(s) and SMA(s) regressors with
(p,q) ⊆ {0,1,2,3,4}2 and s = {4}.
The three information criteria used are as follows:
AIC : –2(ℓ/T ) + 2(k/T ),
BIC : –2(ℓ/T ) + k log (T )/T,
(7)
HQ : –2(ℓ/T ) + 2k log(T )/T,
13. Coincidentally, Granger and Jeon (2004) find that an AR(4) model re-estimated when a new monthly
observation has been added to the sample outperforms those chosen by AIC and BIC.
C.A. Medel | How Informative are In-sample Information Criteria to Forecasting?
143
where ℓ is the value of the log-likelihood function, with k parameters
estimated using T observations. From an empirical viewpoint, the
selection of a model depends strictly on how the lag structure in the
model fits the series, i.e., how systematic the behavior of the series is
across the sample span.
Accounting for five transformations, six specifications, 34 variables,
four horizons, three criteria, 28.6 averaged combinations of AR, MA,
SAR, SMA regressors, 59 observations within the evaluation window,
the number of estimated equations raises:
5 × 6 × 34 × 4 × 3 × 28.6 × 59 = 20,653,776 equations.
The number of estimations carried out is slightly lower because in a few
cases the value of the log-likelihood function could not be computed.
3.2.Forecast evaluation
The evaluation of forecasts is based on comparisons of root mean
squared forecast error (RMSFE), defined as:
1
 1 T (h )
2
2
ˆ
(
RMSFEh = 
y
y
)
∑ t +h t +h t  ,
 T (h ) t =1

(8)
where y^t + h is the forecast of yt + h (with seasonal correction when
appropriate), h-quarters ahead (h = {1,2,3,4}, and T(h) is the size of the
evaluation sample (from 1996.I to 2010.III, and 59 observations for onestep-ahead forecasts). Notwithstanding the transformations, the results
are shown and the conclusions made with year-on-year variation of the
series in levels. The use of the mean absolute percentage error (MAPE)
statistic could be reasonable at this point to avoid homogenization of
results and compare it directly, as MAPE is less sensitive to measuring
units. But MAPE is more suitable when comparing forecasts of two
or more dif ferent target series denominated in dif ferent units rather
than the same predicted variable as in this paper.
I perform the Giacomini and White (2006) test of equal predictive
ability to determine if the best model chosen with each information
criterion is statistically better than its competitor. As the test is
designed for pairwise comparisons, the strategy consists of comparing
the best result (lower RMSFE) achieved with (i) AIC versus the BIC,
144
LATIN AMERICAN JOURNAL OF ECONOMICS | Vol. 50 No. 1 (May, 2013), 133–161
(ii) AIC versus HQ, and (iii) finally, BIC versus HQ. Thus, the null
(NH) and alternative hypotheses (AH) of the test are:
NH : (dijh ) = 0,
(9)
AH : (dijh ) ≠ 0,
where dihj is the dif ference of squared forecast error between the best
result achieved with the criterion i versus the best result achieved
with the criterion j, across all forecasts for each horizon h:
dijh = (yt +h - yˆti +h t )2 - (yt +h - yˆtj+h t )2 .
(10)
The null hypothesis is rejected at a significance level of a% if the
calculated statistic (GWh):
GW h =
dith
σˆd h / T (h )
,
(11)
it
is bigger than δ , where δ is a value obtained from the percentile a%/2
of a normal standard distribution. Note that σˆdijh is a heteroskedasticity
and autocorrelation-corrected estimation of the standard deviation
of dihj (Newey and West, 1987).
4.
Results
This section shows the results of the most accurate forecast achieved
by each information criterion, given its type of model, aggregation,
and stationary transformation, for both kinds of data. It is worth
mentioning that the results obtained are fully computer-based,
implying that extreme forecasts are not trimmed by the judgment of
the forecaster.14 Thus, the RMSFE results can be easily improved,
avoiding such “crazy” forecasts by allowing for a “no change” forecast
(yt+h|t = yt) given a threshold, as used by Stock and Watson (2007). I
opt not to intervene to ensure a fairer comparison of the information
criteria, and because an intervention could result in important ef fects
on accuracy and inference as highlighted by Mélard and Pasteels (2000).
14. See Goodwin, Önkal, and Lawrence (2011) for a formal treatment of the role of the forecaster’s
judgment in forecasting practice.
C.A. Medel | How Informative are In-sample Information Criteria to Forecasting?
145
There are two kinds of results: with and without seasonal adjustment,
to allow for fair comparison of the RMSFE for the same dependent
variable. I also report some results on models, aggregations, and
stationary transformations as special topics.
4.1.Results of actual series
Tables 3 to 6 show the RMSFE estimates for h = 1 to 4 obtained with
actual series. Note that some cells of the tables show a value of “>100”
implying the presence of outliers in sector forecasts or in the GDP
itself (Aggregation 6). These outliers are due to data and/or numerical
misestimation of the log-likelihood function. As X12-ARIMA filters
the series for outliers, among other anomalies, this kind of result is
not present with seasonally adjusted data.
From Table 3, we see that for h = 1 the best results are obtained with
AIC, achieving a RMSFE of 1.87, followed closely by BIC with 1.90,
and finally HQ with 1.92. The same qualitative result holds true for
h = 2. Table 4 show that the minimum RMSFE is 1.81, followed by
1.87, and finally 1.89, in the same order: AIC, BIC, and HQ. For
h = 3 there is virtually a tie between the BIC and AIC, showing an
RMSFE of 1.75 and 1.76, respectively. Lastly, HQ shows a RMSFE of
1.93. For one-year ahead the same situation occurs as for h = 3: BIC
and AIC show an RMSFE of 1.50 and 1.51, respectively. Finally, HQ
achieves an 1.63 in RMSFE.
As shown in Table 11 (upper panel), the results of the Giacomini and
White (2006) test reveal that there are no significant statistical dif ferences
among the results of each horizon for the three information criteria. But
despite this finding and based purely on RMSFE, the recommendation is
that AIC be used for short horizons and BIC for intermediate horizons
shorter than one year when predicting actual series.
4.2. Results of seasonally adjusted series
Tables 7 to 10 show the RMSFE estimates for h = 1 to 4 obtained with
the seasonally adjusted series. The only dif ference from the other series,
as previously mentioned, is the treatment of the Easter ef fect. The
upper panels of Tables 7 through 10 show the results considering the
row vector X of working days and holidays within the adjusting process
(“With E-e ”). The lower panel, conversely, does not control for working
days and only accounts for the remaining anomalies listed in Section 2.
1
2
3
4
5
6
1
2
3
4
5
6
1
2
3
4
5
6
SARMAX
ARMA
SARMA
>100
>100
>100
2.70
>100
2.54
3.80
4.39
5.88
2.08
6.69
2.39
>100
>100
>100
3.26
>100
8.86
5.05
23.71
19.51
2.06
8.15
3.30
d1
>100
33.89
>100
3.45
>100
48.16
3.06
3.06
3.35
1.92
5.94
2.47
>100
>100
>100
>100
>100
>100
4.02
4.81
4.03
2.03
6.10
1.98
d2
>100
6.81
11.63
6.10
10.98
6.04
3.37
3.82
2.82
2.30
4.93
2.05
>100
>100
>100
4.35
>100
5.07
>100
4.01
3.12
2.05
5.44
1.87
d3
AIC
>100
10.83
6.93
20.84
>100
3.82
4.91
5.17
4.36
2.99
7.21
2.86
>100
7.47
18.28
3.16
>100
3.68
4.90
6.14
6.08
3.20
6.41
2.71
d4
>100
>100
>100
>100
>100
56.74
8.01
7.59
8.70
4.01
5.96
4.56
>100
>100
>100
17.61
>100
>100
9.65
9.43
11.52
5.24
8.73
6.09
%
>100
>100
>100
2.33
>100
2.09
3.87
3.52
3.40
2.35
6.72
2.44
>100
>100
>100
2.82
>100
7.26
4.64
4.76
4.00
2.52
7.55
3.43
d1
>100
20.90
>100
4.03
>100
48.14
3.09
2.80
3.41
1.98
5.93
2.58
>100
>100
>100
>100
>100
>100
3.63
4.67
4.43
2.04
6.12
2.35
d2
8.26
4.52
9.70
3.64
8.53
4.25
3.57
3.56
2.79
1.93
5.25
2.36
5.80
4.53
>100
4.05
>100
4.25
3.61
3.82
3.39
2.02
5.32
1.90
d3
BIC
5.68
5.89
5.86
2.95
6.53
2.63
d4
4.80
9.71
7.66
2.95
>100
3.34
5.05
4.67
4.11
2.78
6.68
2.77
>100
6.33
8.21
2.94
>100
3.08
Source: Author's computations.
Shaded cells indicate the minimum RMSFE within a model specification and information criterion.
1
2
3
4
5
6
ARMAX
Aggr.
Table 3.RMSFE estimates for h = 1
>100
>100
>100
>100
>100
>100
6.59
6.40
8.21
3.91
5.74
4.79
>100
>100
>100
28.54
>100
>100
9.92
8.87
9.36
5.22
8.67
5.95
%
>100
>100
>100
2.61
>100
2.21
3.95
4.30
5.20
2.18
6.88
2.28
>100
>100
>100
3.14
>100
7.34
5.13
7.67
18.10
2.37
8.13
3.52
d1
>100
20.73
>100
3.53
>100
48.14
3.01
2.85
3.13
1.96
6.04
2.50
>100
>100
>100
>100
>100
>100
3.95
4.70
4.11
2.08
6.11
2.24
d2
>100
7.20
11.56
4.55
8.38
4.27
3.41
4.16
2.78
2.15
5.03
2.21
>100
>100
>100
4.41
>100
5.02
>100
4.04
3.11
2.06
5.38
1.92
d3
HQ
6.16
10.67
6.80
3.49
>100
3.61
5.03
4.82
4.44
3.03
7.13
2.79
>100
6.23
17.78
2.98
>100
3.46
5.04
6.02
5.74
3.13
6.19
2.74
d4
>100
>100
>100
>100
>100
56.49
7.70
7.26
8.45
3.99
5.68
4.60
>100
>100
>100
18.08
>100
>100
9.78
9.07
10.39
5.07
9.08
6.06
%
146
LATIN AMERICAN JOURNAL OF ECONOMICS | Vol. 50 No. 1 (May, 2013), 133–161
1
2
3
4
5
6
1
2
3
4
5
6
1
2
3
4
5
6
SARMAX
ARMA
SARMA
d1
>100
>100
6.25
2.85
>100
2.53
3.93
3.65
6.78
2.16
6.27
2.08
>100
>100
>100
>100
>100
4.57
5.94
7.77
6.87
2.27
8.05
3.48
d2
>100
>100
>100
>100
7.80
2.76
3.09
3.23
3.62
1.99
6.20
2.62
>100
6.29
5.70
>100
>100
>100
4.36
5.12
4.15
1.91
6.13
2.12
d3
21.71
6.37
11.76
6.41
9.82
5.89
3.10
3.91
3.29
2.20
4.84
1.98
>100
>100
>100
4.67
>100
4.61
>100
3.82
3.30
2.19
>100
1.81
d4
>100
>100
>100
3.15
>100
3.30
3.78
5.41
4.55
2.85
5.58
2.61
>100
>100
69.52
2.55
>100
2.72
4.56
5.65
6.06
2.87
6.24
2.80
%
>100
>100
>100
>100
>100
65.77
8.36
9.06
11.40
3.52
5.80
3.88
>100
>100
>100
21.35
>100
>100
8.41
10.14
13.38
4.67
8.78
5.02
d1
>100
>100
>100
2.59
>100
2.15
4.21
3.69
3.69
2.31
6.65
2.20
>100
>100
>100
>100
>100
3.35
5.57
6.95
4.21
2.62
7.90
3.75
d2
>100
>100
>100
>100
7.11
2.87
3.12
2.85
3.52
1.97
6.34
2.79
>100
5.75
4.88
>100
>100
>100
4.15
5.20
4.26
2.03
6.35
2.59
4.94
5.45
9.82
3.80
7.44
4.06
3.35
3.32
3.28
2.04
4.90
1.99
10.71
4.13
>100
4.01
3.35
3.32
4.06
3.78
3.11
2.22
>100
1.87
d3
BIC
d4
5.04
5.62
6.01
2.82
6.08
2.71
4.48
7.04
>100
2.66
7.91
2.86
3.93
4.30
3.69
2.85
6.00
2.69
>100
6.38
52.95
2.63
>100
2.44
Source: Author's computations.
Shaded cells indicate the minimum RMSFE within a model specification and information criterion.
1
2
3
4
5
6
Aggr.
ARMAX
AIC
Table 4.RMSFE estimates for h = 2
%
>100
>100
>100
>100
>100
64.90
6.48
7.75
9.57
3.38
5.71
4.27
>100
>100
>100
33.40
>100
>100
7.71
9.36
11.16
4.84
8.61
5.16
d1
>100
>100
>100
3.01
>100
2.28
4.25
4.39
5.93
2.25
6.82
2.11
>100
>100
>100
>100
>100
3.45
4.90
8.00
5.76
2.44
8.33
3.74
d2
>100
>100
>100
>100
7.14
2.73
3.08
2.92
3.64
1.95
6.25
2.75
>100
6.54
5.06
>100
>100
>100
4.38
5.03
4.10
1.89
6.18
2.56
21.49
6.47
10.79
4.99
8.02
4.04
3.07
3.95
3.46
2.22
5.00
2.21
>100
>100
>100
4.36
>100
4.64
>100
3.76
3.27
2.19
>100
1.92
d3
HQ
d4
>100
>100
>100
2.98
7.83
3.15
3.81
4.88
4.24
2.81
5.89
2.69
>100
>100
62.43
2.62
>100
2.81
4.59
5.67
6.15
2.85
6.47
2.77
%
>100
>100
>100
>100
>100
65.42
7.51
8.77
10.80
3.49
5.40
3.94
>100
>100
>100
21.64
>100
>100
8.22
9.55
11.78
4.80
8.76
5.03
C.A. Medel | How Informative are In-sample Information Criteria to Forecasting?
147
1
2
3
4
5
6
1
2
3
4
5
6
1
2
3
4
5
6
SARMAX
ARMA
SARMA
>100
>100
>100
2.81
>100
>100
3.98
3.92
5.48
2.06
5.93
2.06
>100
>100
>100
>100
>100
5.63
5.17
23.88
39.09
2.20
7.58
3.21
d1
>100
>100
>100
1.85
>100
14.71
2.83
3.22
3.35
1.76
5.73
2.42
>100
>100
>100
>100
>100
>100
4.11
4.85
3.99
2.08
5.35
2.21
d2
>100
>100
>100
9.09
>100
6.63
2.89
4.00
3.08
2.44
4.62
2.09
>100
>100
>100
4.97
>100
4.93
>100
4.66
3.47
2.07
4.66
1.94
d3
AIC
>100
7.91
6.90
2.96
>100
2.78
4.15
5.17
4.84
2.66
6.45
2.57
>100
>100
>100
2.61
>100
2.85
4.77
5.52
4.80
2.93
6.02
2.97
d4
>100
>100
>100
>100
>100
75.89
7.58
8.64
9.40
3.79
5.33
3.82
>100
>100
>100
42.29
>100
6.89
8.23
8.38
8.73
4.21
7.70
4.51
%
>100
>100
>100
2.28
>100
2.45
4.22
3.37
3.06
2.19
6.40
2.09
>100
>100
>100
>100
>100
4.22
4.32
5.22
3.75
2.45
6.84
3.45
d1
>100
>100
>100
1.88
>100
14.57
2.92
2.80
3.49
1.85
6.02
2.54
>100
>100
>100
>100
>100
>100
3.83
4.56
4.20
1.98
5.47
2.62
d2
>100
>100
>100
7.69
>100
6.66
2.52
3.22
3.17
2.13
4.53
2.15
8.50
>100
>100
4.67
>100
4.15
3.79
4.42
3.40
2.02
4.67
1.75
d3
BIC
3.99
4.35
3.43
2.51
6.12
2.29
>100
6.97
>100
2.38
>100
2.22
5.35
5.03
4.62
2.83
5.98
2.73
d4
>100
8.36
7.72
2.59
>100
2.66
Source: Author's computations.
Shaded cells indicate the minimum RMSFE within a model specification and information criterion.
1
2
3
4
5
6
ARMAX
Aggr.
Table 5.RMSFE estimates for h = 3
>100
>100
>100
>100
>100
76.87
5.83
7.42
7.58
3.56
5.73
4.33
>100
>100
>100
55.17
>100
5.43
7.07
7.38
7.93
4.16
6.50
4.57
%
>100
>100
>100
2.74
>100
2.66
3.94
3.74
5.01
2.16
6.37
1.96
>100
>100
>100
>100
>100
4.50
5.18
7.48
36.68
2.40
7.69
3.43
d1
>100
>100
>100
1.96
>100
14.56
2.86
2.84
3.38
1.83
5.78
2.52
>100
>100
>100
>100
>100
>100
4.13
4.55
4.20
1.93
5.43
2.64
d2
>100
>100
>100
8.65
>100
6.30
2.62
3.75
2.85
2.38
4.55
2.43
>100
>100
>100
4.69
>100
4.95
>100
4.54
3.49
1.99
4.56
1.95
d3
HQ
>100
7.87
7.03
2.90
>100
2.71
3.90
4.64
4.22
2.70
6.46
2.53
>100
>100
>100
2.38
>100
2.79
4.58
5.28
4.60
2.84
5.77
2.90
d4
>100
>100
>100
>100
>100
75.65
7.22
8.49
8.60
3.76
5.24
4.00
>100
>100
>100
42.06
>100
6.16
7.30
8.48
8.55
4.22
6.77
4.48
%
148
LATIN AMERICAN JOURNAL OF ECONOMICS | Vol. 50 No. 1 (May, 2013), 133–161
1
2
3
4
5
6
1
2
3
4
5
6
1
2
3
4
5
6
SARMAX
ARMA
SARMA
>100
>100
>100
2.44
>100
1.99
3.46
3.88
6.14
2.20
5.92
1.72
>100
>100
>100
6.54
>100
11.45
4.90
6.35
6.93
2.09
7.67
3.09
>100
>100
>100
>100
>100
>100
2.49
2.86
3.44
1.57
5.84
2.38
>100
>100
>100
>100
>100
7.21
3.87
4.89
4.01
1.90
5.74
2.07
d2
26.66
5.22
11.81
6.63
9.37
6.37
2.82
3.75
3.34
1.92
4.31
1.72
>100
>100
>100
5.03
>100
4.66
>100
3.82
3.11
1.78
>100
1.51
d3
AIC
>100
>100
7.02
3.02
7.70
3.23
4.01
5.57
5.33
2.91
6.41
2.56
>100
>100
49.49
2.49
>100
2.49
4.28
6.05
5.82
2.66
6.35
2.41
d4
>100
>100
>100
>100
>100
87.51
7.98
8.01
8.57
4.09
5.02
4.37
>100
>100
>100
51.47
>100
6.91
7.46
8.64
10.38
5.23
7.27
4.71
%
>100
>100
>100
2.55
>100
2.09
3.83
3.17
3.39
2.17
6.11
1.96
>100
>100
>100
5.84
>100
10.25
4.54
6.65
4.03
2.30
7.10
3.37
d1
>100
>100
>100
>100
>100
>100
2.55
2.36
3.54
1.67
5.77
2.46
>100
>100
>100
>100
>100
>100
3.51
4.65
4.04
1.71
5.74
2.46
d2
4.36
9.06
14.58
3.75
7.30
4.13
2.75
2.94
2.90
1.89
4.61
2.00
5.01
4.95
>100
4.30
>100
3.77
3.36
3.28
2.63
1.68
>100
1.50
d3
BIC
Source: Author's computations.
Shaded cells indicate the minimum RMSFE within a model specification and information criterion.
1
2
3
4
5
6
ARMAX
d1
RMSFE estimates for h = 4
Aggr.
Table 6.
3.66
6.57
7.69
2.49
7.83
2.78
3.80
4.29
4.52
2.81
6.84
2.66
>100
6.44
37.94
2.47
>100
2.32
5.31
5.45
5.68
2.55
6.36
2.18
d4
>100
>100
>100
>100
>100
87.07
6.23
6.60
7.38
3.97
5.15
4.78
>100
>100
>100
67.89
>100
6.23
6.47
8.47
7.83
4.90
7.07
4.99
%
>100
>100
>100
2.47
>100
1.99
3.29
3.96
5.21
2.06
6.26
1.95
>100
>100
>100
6.07
>100
10.38
4.89
7.09
5.76
2.33
7.61
3.34
d1
>100
>100
>100
>100
>100
>100
2.57
2.59
3.36
1.64
5.88
2.48
>100
>100
>100
>100
>100
7.08
3.86
4.54
3.99
1.90
5.67
2.47
d2
26.27
5.05
11.41
4.87
7.50
4.12
2.62
3.62
2.91
1.87
4.31
1.96
>100
>100
>100
4.63
>100
4.70
>100
3.54
3.03
1.76
>100
1.63
d3
HQ
>100
>100
7.58
2.65
7.42
2.93
3.98
4.88
4.92
2.97
6.68
2.63
27.07
6.13
48.93
2.53
>100
2.32
4.36
5.79
5.63
2.67
6.24
2.44
d4
>100
>100
>100
>100
>100
87.06
7.89
8.24
8.27
4.14
4.98
4.54
>100
>100
>100
51.66
>100
6.28
7.62
7.85
8.92
5.19
7.54
4.76
%
C.A. Medel | How Informative are In-sample Information Criteria to Forecasting?
149
2.46
1.50
5.76
1.72
3
4
5
6
2.93
2.86
1
5.46
1.26
5
6
2
2.96
1.30
3
4
2.91
2.98
1
2
d1
1.38
5.30
1.37
2.14
2.58
2.76
1.19
5.14
1.30
1.84
2.55
3.02
d2
1.45
4.17
2.14
2.49
2.96
2.88
1.19
3.97
1.24
2.41
2.73
2.94
d3
AIC
2.08
5.13
3.56
5.16
6.09
3.75
2.78
4.70
1.52
3.12
3.89
3.54
d4
2.89
4.01
2.87
6.75
5.45
3.78
2.64
3.73
2.47
5.77
4.27
4.18
%
1.77
6.13
1.54
2.26
3.03
2.83
1.40
5.58
1.18
2.16
2.70
3.06
d1
1.43
5.52
1.37
1.69
2.55
2.85
1.27
5.08
1.24
1.80
2.47
2.72
d2
1.36
4.12
1.46
2.66
3.05
2.95
1.13
3.83
1.18
2.20
2.79
3.00
d3
BIC
Source: Author's computations.
Shaded cells indicate the minimum RMSFE within a model specification and information criterion.
Without E-e
With E-e
Aggr.
Table 7.RMSFE estimates for h = 1
2.08
5.02
2.06
4.85
5.95
3.96
1.92
4.68
1.57
2.38
3.41
3.41
d4
3.23
4.03
2.87
4.94
3.93
2.87
2.81
3.79
2.43
4.55
4.10
3.69
%
1.79
5.99
1.57
2.43
3.04
2.97
1.29
5.60
1.22
2.61
2.96
2.96
d1
1.30
5.40
1.29
2.07
2.65
2.83
1.23
5.16
1.20
2.15
2.54
2.83
d2
1.38
4.35
1.72
2.26
2.93
2.68
1.20
3.80
1.17
2.30
2.79
3.04
d3
HQ
1.95
4.70
3.14
6.53
7.37
3.68
2.84
4.53
1.58
2.85
3.74
3.43
d4
2.95
3.83
2.84
6.18
4.49
3.34
2.73
3.63
2.39
5.00
4.23
4.18
%
150
LATIN AMERICAN JOURNAL OF ECONOMICS | Vol. 50 No. 1 (May, 2013), 133–161
2.38
1.33
5.38
1.70
3
4
5
6
2.99
3.06
1
5.01
1.42
5
6
2
2.70
1.25
3
4
3.13
2.89
1
2
d1
1.47
5.53
1.31
2.01
2.70
2.96
1.20
4.99
1.31
1.83
2.54
3.18
d2
1.34
4.10
2.25
2.78
2.94
2.99
1.30
3.87
1.18
2.04
2.59
2.87
d3
AIC
2.08
5.01
3.56
5.17
5.40
3.64
2.83
4.70
1.51
3.19
3.56
3.21
d4
3.11
3.99
3.31
7.00
5.62
3.93
2.56
3.42
2.36
6.21
5.09
3.95
%
1.71
5.21
1.45
2.24
2.79
2.87
1.29
5.30
1.24
2.29
2.78
3.20
d1
1.37
5.65
1.36
1.84
2.47
2.88
1.34
5.09
1.28
1.75
2.35
2.74
d2
1.46
4.35
1.56
2.64
3.09
2.70
1.31
3.81
1.27
2.20
2.48
2.69
d3
BIC
Source: Author's computations.
Shaded cells indicate the minimum RMSFE within a model specification and information criterion.
Without E-e
With E-e
Aggr.
Table 8.RMSFE estimates for h = 2
1.94
4.84
2.19
4.63
5.29
3.43
1.88
4.50
1.60
2.60
3.01
2.90
d4
3.28
4.15
3.02
4.60
4.04
3.35
2.63
3.61
2.33
5.43
4.68
3.23
%
1.67
5.32
1.40
2.40
3.05
2.97
1.36
5.10
1.11
2.32
2.97
3.15
d1
1.35
5.58
1.25
2.10
2.60
2.88
1.25
5.04
1.25
1.88
2.59
3.00
d2
1.46
4.30
1.82
2.70
3.00
2.80
1.34
3.91
1.25
2.17
2.52
2.87
d3
HQ
2.01
4.99
3.11
5.95
6.44
3.48
2.81
4.80
1.53
2.84
3.28
3.05
d4
3.00
4.13
3.17
6.34
4.44
3.58
2.68
3.39
2.38
5.86
4.847
3.80
%
C.A. Medel | How Informative are In-sample Information Criteria to Forecasting?
151
2.25
1.49
5.05
1.77
3
4
5
6
2.87
2.97
1
4.77
1.24
5
6
2
2.58
1.22
3
4
2.80
2.56
1
2
d1
1.58
5.26
1.38
2.00
2.61
2.54
1.16
5.00
1.24
1.76
2.49
2.76
d2
1.46
4.09
2.16
2.18
2.49
3.11
1.23
3.96
1.23
2.22
2.86
3.09
d3
2.37
5.04
2.97
5.26
5.66
3.39
2.71
5.18
1.91
3.22
3.28
3.45
d4
2.69
3.67
3.17
6.80
5.23
4.54
2.33
3.37
2.20
6.45
4.75
3.90
%
1.79
5.22
1.39
1.97
2.78
2.81
1.41
5.05
1.16
1.86
2.54
2.77
d1
1.48
5.29
1.37
1.76
2.41
2.58
1.19
4.96
1.29
1.64
2.34
2.52
d2
1.48
4.34
1.58
2.67
2.70
2.67
1.26
3.72
1.28
2.30
2.87
3.09
d3
BIC
Source: Author's computations.
Shaded cells indicate the minimum RMSFE within a model specification and information criterion.
Without E-e
With E-e
Aggr.
AIC
Table 9.RMSFE estimates for h = 3
2.06
4.98
2.48
4.95
5.57
3.18
1.75
4.69
1.57
2.47
3.44
3.34
d4
2.99
3.97
3.08
4.19
3.67
3.22
2.24
3.71
2.21
5.25
4.29
3.48
%
1.79
5.04
1.45
2.20
2.84
2.79
1.38
4.89
1.14
2.10
2.68
2.67
d1
1.48
5.31
1.32
1.84
2.55
2.59
1.19
4.94
1.28
1.83
2.48
2.69
d2
1.51
4.26
1.84
2.52
2.51
2.84
1.28
3.78
1.26
2.12
3.00
3.14
d3
HQ
2.19
5.11
2.45
6.17
6.93
3.44
2.58
5.35
1.74
2.80
3.35
3.58
d4
2.70
3.92
3.23
5.61
4.17
3.65
2.37
3.50
2.26
5.73
4.52
3.62
%
152
LATIN AMERICAN JOURNAL OF ECONOMICS | Vol. 50 No. 1 (May, 2013), 133–161
2.22
1.26
5.09
1.51
3
4
5
6
2.79
2.72
1
4.62
1.21
5
6
2
2.52
1.31
3
4
2.98
2.68
1
2
d1
1.53
5.10
1.19
1.93
2.20
2.39
1.15
4.93
1.28
1.73
2.19
2.47
d2
1.41
4.18
2.03
2.66
2.70
2.72
1.22
3.88
1.13
1.85
2.70
2.65
d3
AIC
1.64
4.98
2.85
5.65
5.80
3.14
2.51
5.23
1.58
3.05
3.59
2.92
d4
2.53
3.77
2.56
6.95
5.75
4.38
2.29
3.81
1.92
5.47
3.39
4.02
%
1.52
4.92
1.32
1.92
2.60
2.81
1.27
4.83
1.07
2.12
2.32
2.80
d1
1.47
5.32
1.26
1.85
2.21
2.40
1.18
4.92
1.26
1.74
1.96
2.19
d2
1.42
4.28
1.54
2.65
2.49
2.26
1.24
3.76
1.23
2.00
2.37
2.55
d3
BIC
Source: Author's computations.
Shaded cells indicate the minimum RMSFE within a model specification and information criterion.
Without E-e
With E-e
Aggr.
Table 10.RMSFE estimates for h = 4
1.86
4.88
2.18
4.76
4.88
3.23
1.86
4.87
1.55
2.39
3.10
2.87
d4
2.65
3.91
2.52
3.96
4.23
3.31
2.17
3.98
2.10
4.35
3.55
3.23
%
1.45
5.04
1.27
1.95
2.74
2.77
1.25
4.80
1.14
2.11
2.50
2.78
d1
1.59
5.16
1.19
1.91
2.22
2.36
1.18
4.91
1.24
1.85
2.02
2.44
d2
1.48
4.43
1.65
2.61
2.48
2.28
1.24
3.81
1.10
2.16
2.65
2.63
d3
HQ
1.59
4.87
2.31
6.14
6.73
3.38
2.47
5.34
1.49
2.78
3.43
2.92
d4
2.50
3.96
2.60
5.73
4.64
3.85
2.24
3.86
1.94
4.49
3.21
3.85
%
C.A. Medel | How Informative are In-sample Information Criteria to Forecasting?
153
154
LATIN AMERICAN JOURNAL OF ECONOMICS | Vol. 50 No. 1 (May, 2013), 133–161
Table 11. Giacomini and White (2006) test results
Null Hypothesis:
AIC vs BIC
AIC vs HQ
BIC vs HQ
Actuals
h = 1
GW core stat
p-value
0.000
0.962
0.000
0.904
0.000
0.795
h = 2
GW core stat
p-value
0.000
0.423
0.000
0.817
0.000
0.701
h = 3
GW core stat
p-value
0.000
0.676
0.000
0.195
0.000
0.820
h = 4
GW core stat
p-value
0.000
0.792
0.000
0.368
0.000
0.450
Seasonally adjusted
h = 1
GW core stat
p-value
0.000
0.253
0.000
0.667
0.000
0.269
h = 2
GW core stat
p-value
0.000
0.734
0.000
0.384
0.000
0.074
h = 3
GW core stat
p-value
0.000
0.373
0.000
0.309
0.000
0.636
h = 4
GW core stat
p-value
0.000
0.456
0.000
0.472
0.000
0.682
Source: Author's computations.
The result suggests that exclusion of the Easter ef fect was not helpful
in terms of accuracy in any of the cases, in line with the in-sample
findings of Cobb and Medel (2010). For h = 1 and 4, the best results
are obtained with BIC, and for the remaining horizons with HQ. For
h = 1, BIC achieves an RMSFE of 1.13, followed by HQ with 1.17,
and AIC with 1.19. For h = 2, the best result is obtained with HQ,
showing an RMSFE of 1.11, then AIC with 1.18, and lastly, BIC
with 1.24. For h = 3 HQ achieves an RMSFE of 1.14, tied with
AIC, followed by BIC with 1.16. Lastly, for h = 4, the best result is
obtained with BIC for an RMSFE of 1.07, then HQ with 1.10, and
finally AIC with 1.13. According to Table 11 (lower panel), there
is only one case in which the dif ference among the two information
criteria is statistically dif ferent at a 10% level of confidence: BIC
and HQ at h = 2.
155
C.A. Medel | How Informative are In-sample Information Criteria to Forecasting?
Table 12.Summary of findings
h = 1
h = 2
h = 3
h = 4
h = 1
Actuals
h = 2
h = 3
h = 4
Seasonally adjusted
Model
ARMAX
➀➁➂
➀➁➂
➀
➀➁➂
✖
✖
✖
✖
SARMAX
-
-
-
✖
✖
✖
✖
ARMA
-
-
➁➂
-
✖
✖
✖
✖
SARMA
-
-
-
-
✖
✖
✖
✖
ARMA
✖
✖
✖
✖
➀➁➂
➀➁➂
➀➁➁
➀➁➂
ARMA
✖
✖
✖
✖
-
-
-
-
Transformation
d1
-
-
d2
➀➁➂
➂
➀➁
d3
➁➂
➀
-
-
➀➂
➀➁
➀
➀➁➂
➀➁➂
-
➁
➁
-
➁➂
d4
-
-
-
-
-
-
-
-
%
-
-
-
-
-
-
-
-
1
-
-
-
-
-
-
-
-
2
-
-
-
-
-
-
-
-
➀➁➂
➀➁
➀➁➂
-
-
-
-
➁
-
Aggregation
3
-
-
-
-
➂
➁➂
-
4
-
➁
5
➀➁➂
➀➁
➀
➀➁➂
➀➂
6
Source: Author's computations.
Finally, when using seasonally adjusted data, the recommendation
is to base the model selection for forecasting purposes on BIC when
predicting at one and four steps ahead and HQ for the horizons in
between. Also, it is recommended that the Easter ef fect (the Chilean
calendar) always be included in the estimates.
4.3.Special issues: models, aggregations, and
transformations
Besides the information criteria results, I treat some elements related
to the estimation as an additional topic. These elements are the kinds
of models, aggregations, and transformations to achieve stationarity
used to determine which information criterion is best for forecasting. In
Table 12, I transcribe the information of the aforementioned elements
156
LATIN AMERICAN JOURNAL OF ECONOMICS | Vol. 50 No. 1 (May, 2013), 133–161
from Tables 3 to 10 by locating the three best results for the three
information criteria analyzed, by each horizon, in the corresponding
cell for model, transformation, and aggregation.
The results regarding models reveal that it is always desirable, for all
horizons, to include the Easter ef fect with both actual and seasonally
adjusted data. Note that for all horizons the first three places are
located in the “ARMAX” or “ARMAE-e” rows, except two cases with
the actual series at h = 3. Regarding data transformations, with actual
series it is clear that the third order of dif ferencing is preferred, as
well as with seasonally adjusted data at h = 1. For the remaining
horizons, with seasonally adjusted data, the first order of dif ferencing
delivers the most accurate results. A similar situation occurs with
aggregations. With actual data it is clear that aggregation 6 (the
GDP alone) shows the best results, and the same is true for seasonally
adjusted data at h = 1. For h = 2 to 4, with the latter kind of data,
aggregation 4, that is, the weighted sum of supply components at one
level of disaggregation, should be used.
5.
Concluding remarks
The main aim of this paper is to identify which of three commonly
used information criteria has the highest predictive power to forecast
Chilean GDP and its components. Over 20 million ARMA models
were estimated using stationary transformations of the original series.
Then computer-based, non-trimmed projections from one to four steps
ahead were developed using a rolling window estimation scheme for
a sample of 59 observations. The ef fect of seasonality and the impact
of the Easter ef fect on the RMSFE were also investigated.
The results show that, on average, with actual series the information
criteria that show lower RMSFE across all horizons are AIC and BIC.
While AIC forecasts better one and two steps ahead, BIC is better at
the remaining horizons.
In the case of seasonally adjusted data, BIC shows better performance
one and four steps ahead, while HQ is the best at intermediate horizons.
In all cases, with and without seasonal adjustment, the dif ferences are
not statistically significant. Accounting for the Easter ef fect improves
the forecast accuracy for both actual and seasonally adjusted data.
C.A. Medel | How Informative are In-sample Information Criteria to Forecasting?
157
References
Akaike, H. (1974), “A new look at the statistical model identification,” IEEE
Transactions on Automatic Control 19(6): 716–23.
Bell, W.R. and S.C. Hillmer (1983), “Modelling time series with calendar variation,”
Journal of the American Statistical Association 78: 526–34.
Bell, W.R. and E. Sotiris (2010), “Seasonal adjustment to facilitate forecasting:
Empirical results,” Research Staf f Papers, U.S. Census Bureau.
Box, G.E.P. and G.M. Jenkins (1970), Time series analysis: Forecasting and
control, San Francisco: Holden-Day.
Box, G.E.P., G.M. Jenkins and G.C. Reinsel (2008), Time series analysis: Forecasting
and control (4th ed.), New York: Wiley.
Canova, F. (2007), Methods for applied macroeconomic research, Princeton:
Princeton University Press.
Capistrán, C., C. Constandse, and M. Ramos-Francia (2010), “Multi-horizon inflation
forecasts using disaggregated data,” Economic Modelling 27(3): 666–77.
Clark, T.E. (2004), “Can out-of-sample forecast comparisons help prevent
overfitting?” Journal of Forecasting 23(2): 115–39.
Clements, M.P. and A.B. Galvão (2010), “Real-time forecasting of inflation and
output growth in the presence of data revisions,” the Warwick Economics
Research Paper Series 953, Department of Economics, University of
Warwick, United Kingdom.
Cobb, M. (2009), “Forecasting Chilean inflation from disaggregate components,”
Central Bank of Chile Working Paper 545.
Cobb, M. and C.A. Medel (2010), “Una estimación del impacto del efecto calendario
en series desestacionalizadas chilenas de actividad y demanda” [An estimation
of the impact of the calendar ef fect in seasonally adjusted Chilean activity
and demand series], Revista Economía Chilena 13(3): 95–103.
Dickey, D.A. and S.G. Pantula (1987), “Determining the order of dif ferencing
in autoregressive processes,” Journal of Business and Economic Statistics
5(4): 455–61.
Findley, D.F., B.C. Monsell, W.R. Bell, M.C. Otto, and B. Chen (1998), “New
capabilities and methods of the X12-ARIMA seasonal adjustment program,”
Journal of Business and Economics Statistics 16(2): 127–52.
Ghysels, E., D. Osborn, and P.M.M. Rodrigues (2006), “Forecasting seasonal time
series,” in G. Elliot, C.W.J. Granger and A. Timmermann (eds.), Handbook
of economic forecasting, North Holland: Elsevier.
Giacomini, R. and H. White (2006), “Test of conditional predictive ability,”
Econometrica 74: 1545–78.
Goodwin, P., D. Önkal, and M. Lawrence (2011), “Improving the role of judgment
in economic forecasting,” in M. Clements and D.F. Hendry (eds.), The
Oxford handbook of economic forecasting, Oxford: Oxford University Press.
158
LATIN AMERICAN JOURNAL OF ECONOMICS | Vol. 50 No. 1 (May, 2013), 133–161
Granger, C.W.J. (1979), “Seasonality: causation, interpretation, and implications,”
in A. Zellner (ed.), Seasonal analysis of economic time series, Cambridge,
MA: National Bureau of Economic Research.
Granger, C.W.J. and Y. Jeon (2001), “The roots of US macro time series,” Working
Paper, University of California at San Diego.
Granger, C.W.J. and Y. Jeon 2004, “Forecasting performance of information criteria
with many macro series,” Journal of Applied Statistics 31(10): 1227–40.
Hannan, E.J. and B.G. Quinn (1979), “The determination of the order of an
autoregression,” Journal of the Royal Statistical Society B41: 190–5.
Hansen, P.R. (2009), “In-sample fit and out-of-sample fit: Their joint distribution
and its implications for model selection,” preliminary version April 23,
2009, Department of Economics, Stanford University.
Holan, S.H., R. Lund, and G. Davis (2010), “The ARMA alphabet soup: A tour
of ARMA model variants,” Statistics Surveys 4: 232–74.
Koehler, A.B. and E.S. Murphree (1988), “A comparison of the Akaike and Schwarz
criteria for selecting model order,” Journal of the Royal Statistical Society,
Series C (Applied Statistics) 37(2): 187–95.
Lütkepohl, H. (1985), “Comparison of criteria for estimating the order of a vector
autoregressive process,” Journal of Time Series Analysis 6(1): 35–52.
Medel, C.A. and M. Urrutia (2010), “Proyección agregada y desagregada del PIB
chileno con procedimientos automatizados de series de tiempo” [Aggregated
and disaggregated forecast of Chilean GDP with automatic time series
procedures], Central Bank of Chile Working Paper 577.
Mélard, G. and J.M. Pasteels (2000), “Automatic ARIMA modeling including
interventions, using time series expert software,” International Journal of
Forecasting 16(4): 497–508.
Newey, W. and K. West (1987), “A simple, positive semi-definite, heteroskedasticity
and autocorrelation-consistent covariance matrix,” Econometrica 55(3): 703–8.
Nishii, R. (1984), “Asymptotic properties of criteria for selection of variables in
multiple regression,” Annals of Statistics 12(2): 758–65.
Pincheira, P. and A. García (2009), “Forecasting inflation in Chile with an accurate
benchmark,” Central Bank of Chile Working Paper 514.
Rissasen, J. (1978), “Modeling by shortest data description,” Automatica 14(5): 465–71.
Schwarz, G.E. (1978), “Estimating the dimension of a model,” Annals of Statistics
6(2): 461–64.
Shibata, R. (1976), “Selection of the order of an autoregressive model by Akaike
information criterion,” Biometrika 63(1): 117–26.
Stock, J. and M. Watson (2007), “A comparison of linear and nonlinear univariate
models for forecasting macroeconomic time series,” in R.E. Engle and H.
White (eds.), Cointegration, causality, and forecasting: A festschrift in
honour of Clive W.J. Granger, Oxford: Oxford University Press.
U.S. Census Bureau, 2007, X12-ARIMA version 0.3 Reference Manual, available
at http://www.census.gov/ts/x12a/v03/x12adocV03.pdf.
62.8% Household
consumption expenditure (cn + cd)
20.1% Investment
(meq + ccw + cci)
58.2% Household consumption
expenditure: nondurables
Source: Author's computations.
(*) Not considered in analysis. (**) Subtracted.
6.4% Imports of services (**)
26.0% Imports of goods (**)
1.0% Changes in inventories (*) 32.4% Imports (**)
(mg + ms)
12.0% Government consumption
1.0% Changes in
expenditure
inventories (*)
29.2% Exports of goods
7.3% Exports of services
6.9% Machinery and equipment 12.0% Government
consumption
expenditure (g)
13.3% Construction and works
36.5% Exports (xg + xs)
4.6% Household consumption
expenditure: durables
Aggregation 2
Aggregation 1
Aggregation 4
95.9% Internal 12.5% GDP Natural
demand
resources
(c + i + g)
(egw + caf +  min)
4.1% External 82.5% GDP Non-natural
demand
resources (com + man
(x - m)
 + con + agr + tra + fin
 + per + ood + pub)
5.0% Other sectors
(-dut + vat + cif)
Aggregation 3
Table A1. Shares of sectorial components on real GDP
Appendix A
3.6% Agriculture and forestry
9.2% Transportation and
communications
15.0% Financial intermediation
and business services
11.6% Personal and social services
5.8% Owner-occupied dwellings
4.3% Public administration
3.4% Duties + taxes on goods
and services (**)
7.4% Non-deductible VAT
1.0% Imports CIF
6.9% Construction
9.7% Wholesale and retail trade,
hotels and restaurants
16.4% Manufacturing
8.4% Mining
1.2% Capture fishery
2.9% Electricity, gas and water
Aggregation 5
C.A. Medel | How Informative are In-sample Information Criteria to Forecasting?
159
160
LATIN AMERICAN JOURNAL OF ECONOMICS | Vol. 50 No. 1 (May, 2013), 133–161
Appendix B
Table B1.Typical statistics of demand side series, full sample
Mean
(Standard deviation)
d1
d2
d3
d4
Maximum
(Minimum)
%
d1
d2
d3
d4
%
cn
0.014
0.000
0.001
-0.003
0.058
(0.054) (0.100) (0.194) (0.381) (0.029)
0.135
0.172
0.331
0.670
0.157
(-0.084) (-0.203) (-0.358) (-0.634) (-0.018)
cd
0.033
-0.006
0.003
-0.021
0.139
(0.152) (0.247) (0.449) (0.845) (0.209)
0.582
0.637
1.019
1.267
0.769
(-0.341) (-0.695) (-1.002) (-1.909) (-0.361)
meq
0.031
-0.002 -0.005 -0.009
0.147
(0.114) (0.175) (0.313) (0.584) (0.207)
0.310
0.382
0.714
1.215
0.645
(-0.384) (-0.423) (-0.746) (-1.199) (-0.319)
cw
0.014
0.000
-0.002 -0.005
0.062
(0.080) (0.120) (0.199) (0.349) (0.096)
0.203
0.263
0.342
0.702
0.327
(-0.176) (-0.235) (-0.438) (-0.617) (-0.218)
g
0.011
-0.001 -0.001 -0.007
0.041
(0.117) (0.197) (0.350) (0.652) (0.023)
0.199
0.357
0.517
0.663
0.086
(-0.182) (-0.184) (-0.536) (-1.046) (-0.039)
xg
0.017
-0.002
0.004
-0.004
0.073
(0.102) (0.154) (0.241) (0.392) (0.071)
0.235
0.364
0.639
1.076
0.246
(-0.213) (-0.363) (-0.613) (-0.981) (-0.065)
xs
0.016
0.014
-0.011
0.003
0.130
(0.383) (0.580) (0.954) (1.686) (0.291)
1.290
2.287
2.643
5.206
1.693
(-1.236) (-2.242) (-3.177) (-5.270) (-0.462)
(mg)
0.031
-0.001 -0.001 -0.003
0.128
(0.074) (0.100) (0.164) (0.285) (0.136)
0.195
0.300
0.465
0.658
0.413
(-0.260) (-0.215) (-0.287) (-0.743) (-0.224)
(ms)
0.020
-0.003
0.004
-0.004
0.077
(0.112) (0.176) (0.302) (0.551) (0.101)
0.380
0.541
0.762
1.480
0.473
(-0.226) (-0.400) (-0.941) (-1.703) (-0.279)
c
0.016
0.000
0.000
-0.004
0.063
(0.058) (0.109) (0.210) (0.412) (0.039)
0.138
0.173
0.344
0.699
0.181
(-0.093) (-0.208) (-0.369) (-0.652) (-0.051)
i
0.022
-0.001 -0.003 -0.007
0.098
(0.071) (0.109) (0.198) (0.372) (0.132)
0.219
0.268
0.485
0.730
0.368
(-0.235) (-0.297) (-0.553) (-0.913) (-0.256)
x
0.017
0.001
0.001
-0.002
0.077
(0.099) (0.146) (0.227) (0.368) (0.069)
0.261
0.286
0.552
1.013
0.271
(-0.203) (-0.376) (-0.533) (-0.654) (-0.072)
(m)
0.029
-0.002
0.001
-0.004
0.116
(0.069) (0.095) (0.158) (0.281) (0.118)
0.233
0.255
0.388
0.549
0.349
(-0.219) (-0.188) (-0.414) (-0.802) (-0.192)
id
0.018
-0.002
0.000
-0.005
0.069
(0.056) (0.098) (0.185) (0.359) (0.067)
0.159
0.204
0.357
0.707
0.212
(-0.118) (-0.220) (-0.418) (-0.716) (-0.100)
ed
0.015
-0.001
0.000
-0.005
0.059
(0.064) (0.122) (0.240) (0.474) (0.033)
0.117
0.193
0.370
0.704
0.157
(-0.099) (-0.199) (-0.346) (-0.685) (-0.039)
gdp
0.013
-0.001
0.000
-0.005
0.054
(0.040) (0.069) (0.126) (0.238) (0.037)
0.094
0.164
0.272
0.396
0.163
(-0.082) (-0.127) (-0.210) (-0.482) (-0.045)
Source: Author’s computations.
161
C.A. Medel | How Informative are In-sample Information Criteria to Forecasting?
Table B2.Typical statistics of supply side series, full sample
Mean
(Standard deviation)
d1
d2
d3
d4
Maximum
(Minimum)
%
d1
d2
d3
d4
%
egw
0.011 -0.001 0.002 -0.001 0.047
(0.070) (0.090) (0.143) (0.247) (0.141)
0.203
0.246
0.333
0.564
0.483
(-0.219) (-0.215) (-0.295) (-0.618) (-0.387)
min
0.010
0.000 -0.001 -0.004 0.043
(0.069) (0.117) (0.215) (0.408) (0.064)
0.155
0.204
0.468
0.826
0.181
(-0.144) (-0.298) (-0.433) (-0.691) (-0.086)
caf
0.010 -0.008 0.002 -0.011 0.065
(0.283) (0.434) (0.707) (1.215) (0.141)
0.661
0.967
1.567
2.750
0.458
(-0.524) (-0.869) (-1.724) (-3.048) (-0.374)
agr
0.002 -0.007 -0.003 -0.009 0.052
(0.516) (0.729) (1.041) (1.509) (0.048)
0.903
1.070
1.915
2.511
0.177
(-0.892) (-1.076) (-1.948) (-2.307) (-0.097)
man
0.010 -0.001 0.001 -0.003 0.039
(0.041) (0.066) (0.118) (0.221) (0.051)
0.116
0.163
0.356
0.591
0.187
(-0.130) (-0.197) (-0.282) (-0.565) (-0.122)
com
0.016 -0.001 0.001 -0.005 0.067
(0.083) (0.136) (0.241) (0.445) (0.060)
0.204
0.293
0.460
0.811
0.236
(-0.108) (-0.232) (-0.525) (-0.985) (-0.092)
con
0.013
0.000 -0.002 -0.005 0.056
(0.074) (0.113) (0.190) (0.337) (0.085)
0.178
0.210
0.347
0.736
0.304
(-0.149) (-0.245) (-0.456) (-0.620) (-0.190)
tra
0.020
0.000
0.000 -0.003 0.080
(0.033) (0.055) (0.101) (0.190) (0.044)
0.091
0.165
0.296
0.460
0.212
(-0.082) (-0.130) (-0.217) (-0.512) (-0.030)
fin
0.016 -0.001 0.001 -0.004 0.065
(0.044) (0.073) (0.132) (0.248) (0.045)
0.129
0.136
0.321
0.628
0.170
(-0.093) (-0.197) (-0.307) (-0.480) (-0.043)
per
0.012 -0.003 0.001 -0.013 0.036
(0.216) (0.370) (0.673) (1.274) (0.015)
0.357
0.706
1.102
1.534
0.071
(-0.361) (-0.396) (-1.036) (-2.137) (-0.017)
ood
0.006
0.000
0.000
0.000
0.026
(0.003) (0.003) (0.004) (0.005) (0.009)
0.012
0.025
0.026
0.041
0.040
(-0.013) (-0.021) (-0.022) (-0.022) (-0.006)
pub
0.004
0.000
0.002
0.000
0.018
(0.006) (0.009) (0.016) (0.028) (0.013)
0.019
0.026
0.067
0.135
0.040
(-0.026) (-0.041) (-0.068) (0.105) (-0.022)
(dut)
0.016
0.000
0.000 -0.001 0.065
(0.041) (0.058) (0.088) (0.136) (0.049)
0.107
0.173
0.289
0.340
0.186
(-0.096) (-0.116) (-0.193) (-0.458) (-0.046)
vat
0.012 -0.001 -0.002 -0.004 0.051
(0.034) (0.058) (0.108) (0.204) (0.034)
0.079
0.117
0.215
0.330
0.134
(-0.068) (-0.110) (-0.171) (-0.372) (-0.042)
cif
0.033 -0.004 0.000 -0.007 0.140
(0.086) (0.128) (0.221) (0.402) (0.151)
0.256
0.318
0.538
1.055
0.448
(-0.177) (-0.360) (-0.658) (-1.196) (-0.201)
gdp nr
0.010 -0.001 0.000 -0.001 0.043
(0.053) (0.086) (0.153) (0.284) (0.056)
0.196
0.265
0.341
0.593
0.218
(-0.096) (-0.169) (-0.428) (-0.725) (-0.087)
gdp nnr
0.013 -0.001 0.000 -0.004 0.052
(0.037) (0.064) (0.119) (0.228) (0.035)
0.080
0.129
0.222
0.340
0.142
(-0.070) (-0.116) (-0.196) (-0.419) (-0.045)
others
0.013 -0.001 0.000 -0.004 0.052
(0.034) (0.057) (0.105) (0.199) (0.034)
0.078
0.112
0.204
0.316
0.136
(-0.068) (-0.110) (-0.167) (-0.357) (-0.045)
gdp
0.013 -0.001 0.000 -0.005 0.054
(0.040) (0.069) (0.126) (0.238) (0.037)
0.094
0.164
0.272
0.396
0.163
(-0.082) (-0.127) (-0.210) (-0.482) (-0.045)
Source: Author’s computations.