1. science
2. mathematics
3. statistics

SAMPLE SIZE AND CONFIDENCE WHEN APPLYING THE
NSSDA
Ariza López, Francisco Javier (*); Atkinson Gordo, Alan David (*)
(*) Grupo de Investigación en Ingeniería Cartográfica.
Dpto. de Ingeniería Cartográfica, Geodésica y Fotogrametría.
Universidad de Jaén.
Campus “Las Lagunillas” s/n. 23071. Jaén (Spain).
e-mail: [email protected]. Tel: +34953212469
e-mail: [email protected] Tel: +34927257195
ABSTRACT
In this work a simulation process is used in order to study the variation and stability of National Standard Spatial Data
Accuracy (NSSDA) results, depending on the sample size. Empirical results show that the NSSDA underestimated the
error level presented in the population, and that positional accuracy estimation also has a variability of 11%when using
the recommended sample size of 20 points. Simulation results indicate out the use of samples of a hundred points in
order to reach an effective confidence level of 95%. The NSSDA is a methodology of shared risk between users and
producers when accuracy is “as expected”, but for other cases the relation is altered, as simulation results demonstrated.
INTRODUCTION
Since positional quality is essential in cartographic production, all mapping agencies have used statistical methods for
its control, and we call these methods tests. Among the different methods used, we can highlight the National Map
Accuracy Standard (USBB, 1947), the Engineering Map Accuracy Standard (ASCI, 1983; ASP, 1985; Veregin, 1989;
Giordano and Veregin, 1994), the ASPRS (Merchant, 1987; ASPRS, 1989), or the more recent National Standard
Spatial Data Accuracy (FGDC, 1998).
The National Standard Spatial Data Accuracy (NSSDA) established by the Federal Geographic Data Committee in
1998 is a statistical methodology for evaluating the positional quality of a Geographic Data Base (GDB). NSSDA is a
compulsorily fulfilled standard for federal agencies of the USA producing analogical and/or digital cartographic data,
and is ever more widely used all over the world.
Like other positional quality control procedures, the coordinates of a set of points in the GDB are compared to
coordinates of the same points in a higher accuracy source, mainly a field survey. In this way an RMSE is derived from
discrepancies between pairs of coordinates. The NSSDA does not carry out a study on the presence of systematisms, as
it considers that “they might have been eliminated in the best way” (FGDC, 1998). Therefore, the NSSDA only focuses
on the study of data dispersion. The NSSDA gives results in a more open way than the previous test because it leaves to
the user’s understanding whether or not the derived accuracy reaches expectations, which means, in a practical way, if
the product passes or fails the user’s accuracy expectations. So acceptance or rejection is the responsibility of the user.
The test only tells us: "the product has been checked/compiled for N meters of horizontal/vertical accuracy at 95% of
level of confidence". Table 1 summarizes the steps for applying the standard.
From a statistical point of view, one of the most controversial aspects of all the above-mentioned methodologies for
positional control is the number and distribution of the control points. With regard to the number, which is our interest
here, it should always be large enough for the hypothesis of normality to be fulfilled, this being determined by the laws
of large numbers in statistics. For this reason recommendations always suggest at least 20 points (FGDC, 1998;
MPLMIC, 1999). Nevertheless, this size seems to be very small and some authors (Li, 1991) and institutions (Ordnance
Survey GB) suggest larger sizes. Obviously, since an elimination of gross errors should always be performed, a higher
number should be used. This number of points should be enough to ensure, with a given level of confidence, that a
GDB with a non-acceptable quality level will not be acquired. On the other hand, the number of points to be used for
the control must be the lowest possible in order to minimize the cost of such a control (Ariza, 2002).
Under the assumption of no systematic errors, our research focuses on the study of the variability of the estimated
positional (only horizontal) accuracy of a GDB when applying the NSSDA with different sample sizes. In order to
enable the generalization of results, synthetic Normal (µP= 0,  2P = 1) distributed populations of data are used.
Simulation is developed using the bases of the process shown in “Positional quality control by means of the EMAS test
and acceptance curves” (Ariza and Atkinson, 2005) also presented to this XXII International Cartographic Conference.
The confidence of the results is analyzed with reference to so-called user and producer risks.
Table 1.- Summary of the NSSDA when applied to the horizontal component
Select a sample of a minimum of 20 check points (n>=20)
Compute individual errors for each point i:
e xi  x ti  x mi
Compute RMSE for each component:
RMSE X 
e yi  y ti  y mi
e x2
RMSEY 
i
n
Compute the horizontal RMSE using appropriate expression:
If RMSEX = RMSEY
e y2
n
i
If RMSEX  RMSEY and 0.6 < (RMSEmin / RMSEmax) < 1,
RMSE R  RMSE X2  RMSEY2
ACCURACYR= 1.7308 RMSER = 2.4477 RMSEX = 2.4477 RMSEY
ACCURACYR  2.4477 0.5 (RMSEX + RMSEY)
Note: If error is normally distributed and independent in each of the x- and y-components, the factor 2.4477 is used to
compute horizontal accuracy at the 95% confidence level (Greenwalt and Schults, 1962).
This presentation is organized in four sections: The first deals with the analysis of the accuracy variability depending on
sample size but with fixed population variability, the second presents properties (underestimation and risks) of this
methodology that have not been mentioned previously, and the next presents the analysis of the user’s and producer’s
risks when variability of populations is considered. Finally, conclusions are presented.
Nowadays there is a proposal for the revision of the NSSDA based on various suggestions, like those coming from the
National Digital Elevation Program oriented towards adding instructions for how to test and report vertical accuracy in
areas where a normal distribution can not always be attained, or the proposal of Tilley (2002) in order to classify
accuracy results derived from the NSSDA, or the claim by McCollum (2003) that the Greenwalt and Shultz (1962)
estimator is inappropriately used in the NSSDA to determine a probability of 95%. So that our results can also give
more ideas for the redefinition of this standard.
VARIABILITY OF NSSDA ACCURACY ESTIMATIONS
In this paper simulation has been used as the base tool for analyzing the behaviour of the NSSDA methodology. The
simulation process is similar to that applied to other positional control methodologies (for more detail see Ariza and
Atkinson 2005); it basically consists of three main steps:
¬ Simulation of populations. A hundred synthetic populations of well known parameters (µP= 0,  2P = 1, where "P"
means population) are derived from a controlled statistical random values generation process. Single population
values are considered positional error values.
¬ Simulation of samples. A thousand samples of different sizes (n=10, 20, 30, and so on) are extracted from each
population. The NSSDA is applied to each sample as if it were a single positional control test.
¬ Statistic computations. Results values are aggregated deriving mean errors and variation of error, the later giving an
idea of the stability and reliability of the process.
Results of the process are shown in Table 2. Because the simulation is performed using a Normal ≈ N(µP= 0,  2P = 1)
distributed population, the theoretical value to be detected by the NSSDA is ACCURACYR = 2.447 m, which
corresponds to a circular error estimation with a probability of the 95%. Because of the large number of simulations, the
final results are very sound, with decreasing stability values ranging from 0.7 for samples of minimum size, and up to
0.2 for samples with maximum size. As can be observed in Table 2, mean values for 2,416 m  0,382 m are obtained
for sample sizes of 10 points. This last supposes a 15.8 % of variability with respect to the mean observed value, and is
inadmissible. For the size recommend by the NSSDA (n=20) the observed values obtained is ACCURACYR = 2.434 m
 0.241 m. This value is 0.6% less than the corresponding theoretical value to be detected, and in this case variability is
in the order of  11%, so that actually accuracy has a 89% of confidence. The variation range decreases when sample
size increases, so that for sample sizes of 700 points it is circa 1%. In this way, taking a sample size of 95 points the
mean observed value is ACCURACYR = 2.443 m  0.121 m and the variability is within ± 5 % of that value. So if we
want to work with a confidence level of 95% it is not advisable to use less than 95-100 points for the control sample. In
this case the simulation variability is about 0.4% which means a variation interval between ±4.6% and ±5.46%. Also, if
we want to limit the variability to a maximum of 5%(= ±2,5%) a sample with at least n=275 points will be needed. But
it is also possible to obtain the maximum variation value for a given sample size. The deviation value presented in
column “c” of Table 2 should be multiplied by a K factor related to the desired confidence level, and added to the mean
value of the same table. For instance, when n = 275, and the desired confidence level is 95%, K=1.96, so from Table 2
we obtain:
ACCURACYR (Maximum) = ACCURACYR ± K Deviation = 2.446 m ± 1.96 x 0.061m = 2.446 m ± 0.120 m
Other studies based on population normality (LI, 1991) suggest samples sizes of the same order.
Table 2.- Mean ACCURACYR values and variability obtained by simulation of samples and populations
n
(a)
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
85
90
95
100
125
150
175
NSSDA
m
(b)
2,416
2,426
2,432
2,434
2,437
2,438
2,439
2,440
2,441
2,441
2,442
2,442
2,442
2,443
2,443
2,443
2,443
2,444
2,444
2,444
2,446
2,446
Dev.
±m
(c)
0,382
0,312
0,270
0,241
0,219
0,203
0,189
0,178
0,168
0,160
0,153
0,147
0,141
0,136
0,131
0,127
0,124
0,121
0,115
0,099
0,088
0,083
Variation
±%
(d)
15,8
12,9
11,1
9,9
9,0
8,3
7,8
7,3
6,9
6,6
6,3
6,0
5,8
5,5
5,4
5,2
5,1
5,0
4,7
4,1
3,6
3,4
Stab.
%
(e)
0,7
0,6
0,5
0,5
0,5
0,4
0,4
0,4
0,4
0,4
0,4
0,4
0,4
0,4
0,4
0,4
0,4
0,4
0,4
0,4
0,3
0,4
n
(a)
200
225
250
275
300
325
350
375
400
425
450
475
500
525
550
575
600
625
650
675
700
800
NSSDA
m
(b)
2,445
2,446
2,445
2,446
2,446
2,445
2,446
2,446
2,446
2,446
2,446
2,446
2,446
2,447
2,446
2,446
2,446
2,446
2,446
2,446
2,446
2,447
Dev.
±m
(c)
0,075
0,070
0,065
0,061
0,058
0,055
0,052
0,048
0,046
0,043
0,042
0,040
0,039
0,037
0,034
0,033
0,031
0,030
0,028
0,026
0,025
0,024
Variation Stab.
±%
%
(d)
(e)
3,1
0,4
2,9
0,4
2,6
0,4
2,5
0,3
2,4
0,3
2,3
0,3
2,1
0,3
2,0
0,3
1,9
0,3
1,8
0,3
1,7
0,3
1,6
0,3
1,6
0,3
1,5
0,3
1,4
0,2
1,3
0,3
1,3
0,2
1,2
0,2
1,1
0,2
1,0
0,2
1,0
0,2
1,0
0,2
Columns are: (a) Size (number of points) of the 1000 random samples; (b) simulation mean observed value for
horizontal accuracy (ACCURACYR) by applying the NSSDA with a 95% confidence level; (c) mean deviation of
the simulation process with respect to the mean observed value; (d) previous deviation expressed as a percentage
of mean observed value of the horizontal accuracy; (e) stability of the process when using a hundred random
populations, distributed as N(0,1).
The same tendency results are expressed graphically in Figure 1. Here the X-axis refers to the size of control sample,
and the Y-axis to the mean observed, or estimated, population value through the sample when using N(µP= 0,  2P = 1)
populations in the simulation process. The wider and red dashed line corresponds to the value to be theoretically
detected by the NSSDA: ACCURACYR = 2.447 m. The series of points are the results of the simulation; they have a
very clear tendency, approaching from below the theoretical value when increasing the sample size. The two other
dashed lines represent the decreasing tendency of the variability of the mean values.
Exactitud estimada por NSSDA para 1 -  = 95%
2.450
2.445
NSSDA (m)
2.440
2.435
2.430
2.425
2.420
2.415
2.410
0
100
200
300
400
500
600
700
800
(n) puntos por muestra
Figure 1.- Mean ACCURACYR values (points) and variability (black dashed lines) obtained by simulation. The
theoretical NSSDA value corresponding to a N(µP= 0,  2P =1) distributed population is shown in red
UNDERESTIMATION, USER’S AND PRODUCER’S RISK
As shown in Figure 1, the mean estimated value for a NSSDA control is smaller than the corresponding value for the
N(µP= 0,  2P = 1) distributed population. In other words, in mean values the NSSDA underestimated the error level of
the population or overestimated the accuracy. But variability of mean values can be above or below the mean tendency,
so that for a specific control there is a probability for a better or a worst estimation of the RMSER, and that implies
certain kind of risk that we call producer’s and user’s risk. Figure 2 shows a graphical interpretation: the area between
the upper dashed line and the horizontal red one, which corresponds to expected population value (an ACCURACYR =
2.447 m for N(µP= 0,  2P =1) distributed population), represents a producers risk. The same is true for the area between
the lower dashed line and the same horizontal red line, but in this case corresponding to the user.
The interpretation given here to such probabilities or kind of risks is as follows:
 Producer risk means the probability of an estimation of the population’s RMSER value greater than it actually is. So
the conclusion for the NSSDA is a worse accuracy than the true one. For a given sample size, for instance n1, this
occurs proportionally to the ratio between segments A1B1 and A1D1 (producers risk = A1B1/A1D1). Segment A1B1 is
the width of the risk-producer area for n1, and A1D1 is the total variability for the mean estimation when sample size
is n1.
 User risk means the probability of an estimation of the population’s RMSER value less than it actually is. So the
conclusion of the NSSDA is a better accuracy than the true one. For a given sample size, for instance n1, this occurs
proportionally to the ratio between segments B1D1 and A1D1 (users risk = B1D1/A1D1). Segment B1D1 is the width of
the risk user area for n1, and A1D1 is the total variability for the mean estimation when the sample size is n1.
A
B
NSSDA
Pobl.
NSSDA
RP
RU
D
D
10
15
20
25
30 n35 40
45
(n) Sample size
50
55
60
65
70
Figure 2.- Probabilities of better and worst estimations of RMSER than actually (User’s and producer’s risk)
Numerically speaking, both probabilities are very similar and also present a tendency to be equal (Table 3), but
curiously for lower sample sizes greater differences occur although these are limited to 4.2% for n=10. This behaviour,
in relation to producer and user risk, implies certain equitability because both share, more or less, the same risk.
Table 3.- Probabilities of better and worst estimations
of RMSER than actually (User’s and Producer’s Risk)
size
Probabilities
size
Probabilities
n
RU
RP
n
RU
RP
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
90
0.521
0.517
0.514
0.514
0.512
0.511
0.511
0.510
0.509
0.510
0.508
0.509
0.509
0.508
0.508
0.508
0.479
0.483
0.486
0.486
0.488
0.489
0.489
0.490
0.491
0.490
0.492
0.491
0.491
0.492
0.492
0.492
95
100
110
125
150
175
200
225
250
275
300
400
500
600
700
800
0.506
0.494
0.507
0.507
0.508
0.503
0.503
0.507
0.504
0.508
0.506
0.504
0.506
0.507
0.508
0.510
0.500
0.493
0.493
0.492
0.497
0.497
0.493
0.496
0.492
0.494
0.496
0.494
0.493
0.492
0.490
0.500
VARAIBILITY OF USER’S AND PRODUCER’S RISK
Until now we have worked under the assumption of having a N(µP= 0,  2P = 1) distributed population and analyzed
what can occur when estimating from a sample of a given size. Now we are going to analyze the behaviour of the
NSSDA when expecting N(µP= 0,  2P = 1) distributed population but actually working with other normal N(µP= 0, 
2
2
P = D ) distributed population (D<>1). So we are going to determine the user’s and producer’s risk for that condition.
For this analysis we have used a simulation process similar to that mentioned above, but changing the variation
behaviour when creating random populations. So a set of populations normally distributed has been synthetically
created following a N(µP= 0,  2P = D2), where D = 0,8; 0,85; 0,9; 0,95; 1,00; 1,05; 1,10; 1,15 and 1,20. For each
synthetic population a thousand samples of different sizes (n=10, 20, 30, and so on) were extracted. The NSSDA was
applied to each sample as if it were a single positional control test.
The results of this process are presented in Figure 3. The horizontal axis corresponds to sample size and vertical axis to
ACCURACYR (left) and D (right). Figure 3 shows very similar tendency lines, but shifted to the vertical. The wider and
dashed line corresponds to the previously studied situation where population follows a N(µP= 0,  2P = 1) and so is the
same line as presented in Figure 1. Tendency curves above the wider and dashed line correspond to those cases were D
> 1, and curves below to those where D<1.
NSSDA (m)
Deviation (m)
3.0
1.2
2.9
1.15
2.8
1.1
2.7
2.6
1.05
2.5
1.0
2.4
0.95
2.3
2.2
0.9
2.1
0.85
2.0
0.8
1.9
0
20
40
60
80
100
120
140
Sample Size
Figure 3.- Evolution of ACCURACYR values for different population deviations versus sample size
The different values of D can be considered, in relation to D=1, as exigency ratios (ER) implying a detected nominal
accuracy value when applying the NSSDA, and vice versa. This idea is presented in Table 4. For example: ER = 0,8
means that we require an ACCURACYR = 1,957 m but the estimation from sample gives ACCURACYR = 2,446 m. So
that our exigency is the 80% (= 1,957 / 2,446) of the actual accuracy of the product. The opposite case occurs when ER
> 1, for instance if ER = 1,2 that means we require an ACCURACYR = 2,935 m but the estimation from sample gives
ACCURACYR = 2,446 m, so the product is a 120% (= 2,935 / 2,446) more accurate than expected.
Table 4.- Exigency ratios or D values and corresponding detected nominal
values for the NSSDA (ACCURACYR)
Exigency Ratios (ER)
ACCURACYR
0,80
0,85
0,90
0,95
1,957
2,079
2,201
2,324
1,00
1,05
1,10
1,15
1,20
2,446
2,568
2,691
2,813
2,935
The described variability behaviour of values around the mean (previous section) has a very important role now when
studying the behaviour of the NSSDA when expecting a N(µP= 0,  2P = 1) distributed population but actually working
with a N(µP= 0,  2P = D2) distributed population. In order to use an acceptance index in the form of probability we
consider the following rules:
 R1: If D < 1  ER > 1, the accuracy of the population is better than expected, and this means that it would be
considered as satisfactory or accepted. So when performing the simulation, we will take into account the number of
cases where observed resulting values for ACCURACYR < 2.477m. The number of such cases will be expressed as
an acceptance percentage of total cases (the number of times we are able to say that accuracy is better than
expected).
 R2: If D >1 ER < 1, the accuracy of the population is worse than expected, and this means that it would be
considered as not satisfactory or not accepted. So when performing the simulation, we will take into account the
number of cases where observed resulting values for ACCURACYR < 2.477m. The number of such cases will be
expressed as an acceptance percentage of total cases (the number of times we are not able to say that accuracy is
worse than expected).
Figure 4 shows obtained results for the above mentioned process when representing acceptance values as a percentage.
The black wider curve corresponds to the case where D=1 (labelled with 1). Its value is a little more than 50% because
the accuracy underestimation of the NSSDA generates that situation. The red curves correspond to cases where D<1
(labelled with 1.05 up to 1.20). Here quality is better than expected, user acceptance increases and producer’s risk
decreases to a percentage equals to 100% minus Acceptance (%) (a good product can be rejected in this percentage).
The blue curves correspond to cases where D>1 (labelled with 0,95 up to 0,80). Here quality is worse than expected,
and user acceptance is a risk (a bad product can be accepted in that percentage) which decreases when sample size n
increases.
(%) Acceptance
100
1,20
1,15
90
1,10
80
1,05
70
60
1,00
50
40
30
0,95
20
10
0,85
0,80
0,90
0
0
20
40
60
80
100
120
140
Sample Size
Figure 4.- Evolution of acceptance levels for different population deviations versus sample size
CONCLUSIONS
By using a simulation based methodology the NSSDA ACCURACYR variations and risk behaviours have been analyzed.
The statistical analysis is based on the use of normal distributed synthetic populations, which ensures the control of the
process and the generality of results and its easy applicability to real cases. The main conclusions derived from the
results are:
 The NSSDA has a little tendency to underestimate accuracy.
 For the minimum proposed sample size (20 points) the variability of results is in the order of 11%, which actually
means a confidence level of the 89%.
 In order to have a 95% confidence level on estimation, and variability within a range of ±5%, the sample size must
be in the order of 100 points.
 Because of its statistical formulation, the NSSDA accuracy estimation gives similar user’s and producer’s risks,
which means a shared risk behaviour.
 If the variability of the population is greater or lesser than expected, user’s and producer’s risks change. We have
derived a family of curves that can be used by users to determine the sample size for limiting their risk but also by
producers to analyze the tradeoffs between their product’s quality and acceptance to decide and establish the
capacity of the production process.
ACKNOWLEDGEMENTS
This work has been partially funded by the National Ministry of Sciences and Technology under grant nº BIA200302234.
REFERENCES
ARIZA, F.J. (2002). Control de Calidad en la Producción Cartográfica. Ra-Ma.
ARIZA, F.J., ATKINSON, A. (2005). Positional quality control by means of the emas test and acceptance
curves. In proceedings of the XXII International Cartographic Conference, La Coruña, España.
ASCI (1983). Map Uses, scales and accuracies for engineering and associated purposes. American Society
of Civil Engineers, Committee on Cartographic Surveying, Surveying and Mapping Division, New
York.
ASP (1985). Accuracy Specification for Large-Scale Line Maps. In PE& RS, vol 51, nº 2.
ASPRS (1989). Accuracy standards for large scale maps. In PE&RS, vol. 56, nº7.
ATKINSON, A. (2005). Control de calidad posicional en cartografía: análisis de los principales estándares y
propuesta de mejora. Tesis doctoral, Universidad de Jaén, Jaén.
FGDC (1998). Geospatial Positioning Accuracy Standards, National Standard for Spatial Data Accuracy.
FGDC-STD-007-1998, http://www.fgdc.gov/
GIORDANO, A.; VEREGIN, H. (1994). Il controllo di qualitá nei sistemei informative territoriali. Il Cardo
Editore, Venetia.
GREENWALT, C.; SHULTZ, M. (1962). Principles of error theory and cartographic applications. ACIC
Technical Report nº 96. ACIC, St Louis.
MCCOLLUM, J. (2003). Map Error and Root Mean Square. In proceedings of the Sixteenth Annual
Geographic Information Sciences Conference of the Towson University and Towson University's
Department of Geography and Environmental Planning (TGGIS 2003).
MERCHANT, D. (1987). A Spatial Accuracy Specification for Large Scale Topographic Maps. In PE&SR,
vol. 53.
MPLMIC (1999). Positional Accuracy Handbook. Minnesota Planning Land Management Information
Center; http://www.mnplan.state.mn.us/press/accurate.html.
TILLEY, G. (2002). A classification system for National Standards for Spatial data Accuracy. In proceedings
of the Fifteenth Annual Geographic Information Sciences Conference of the Towson University and
Towson University's Department of Geography and Environmental Planning (TGGIS 2002).
USBB (1947). United States National Map Accuracy Standards. U.S. Bureau of the Budget.
VEREGIN H. (1989). Taxonomy of Errors in Spatial Data Bases, Technical Paper 89-12, NCGIA, Santa Barbara.