Competitive Work Sample
decade, work sample testing has
preeminent
position within the field
come to occupy a
of vocational evaluation. Indeed, to a certain extent,
During the
Norms and Standards:
Some Considerations
Abstract: The article addresses several critical tsmes with regard to
the use o] competitive work sample norms. The author discusses
three bast• techniques for developing competitive norms and the
basic limitations of each technique. The main thesis is that competitive norms, when properly understood and applied, can be an
asset to the practice oy vocational evaluation. Rehabtlitaton educators are challenged to make training in the interpretation oy work
sample norms one of the primary goals of evaluator training.
24
samples represents a
vocational
distinguishes
some ways
evaluation from other, more traditional psychological
testing approaches. To a large extent, interest in work
samples grew out of a desire to provide handicapped
people with what were commonly referred to as "more
meaningful and valid" test experiences than those afforded by paper and pencil tests. In many eases,
standardized psychometric tests were viewed as inadequate for the unique needs and limitations of
handicapped people. Yet, in spite of this observation,
the history of work sample development suggests that
work samples, like their predecessors, have followed
the same basic pattern of growth. That is, emphasis
has been placed on utilizing a trait-factor approach to
understanding human behavior (Dunn, 1977), standardizing all testing procedures, and essentially harboring the philosophy that individual performance is
most meaningful when compared to the collective performanee of a specified group (Buros, 1978). With this
background in mind, the use of norms in general, and
more recently, the mandate to develop and use
competitive work sample norms is not surprising.
the
widespread
practice which
PAUL McCRAY
Paul McCray is a Development Specialist with the
Materials Development Center, Stout Vocational Rehabilitation Institute, University of Wisconsin-Stout.
He was previously employed as Director of Rehabilitation Services for a private nonprofit rehabilitation
facility in Arizona. He holds a B.S. degree in Psychology from Arizona State University and an M.S.
degree from the University of Arizona.
last
use
of work
in
The purpose of this paper is not to discuss whether
competitive work sample norms are needed. Indeed,
this issue has already been decided. The Commission
on Accreditation of Rehabilitation Facilities (CARF)
and the Vocational Evaluation and Work Adiustment
Association (VEWAA) have developed work sample
standards which state: "3.4.3.1.1.7.2. If work sampies are used.., e. competitive norms or industrial
standards shall be established and used" (CARF 1978,
p. 28). Where appropriate, this standard is now applied to vocational evaluation programs as part of the
accrediting process. Thus, the mandate to establish
and use competitive work sample norms is dear. The
purpose of this paper is twofold: first, to review some
important considerations in using competitive norms,
and seeond• to suggest a possible course of action
which will help insure that this standard, when satisfied, will serve the purpose for which it was intended;
namely, to facilitate an accurate and reliable understanding of an individual client's rehabilitation potential.
The rationale for using competitive norms may be
viewed in some ways as a reaction to vocational evaluation's dissatisfaction with client and general population norms. For a long time, the inherent problems in
using such norms were well-known, or at least suspected. Allen and Sax (1972) pointed out that:
sample norms based on the performof
former clients or members of a genance
eral, non-disabled population offer no direct
basis for determining whether a client is eal3-
Work
Vocational Evaluation and Work
Adiustment Bulletin 19
79
able of
functioning
at
or near
competitive industrial workers
on
the rate of
a
particu-
(page 2)
Trying to predict someone's ability to do a iob by
comparing their performance to a norm group of
people for which there may be no direct evidence that
they can do the job is usually not a strong foundation
for making reliable predictions as to iob potential. Yet
this practice was commonly accepted and iustffied on
the grounds that it would be "unfair" to compare a
handicapped worker's performance to that of a
competitively employed worker (Feingold, 1961).
Eventually, however, the pendulum of change
swung toward the philosophy that since, in most eases,
lar task
clients would be competing for iobs against competitively employable people, competitive norm groups
needed. Hence, CARF standard 3.4.3.1.1.7.2.e
established. Not unexpectedly, however, competitive work sample norms, like their forerunners, have
been found to have significant limitations which must
be recognized and understood if such norms are to be
an asset, rather than a detriment, to the practice of
vocational evaluation.
Dunn (1976) and MeCray (1979) have described
three basic methods for obtaining competitive norms.
First, workers from industry can be administered the
work sample, and their performance used as a basis
for establishing a norm group. Second, in cases where
industry has already developed production standards
for a specific job, a job sample may be developed and
the existing standards directly applied to the assessment tool. Third, industrial engineering techniques
such as time study or predetermined motion time systems (PMTS) may be used to analyze a work sample
and develop production standards for the work sampie. All three techniques have important considerations which can have a pronounced effect on the way
they are used within the context of vocational evaluawere
was
tion.
Having competitively employed workers perform
a
sample, in order to establish norms, can be partieularly useful when an evaluator has a work sample
which has already been developed, but lacks any
work
normative data. It can also be useful with a work
which is based on a iob taken from industry,
sample
but which has no concrete production standards e.g.,
maid service. There are, however, three important
issues that must be considered when using a group of
competitively employed workers for establishing a
norm group. First, it is imperative that members of
the norm group be employed in iobs directly related to
the work sample (MeCray, 1979). As obvious as this
may seem, in practice it may be very difficult to
achieve, since jobs and work samples with similar
titles and tasks may still be substantially different in
many critical areas. For example, it may appear that a
group of solderers would be ideal for establishing
Vocational Evaluation and Work
norms on a soldering work sample, yet closer examination could reveal that the type of work solderers do on
one job is wholly different than that done on the work
sample. Thus, it is essential that the norm group be
thoroughly described in the work sample manual and
understood by the evaluator in terms of its relationship
to local iob demands. Unless this is achieved, it is easy
for an evaluator to erroneously compare a client's per-
formance to an inappropriate norm group.
The second problem with this norming procedure is
that even if norms are derived from an occupational
group that is closely related to the work sample, the
evaluator still must frequently compare an inexperienced client's performance to that of experienced
workers. Thousands of practice trials on the iob may
dramatically facilitate employed workers' work
sample performance. Unless this is understood, the
evaluator may easily misinterpret a novice client's
initial subcompetitive performance as necessarily
indicative of a lack of task related ability. Yet, as
Curtin (1973) has suggested, unless a learning curve
has been established for a task, there will be no objective data that can either confirm or discount the notion
that there may be a dramatic difference between
trainee and experienced worker performance levels.
Nor is there likely to be any information which can
suggest that point at which a worker should begin to
approach an average competitive production standard. Depending on the iob, the average number of
trials necessary to reach a competitive production level
might vary from one or two trials to several thousand.
Thus, a client's performance when compared to a
group of experienced workers might appear to be very
low, but if that same client's performance was compared to a group of newly hired workers, it might then
be competitive. One simply has no way of knowing
unless a learning curve has been established for the
task. Ideally, then, it is advantageous for the evaluator
to understand the entry level production standard
necessary to obtain a iob, how long it generally takes
to obtain that standard, and the average competitive
production standard.
The third issue revolves around the problem of narrowness of the worker norm group. Since all members
of this group are already competitively employed on
the job, even the bottom 1% of the group has to be
recognized as employable. If client's performance is
only at the first percentile, ita still suggests potential
ability to do the iob. Thus, the evaluator must be particularly sensitive to the description of the reference
group. Oherwise a client may be mistakenly screened
out of a iob or training program because of what
superficially appears to be a low score. An even more
critical problem is the fact the evaluator still has no
idea of what the minimum performance standards
are. That is, the evaluator still doesn't know how far a
client can score below the 1% level and still possibly
Adiustment Bulletin 1979
25
do the job. For example, if the poorest competitively
employed member of the norm group had an average
work sample performance of 50 units per hour, does
that mean that a client who only averages 35 pieces
per hour couldn't compete in industry? Once again,
the evaluator has no reliable way of answering this
question, yet this information is used as a basis for decision-making and production.
The second norming method, developing job samples from industry, can be a highly effective technique
for obtaining reliable, competitive production standards for a specific job. In effect, the evaluator essentially uses the production standards established by
industry as the comparison group and buffcls a job
sample which entirely replicates the actual job. Thus,
there is little question as to whether•th• standards used
with the _job sample really reflect the standards used
by industry. In addition to this, industry may also be
able to delineate the acceptable performance range:
e.g., 100 pieces per hour is average for experienced
workers, workers are employed who range in production from 70-120 pieces per hour average, and workers
with less than 40 hours on the job are expected to average at least 50 pieces per hour.
As useful as the job sample can be in terms of establishing competitive norms, it has two important limitations. First, it requires the evaluator to entirely
replicate the job, not only in terms of the production
standards used, but also the work layout, method,
tools, training procedures, etc. In many cases this will
not be practically feasible, yet it must be done if the
standards are going to be used. If the replication is not
complete, the performance standards may no longer
be applicable, since different work sample procedures
may strongly affect performance. Evalators must be
made aware that only if job conditions are duplicated,
can the production standards applied to the job also be
applied to the job sample.
The second major problem with this approach is its
limited capacity for generalization. Industrial production standards derived from a single job can rarely be
extrapolated to other iobs. This is true even if the jobs
are related. New standards must be developed for
each job. It is evident, then, that in spite of the advantages of borrowing standards directly from industry, the ensuing application of such standards is likely
to be expensive and time-consuming.
Establishing independent production standards
solely for a work sample is the third major method for
obtaining competitive norms. It requires expertise in
the use of industrial engineering techniques such as
time study or any of the predetermined motion time
systems (PMTS), such as MTM or MODAPTS. These
methods can provide highly accurate and reliable data
with regard to establishing work sample
standards. However, there are two important factors
which evaluators must understand before attempting
production
26
to interpret a client's performance in comparison to
these standards. First, no matter which system is used,
they all share one common assumption: The standards are based on average, experienced workers. It is
assumed that the worker is thoroughly familiar and
proficient with all aspects of the task. This is obviously
not the ease with many rehabilitation clients. Second,
the standards developed for the work sample only
apply to the work sample; they may not have any
transferability to the local job market's production
standards. A nut and bolt assembly work sample may
indicate that 50 units per hour is average productivity;
but in any single community, similar jobs may have
widely differing standards both above and below the
work sample standard. Real production standards
used in industry are often affected by uncontrolled
variables such as union demands, skill of the local
labor force, local wages, and variation among tasks.
The range of acceptable performance on the same job
may vary dramatically from one business to another.
How then does an evaluator take these factors into
consideration when weighing a client's performance
versus an ideal industrial standard? The answer lies in
the notion that evaluators must be aware of local produetion standards. A theoretically ideal work sample
production standard, which has no established, concrete relationship to the demands of real work or the
immediate work available to a client, has limited usefulness in terms of improving an evaluator's understanding of a client's job potential.
Competitive norms and standards are neither inherentl good nor bad. Their usefulness is determined to a
great extent by the evaluator's ability to effectively
interpret their meaning in relation to a client's performance. When used by adequately trained vocational evaluators, they can be an invaluable asset to
understanding a client's performance capabilities.
Herein lies the challenge to rehabilitation education.
Many vocational evaluators, both in the field and in
institutions of higher learning, still are not fully aware
of the critical issues surrounding the use of competitive
work sample norms. It is the responsibility then of rehabilitation educators to make proficiency in the use
of work sample norms, one of the primary goals of
vocational evaluator training.
REFERENCES
Allen, C., & Sax, A. Norms and performance standards for work sample scores. Menomonie, Wisconsin: University of Wisconsin-Stout, Department of
Rehabilitation and Manpower Services, Materials
Development Center, 1972.
Buros, Oscar K. (Ed.). The eighth mental measurements yearbook, Volume II. Highland Park, New
Jersey: The Gryphon Press, 1978.
Vocational Evaluation and Work
Continued
on
page 29
Adiustment Bulletin 1979
about rehabilitation clients. For additional information regarding the VDARE Process, contact: Stanley
D. Harris, VDARE Service Bureau, P.O. Box 55,
Roswell, Georgia 30077.
Vocational evaluation and work adjustment practitioners are encouraged to submit material on innovative developments within the field or to suggest topics
for inclusion in this column. Materials for the "Innovations Column" should be submitted directly to:
Dr. Arnold B. Sax, Director
Materials Development Center
University of Wisconsin-Stout
Menomonie, Wisconsin 54751
TOPICAL REVIEW
Continued from page 27
one-hour sessions. The
procedures
used
were
instruc-
tion, discussion, modeling, and feedback. The behav-
iors which were trained in the study were: 1) Length
of eye contact, 2) voice affect, 3) voice loudness, 4)
body posture, 5) positive personal self-statements, 6)
positive work-related statements, 7) appropriate
speech, and 8) interviewee-initiated
Three videotaped interviews using
three equivalent questions were used
statements.
a
standard
to rate the
set
of
parti-
cipants on these behaviors at different points in the
training sequence. A personnel manager also reviewed
pre and post interviews and rated the participants in
four general areas regarding the positiveness of their
interviews. The results indicated that the individuals
improved in six of the eight targeted behaviors. The
personnel manager ratings also indicated that all the
participants improved with the training.
COMPETITIVE WORK SAMPLE
Continued from page 26
Accreditation of Rehabilitation Facilifor rehabilitation facilities.
Chicago,; Author, 1978.
Curtin, K. Work sample dreams industrial reality.
Paper presented at International Association of Rehabilitation Facilities Conference, Miami, June
1973.
Dunn, D. Work samples as predictors of vocational
success. Paper presented at Systems of Vocational
Evaluation, a rehabilitation forum sponsored by
Rehabilitation Services Education, Auburn University, Auburn, Alabama, July 1977.
Dunn, D. Using competitive .norms and industrial
standards with work samples. (Interface Number
9), Menomonie, Wisconsin: University of Wisconsin-Stout, Stout Vocational Rehabilitation Institute,
Research and Training Center, 1976.
Feingold, S. Work evaluation: Which norms? Rehabilitation Counseling Bulletin, 1961, 4, 126-130.
MeCray, P. An interpretation of VEWAA/CARF
work sample standards. Menomonie, Wisconsin:
University of Wisconsin-Stout, Stout Vocational Rehabilitation Institute, Materials Development Center, 1979.
Commission
on
ties. Standards manual
Vocational Evaluation and Work
This researcher has made an admirable effort to
evaluate the effectiveness of a short-term training
program on job interviewing skills with rehabilitation
clients. Unfortunately, there are some problems with
the research design and with the handling of the
study's data which make the interpretations of the
results somewhat questionable.
The Furman et al study and the Pinto study provide
a framework for evaluating and training job interviewing skills. A first step for implementing this type
of program is to devise a standard set of interview
questions and a rating system for interviewee
responses. These studies suggest several target behaviors which can be assessed and trained in order to increase job interviewing effectiveness. A pre-training
interview could be video or audio-taped and assessed
by the vocational evaluator. Information from this
assessment could be described in the evaluation report
with recommendations to the work adjustment
specialist for further training.
The training procedures used in these studies could
then be considered by the work adjustment specialist
when constructing a job interviewing training program. The multiple baseline design across individuals
or interview behaviors could be useful in determing
the effectiveness of a given program with a facility's
particular clientele. It is possible that short-term training with small groups would prove to be a very effective method of teaching these much needed job interviewing skills to vocational rehabilitation clients.
Adjustment Bulletin 1979
29