1. science
2. mathematics
3. statistics

How to Quickly, Easily and
Cheaply Conduct a Survey to
Discover What Your Customers
and Prospects Want So You Can
Give It to Them
Vic Curtis
Business Development & Research International
[email protected]
(+44) 0845 8056705
Copyright © Business Development & Research International 2010, All Rights Reserved
1
Limits of Liability / Disclaimer of Warranty:
This report is NOT legal or accounting advice. You need to do your own due-diligence to
determine if the content of this report is right for YOUR business. No earnings claims are
being made anywhere in this report or in the marketing of this report. The publisher of this
report is not liable for any damages or losses associated with the content in this report. Any
changes you implement in your business are your own responsibility. All trademarks are the
property of their owners.
Copyright © Business Development & Research International 2010, All Rights Reserved
2
Contents
1. Ask Them!
4
2. How and Where Do You Start?
6
3. Simple Statistics – Population and Sampling
9
4. The Quickest, Cheapest and Easiest Way to Survey
13
5. Which Free Survey Software to Use
16
6. Survey Questionnaire Design
21
7. What Sort of Things Do I Ask?
24
8. Open Questions, Closed Questions and Scales
27
9. How to Improve Response Rates
31
10. Pretest Your Survey!
33
11. Checklist
35
Glossary
36
Copyright © Business Development & Research International 2010, All Rights Reserved
3
1. Ask Them!
Suppose you want to find out the level of customer satisfaction with your
company compared to your competitors .... or suppose you want to find
out what features and benefits your prospects would like to see in your
new products .... or suppose you want to find out how people view your
customer service or product pricing. The best way to get this specific
information is to ask them – and one of the quickest, cheapest and easiest
ways to ask them is to conduct a survey or questionnaire.
You may feel that in these days of Web 2.0 and social media, it is just as
easy to go onto Facebook or Twitter or industry forums, and see what
people are saying about your company. However, there are two major
problems with this approach. Firstly, they are unlikely to be talking about
the particular issue you have in mind – and it could also take a lot of time
searching for relevant comments. Secondly, it is often only the people
who either love or hate what you are doing who are committed enough to
mention you and these may not be the views of the silent majority.
Surfing the Internet has its place in seeing if problem areas may exist, but
is unlikely to provide the answers.
So, conducting a survey is a means of gathering information about a
specific topic (eg level of satisfaction) from particular population (eg your
customers or prospects). This is done by correctly sampling some
members of that population, usually through a system of standardized
questions. These questions may be related to behaviors, beliefs, attitudes
and/or characteristics of those who are surveyed.
Analysis and evaluation of the survey data results in a profile of the
population sampled with regard to that topic. Action can then be taken,
Copyright © Business Development & Research International 2010, All Rights Reserved
4
based on the results so that you can do more of what your customers do
want, do less of what they don’t want and so beat your competitors to the
sale.
Surveys can be conducted by mail, telephone, personal interview, or the
Internet. They can be administered either to individuals or groups.
However, irrespective of how the survey is conducted, it must be designed
correctly to produce quality results. There are many free tools which can
be used to put up a survey or questionnaire on the Internet, but they will
not give answers which are actionable unless the sample is correctly
chosen, the questions are correctly set and the survey induces people to
respond.
This report should give you some basic steps to ensure that any surveys
or questionnaires you design, undertake or commission will give you the
best possible results from which you can take the necessary action.
Copyright © Business Development & Research International 2010, All Rights Reserved
5
2. How and Where Do You Start?
A. What is the Problem?
The first thing to do is to write down your problem statement. This is a
clear, concise statement of the problem or issue to be studied and/or the
information desired. It should be put down into writing. Studies have
shown that the very act of writing things down will help clarify your
thoughts and make it much more likely that you will carry through.
It is helpful to list possible causes of the problem, as well as possible
solutions. This will help clarify the survey objectives.
B. Secondary Information
Since almost all surveys can be costly (either in dollars, pounds or euros,
or in hours, days or weeks spent), it is necessary to establish what
information is already known. Begin by contacting persons knowledgeable
in the field and by performing an environmental scan of other studies
conducted on the topics of interest. This work should provide the answers
to the following questions:
1.
Have studies of this subject been done
previously?
2.
Is there literature enough on the subject
to answer the question (i.e., books,
periodicals, reports)?
3.
Have other organizations investigated this
Research Tip:
For a FREE
report on how
to quickly and
easily conduct
secondary
market
research, email
your request to:
[email protected]
area, and do they have information
available on the subject?
Copyright © Business Development & Research International 2010, All Rights Reserved
6
4.
Can the desired information actually be collected by a survey or
would another form of research (eg observation, focus group) be
more appropriate?
5.
Is there adequate time and resources available to conduct a
survey without skipping steps in the process?
Once the context of the survey is determined, the objectives must be
developed for the survey.
C. Survey Objectives
Establishing the survey objectives is the one thing which is very seldom
done but which ensures that the survey is most likely to succeed and
obtain results which are actionable. The objectives will define:
1.
What is the problem to be resolved?
2.
What information is needed in order to understand the problem, its
causes, and possible solutions?
3.
How will the information be used
and by whom?
4.
What/who is the population to be
studied and can all members of
the population be located?
5.
Does the information collected
need to be statistically valid and
does it need to be generalized to
a larger population?
Research Tip:
A discussion of analysing
survey and questionnaire
results and statistics (eg
mean vs median, standard
deviations, distribution
curves, levels of
significance, levels of
confidence, hypothesis
testing, regression analysis
etc) is beyond the bounds of
this report. If you need help
or more information, please
contact BDRI:
[email protected]
Copyright © Business Development & Research International 2010, All Rights Reserved
7
6.
What kinds of analyses would be useful for understanding the
survey results?
7.
What decisions will be made based on the data?
D. Survey Budget
When conducting a survey, an adequate budget must be developed to
cover all phases of work. This should be done early in the planning
process so that expectations for what the survey can accomplish remain
realistic in light of financial constraints.
This report also gives some free resources and software which can be
used to minimize any costs involved – and in some cases to eliminate
them entirely. However, it should be noted that there will still be a time
cost involved with developing the survey, conducting it and analyzing the
results.
Copyright © Business Development & Research International 2010, All Rights Reserved
8
3. Simple Statistics – Population and Sampling
A. Target Population
The target population is all the people or
companies who you would wish to survey,
should you have enough time, money and
resources. If all this population is surveyed,
this would be a census.
For a target population of, say, 200 companies,
this may be possible. Normally however, a
census is too costly and too time consuming to
undertake and so a sample of the target
population is taken to represent it instead.
Research Tip:
A target population
must be definable (so
you can determine
who is in and who is
out), finite and easily
reachable (so you can
communicate with
them). For example,
this could extend
from a small
educational research
project of 10 people
to brand awareness
studies of female
supermarket
shoppers aged 18-45.
B. Sampling
It is important that the sample taken must be representative of the
population and this is usually done by picking respondents randomly from
the population. This could be by:
•
‘picking names from a hat’
•
approaching members of the population at random
•
listing all the members of the population in alphabetical order and
choosing, for example, every 10th member
•
use the Excel random number generator for each member of the
population, sort in numerical order and take the top members
Although taking a sample from a target population is much cheaper and
quicker than conducting a census, this also introduces a degree of error
which must be considered. The results obtained from a sample can only
give an estimate of the true results which would have been obtained from
Copyright © Business Development & Research International 2010, All Rights Reserved
9
a census. This is because two samples taken from the same population
will not give exactly the same results due to variations in the samples
themselves. Consequently, it is only possible to say with a specified
degree of certainty or confidence that the sample result is a good
approximation of the true result. How good an approximation depends
largely on the size of the sample.
C. Sample Size
A common misconception is that in order to take a sample of a
population, taking a fixed percentage (eg 10%) will be sufficient. This is
not the case. Rather, it is the absolute number in the sample that is
important, almost irrespective of the size of the population.
The simplest estimation of sample size is when trying to determine the
proportion or percentage of a large population who are either for or
against a given scenario (eg % of respondents who like a particular brand
of chocolate). To calculate the sample size in this case, the result is
assumed to be half for and half against. This will give the largest sample
necessary and will cover any other possible results.
Before the actual sample size can be estimated, two other parameters
must be first established. Firstly, the amount of error which can be
tolerated must be specified. This is how much leeway can be given on
either side of the sample result so that the real result can lie within that
range. It is given as a percentage eg 10%. So, for a sample answer of
50% and sampling error of 10%, the range where the real result could lie
would be 40-60%. This is known as the confidence interval.
Secondly, the degree of confidence in the real answer lying within this
range must be stated. Typically, 95% confidence is used.
Copyright © Business Development & Research International 2010, All Rights Reserved
10
For large populations (over 5,000) and at
95% confidence, the sample size can be
estimated from the following formula:
Sample size =
9604
.
(Amount of Error)
2
So, for our example, where a 10% error can
be tolerated,
Sample size =
9604
10
=
2
Research Tip:
If you watch opinion
polls at elections,
they often give an
error of ±3%. Put
this into the formula
and it gives a sample
size of 1,067. This is
how national
elections results are
predicted based on
exit polls of just over
one thousand people.
96
It can be seen that there is a relationship between the sample size and
the error. If the formula is switched around, the amount of error can be
calculated if a specific sample size is used. A balance needs to be struck
between minimizing the sample size for cost and timing implications and
minimizing the error so as to make the results worthwhile. In order to
halve the error, the sample size must be quadrupled.
Research Tip:
At a 95% level of confidence, the
error associated with a particular
sample size is given below:
Sample
Size
30
50
75
100
150
200
250
500
1000
Error
(%)
17.9
13.9
11.3
9.8
8.0
6.9
6.2
4.4
3.1
The minimum number of
respondents which allows
statistical results to be determined
is usually taken as 30. At this
level, patterns start to emerge
from the data although the error is
quite large (±17.9%).
Ideally, at least 50-100
respondents would be surveyed,
more if the budget allows.
Copyright © Business Development & Research International 2010, All Rights Reserved
11
It should also be noted that if comparisons are to be made between
different groups or segments from within the survey respondents (eg male
vs female, over 18 vs under 18 etc), then each group requires at least the
minimum number of respondents ie 30. Hence, the total minimum number
of respondents required will be at least 60, preferably more.
D. Sample Size from a Smaller Population
For a population less than 5,000, the sample size calculated from above
must be adjusted to account for the smaller pool of potential respondents.
The new sample size is given by:
New sample size =
Old sample size
.
1 + [Old sample size – 1 ]
Size of Population
So, if the old sample size from a large population was 96, then from a
population of 1,000, the new sample size would be:
New sample size =
96
.
1 + [ 96 – 1 ]
1,000
=
96
.
( 1 + 0.095 )
=
87.6 rounded up to 88
The size of the population sample to be used, the means by which they
can be contacted, how easily they can be contacted and how much that
costs will all influence the survey mode, or how the survey is conducted.
Copyright © Business Development & Research International 2010, All Rights Reserved
12
4. The Quickest, Cheapest and Easiest Way to Survey
The next step in the process is to determine which survey mode to use.
The survey mode is how the survey will be conducted. The most
frequently used modes include face-to-face or personal interviews,
telephone interviews, and written interviews which are usually conducted
by mail or via the internet.
The factors that will determine which mode to choose include ease of
access to the sample; financial constraints; resource constraints; and
question length, complexity, and sensitivity.
A. Face-to-Face Interview
Face-to-face or personal interviews are surveys conducted in person by an
interviewer who usually travels to the person being surveyed.
• Pros—High response rates; can clarify questions, if necessary;
control over respondent selection; can use longer, more complex
questionnaire; and easier to build rapport and motivate the
respondent.
• Cons—High costs, time-consuming, and more administrative
requirements (i.e., selecting and training interviewers, contacting
respondents, travel arrangements). Also, there is a tendency for
respondents to give socially acceptable answers.
B. Telephone Interviews
Telephone interviews are usually conducted from a central office that
places telephone calls to selected households or businesses.
• Pros—Good response rates (although falling), fast, some
anonymity for respondents in answering questions, and control over
respondent selection. Answers can be filled directly into a
Copyright © Business Development & Research International 2010, All Rights Reserved
13
spreadsheet or database. Clarification of any questions not fully
understood possible. If a comprehensive list of the target population
is available, the likelihood of obtaining a representative sample is
high.
• Cons—Questions must be short and not
complex; cannot control interruption by
others in household/ office; hard to find
persons at home, and those that are at
home may resent intrusion; there is
mounting displeasure among
households receiving unsolicited telephone
calls; requires training and quality control
monitoring of the interviewers; and is
usually difficult to target a specific
Research Tip:
In the UK, be aware
of the requirements
of the Telephone
Preference Service
(TPS). Genuine
market research
questions are
allowed, but
marketing or selling
are not. For more
information, go to
www.tpsonline.org.uk
geographical location. Many households
are becoming mobile only and hence more
difficult to contact as not on usual lists.
C. Mail Questionnaires
Mail questionnaires are written surveys that are sent through the post to
selected members of the population to be surveyed.
• Pros—Reasonable response rates provided rigorous follow-up
procedures, relatively easy to obtain a listed population and locate
respondents, can avoid interviewer bias and distortion, answers
unlikely to be socially influenced, easy to administer and relatively
low costs, can cover a wide geographical area, and more
manageable for handling large samples.
• Cons—Questionnaire may be given to someone else to fill out,
may not reach the desired respondent or may be thrown away as
junk mail; most difficult type of questionnaire to design as
Copyright © Business Development & Research International 2010, All Rights Reserved
14
everything needs to be explained on the survey sheet; hard to
interpret open questions; cannot control sequence in which
respondents answer questions; time consuming, given periodic mailout requirements and need to code and enter results into
spreadsheet or database.
D. Internet Questionnaire
An internet questionnaire is a form of a written survey. Respondents may
be invited to participate in the survey through email or because they visit
a particular web page.
• Pros—Fast to conduct and tabulate, some software products allow
questionnaires to be customized depending on the respondent’s
answers (eg with skip patterns or with audio, video and graphical
interfaces), avoids interviewer bias and distortion, answers unlikely
to be socially influenced, easy to administer, especially for larger
samples, and low costs. Allows the opportunity to obtain email
address so easy to follow up to improve response rates. Responses
can be downloaded directly into spreadsheet or database reducing
time and possible transposition errors.
• Cons—Poor control over respondent selection; not everyone owns
a computer so can be somewhat self selecting; difficult to obtain
probability sample; and, like mail surveys, this is the most difficult
type of questionnaire to design.
Overall, the quickest, easiest and cheapest way to survey your customers
is to set up a survey online. Customers fill it in and the results are
downloaded straight into a spreadsheet or database from where they can
be analysed. In addition to email, potential respondents may be contacted
by telephone or post, and directed to the relevant website.
Copyright © Business Development & Research International 2010, All Rights Reserved
15
5. Which Free Survey Software to Use
There are many online survey packages which can be used to put a
questionnaire on the internet. Many of these offer a free trial (often time
limited) or a free version with reduced features compared to the paid-for
version. However, in several cases, the free version is sufficiently good
and has enough features to enable a survey to be undertaken quickly,
easily and at no cost.
Three of the most popular online survey packages are:
•
Survey Monkey – www.surveymonkey.com
•
Zoomerang – www.zoomerang.com
•
SurveyGizmo – www.surveygizmo.com
Each of these online survey options has a free, basic version which allows
surveys to put online at no cost. When you visit the website, you can sign
up for a free account using your email address and a password you
nominate. Once you have signed up, you are ready to create your first
survey.
However, there are differences in the features available for each of these
free survey services. These range from the number of questions which can
be asked through the possibility of embedding the survey into other
websites and blogs to the ability to download the results. These features
dictate which free survey package is the best to use.
The table overleaf shows some of the more useful features at the basic
free level.
Copyright © Business Development & Research International 2010, All Rights Reserved
16
SurveyMonkey
Zoomerang
SurveyGizmo
Free
Free
Free
Unlimited
Unlimited
Unlimited
Questions Allowed
per Survey
10
30
Unlimited
Number of
Responses Allowed
100 / survey
100 / survey
250 / month
Cost of First Paid
Level
$19.95 / month
$199 / yr
$19 / month
Surveys Allowed
Unlimited
Unlimited
Unlimited
Questions per
Survey
Unlimited
Unlimited
Unlimited
Responses Allowed
1000 / month
Unlimited
1000 / month
Cost if over
Response Limit
$0.05 / extra
response
N/A
Upgrade to $49
level for month
Cost at Basic Level
Number of
Surveys Allowed
Style Templates
Available
Various Languages
Section 508
Compliant
Export Survey to
.DOC Format
Embed into Other
Websites & Blogs
Survey Progress Bar
Customize Thank
You Page
Export Data to .CSV
Format
Copyright © Business Development & Research International 2010, All Rights Reserved
17
From this table, it can be seen that SurveyGizmo is the best free online
survey software to use as there are more useful features than either
Survey Monkey or Zoomerang. All free versions allow an unlimited number
of surveys to be undertaken, although SurveyGizmo also allows an
unlimited number of questions per survey. Survey Monkey is poorest in
this respect as it confines this to only 10 questions which could limit the
usefulness of the survey.
SurveyGizmo also has the largest number of responses allowed, at 250
per month, compared to 100 per survey for Survey Monkey and
Zoomerang. Assuming all 250 responses were used for one survey, this
would give an error of just over +/- 6% whereas 100 responses would
give an error of just under +/- 10%.
The surveys are set up and hosted on the websites of the three
companies. Respondent data is then stored on their servers. All three
have style templates which can be used and can be set up in various
languages. Survey Monkey and SurveyGizmo are both Section 508
compliant which means that they fulfill visual disability criteria and have a
progress bar which allows the respondent to see how far through the
survey they are.
However, only with SurveyGizmo can the survey be exported to a .DOC or
Word document format and can the survey itself be embedded into
another website or blog (much like a YouTube video). In this latter case,
the respondent data is still captured and stored by the SurveyGizmo
software. It is also possible to customize the ‘Thank You’ page to which
the respondent is sent once they have completed the survey. This gives
the possibility of allowing the respondent to download some free report or
being offered a sale price on a product as a thank you.
Copyright © Business Development & Research International 2010, All Rights Reserved
18
However, the major reason for choosing the free version of SurveyGizmo
above Survey Monkey or Zoomerang is that the former allows the data to
be downloaded in a .CSV file which can be opened in Microsoft Excel.
Hence, the data can be analysed in much greater depth. The latter two
free versions do not allow download of the data. It should be noted that
all three websites provide online graphical reports of the results.
Therefore, the best free online survey software can be found by opening a
new account at the following website:
http://www.surveygizmo.com/new-account
Once you have followed the link, ensure that the “free version” checkbox
is ticked. Then, at the bottom of the page, fill in your first and last name,
email address and password. Once you tick the checkbox to agree to the
terms and conditions and the privacy policy, click the orange button
saying “Create Account”. You are taken to a Welcome On board page
where you can login at the orange “My account” button or the green
“Login Now” button at the top right. Follow the instructions to create your
first survey.
The free versions of the survey software from Survey Monkey, Zoomerang
and SurveyGizmo are only the precursor to full option, paid-for packages.
These start from $19 / month, but have a much greater feature range for
more complicated surveys. They also allow a larger number of responses,
starting at 1,000 / month. Survey Monkey and Zoomerang allow data
download at their paid levels.
For more extensive surveys, it may be worth considering paying for one or
two months for the higher level. This will allow access to more features
Copyright © Business Development & Research International 2010, All Rights Reserved
19
and more responses which will generate more in depth data. Once the
more extensive survey is complete, you can then go back to the free
version for any smaller survey you may wish to undertake. Although any
survey data is not deleted from any of the software servers, it would be
advisable to download all results first before going back to the free
versions of the software.
Copyright © Business Development & Research International 2010, All Rights Reserved
20
6. Survey Questionnaire Design
The survey questionnaire should be designed to include elements which
make the survey pertinent and relevant to the population to be sampled,
thereby maximizing response rates and minimizing error or bias.
A. Components
The following sections normally comprise a questionnaire:
1.
Request for Cooperation—This might be a brief introductory
paragraph at the beginning or could be a comprehensive cover
letter. It should highlight the reason(s) for the survey, that
participation is voluntary, confidentiality of answers, and data
protection.
2.
Instructions—Always simple, clear, and repetitive where necessary.
Keep to a minimum and make sure they are easy to administer if
given by an interviewer.
3.
Actual Questions—See Sections on
Question Content and Question
Formats.
4.
Classification Data—Normally these are
demographic information and
respondent characteristics to ensure
the target population has been
sampled adequately. These can also be
used as a basis to analyze the survey
results to allow comparison between
Research Tip:
It is sometimes
advisable to put
some screening
questions before the
main body of the
survey. These
screening questions
ensure that the
respondents are part
of the target
population eg when
was the last time
you used the
product in question?
different groups (eg male vs female).
Copyright © Business Development & Research International 2010, All Rights Reserved
21
Research Tip:
It is advisable to keep the
components in this order. Any
personal or demographic
information eg age, gender,
income etc should be collected
at the end of the survey.
Asking these questions first
may discourage potential
respondents.
5.
Identification Data—This may
include names, addresses,
telephone numbers, email
addresses and/or identification
numbers of participants to keep
track of respondents and to
facilitate follow-up procedures.
B. General Layout
The layout of a written questionnaire can have as much to do with
response and error rates as do the actual questions. Therefore, the
following factors need to be carefully addressed:
1.
Length—All surveys should be as brief as possible. Mail and
telephone surveys should be no longer than 10 to 15 minutes.
Internet surveys should ideally be less than that. Personal
interviews should not extend beyond 30 minutes. Stick to tightly
focused surveys to discover what you need to know rather than
long rambling questionnaires which ask views on subjects which
are not relevant.
2.
Appearance—Mail and internet surveys should give the appearance
that they will be easy to complete. Neat, orderly written
questionnaires with a lot of white space will increase response
rates.
3.
Instructions—Clear, unambiguous, and easily readable instructions
work best. In mail and internet surveys, it helps to offset
instructions from the rest of the text so it is clear where the
questions start.
Copyright © Business Development & Research International 2010, All Rights Reserved
22
4.
Vertical Flow—Logical question and section sequencing is critical.
Avoid jumping from topic to topic. Cluster similar types of
questions either by subject, type of response, and/or instruction.
5.
Numbering Sequence—Precoding every item on the questionnaire
allows for ease of data entry. However, coding must be done
discretely if it is to appear on all but the master copy of a written
survey to avoid confusing respondents. Most internet surveys are
set up to code the answers automatically and download to a
spreadsheet, although as previously shown, not all free versions
have this facility. Compared to mail surveys, this simplifies and
quickens the process, and also ensures no transposition errors
occur.
6.
Transition Statements—When shifting topics and/or sections in the
questionnaire, clear and understandable transition elements or
statements are important.
Copyright © Business Development & Research International 2010, All Rights Reserved
23
7. What Sort of Things Do I Ask?
The following factors must be considered when constructing the questions
to be used in the survey:
1.
Will the question give the type of response desired? For example,
“How long have you worked in your current job?” An open question
of this type may result in answers such as “since I left school,”
instead of number of years.
2.
Use words which are simple, familiar,
and unambiguous to the target
population. Do not use colloquialisms
or slang. The best clarity is achieved
when using the reading level of a 10
year old when constructing
questions. The question “Which
detrimental attributes impact on our
transportation system?” contains
words that are too difficult. The
question “What do kids in your
neighbourhood do for fun?” is vague
Research Tip:
Questions should contain
simple, familiar words
rather than complex
phrases or slang e.g.
Complex
proximity
exhausted
leisure time
candid
employment
assistance
rectify
priority
Simplified
closeness
tired
free time
honest
work
help
correct
most
important
and contains slang. Kids does not
define a specific age group and can
refer to young goats.
3.
Avoid double-barrel questions. The question “Do you think the
customer service agent was courteous and knowledgeable?” is two
separate questions. If the respondent agrees with one statement
but not the other, they will not know how to answer. Split doublebarrel questions into two separate questions.
Copyright © Business Development & Research International 2010, All Rights Reserved
24
4.
Determine whether respondents will be able to answer accurately;
are they likely to know, understand, and/or remember items
relating to the desired information? Respondent recall becomes
unreliable quickly and what has happened most recently becomes
the default position, whether good or bad. Structure questions to
assist memory by measuring a discrete and recent time period
rather than a vague reference to the past.
5.
Avoid questions containing double negatives or phrases such as,
“would you agree or disagree that…” These types of questions
confuse respondents who may not correctly interpret the intended
meaning. Also avoid leading questions which steer respondents
towards your favoured answer.
6.
Research Tip:
The first question
is important as it
sets the tone. It
should be short,
simple, relevant
and easy to
answer.
Preferably, the
respondent should
be able to agree
or answer with
certainty as this
creates an air of
confidence. More
complex issues
can come later in
the questionnaire.
This will help
ensure higher
response rates.
Is the person answering the question
the appropriate person to do so? Very
often for example, one member of a
household has more knowledge than
others about the household’s finances.
7.
Will the respondent be willing to answer
certain types of questions truthfully?
Some topics (eg income, criminal and
other kinds of personal behaviors)
regularly elicit biased responses or
higher item nonresponse. For age and
income questions, consider giving a
range so that the respondent does not
have to give a precise figure. Make sure
the ranges given do not overlap.
Copyright © Business Development & Research International 2010, All Rights Reserved
25
8.
Does the question bias the
respondent’s answer? “The
Government believes that public
spending should be cut to reduce
the national debt. Do you agree?”
Research Tip:
The following are four of
the most common types
of question you can ask in
a survey:
•
Behaviours
•
Beliefs
•
Attitude
•
Attributes
If this appeared on a survey, the
answers might reflect feelings
about the Government rather than
what should be done with public
spending.
9.
Questions which appear to be “off
the wall” and unrelated to the
subject matter being explored
should be avoided. The questions
should provide the information
needed as defined in the survey
objectives and purpose. Focus on
the need to know rather than the
nice to know.
10. In multiple-choice or closed
questions, make sure all possible
response choices are included and
are mutually exclusive. When
asking the number of times
something has occurred, it is not
unusual for the answer choice
What respondents do
(present, past and
future)
What respondents
think is true; their
perception of reality
(assessment
orientated, taps what
they know)
Respondents’ views,
perceptions or
feelings; how they feel
(often judgmental)
Personal or
demographic
characteristics (age,
income, occupation)
Again, it is advisable to
keep questions in this
order. Asking about the
satisfaction (ie attitude)
with a product is of little
benefit until the usage (ie
behaviour) is known. Start
with general questions
and then move to more
specific.
“none” or “0” to be missing.
Copyright © Business Development & Research International 2010, All Rights Reserved
26
8. Open Questions, Closed Questions and Scales
Page 5
A. Open Questions
Generally, questions may be either open or closed. Open questions are
ones in which no answer is suggested and the respondent is expected to
spontaneously provide the answer themselves. This will then be in their
own words and as such, answers may be very expansive and detailed.
This type of data is known as qualitative and is good for answering “how”
and “why” type questions. Open questions are more appropriate to
smaller surveys or focus groups where the emphasis is on insight.
Difficulties may be encountered transcribing such answers verbatim (esp.
in personal or telephone interviews) and hence a recording device may be
appropriate. Permission should be asked first. Unless a respondent feels
strongly about a particular subject, open questions may be viewed as
more difficult to answer and hence response rates may be lower.
B. Closed Questions
Closed questions have answers given and hence
the respondent can only choose from those
suggested. This type of data is known as
quantitative and is good for answering “what” or
“how many” type questions. Closed questions are
more appropriate to larger surveys where the
emphasis is on data mining and applying the
results to the target population.
Closed questions may be filled in more quickly
Research Tip:
If statistical
analysis is
required from the
data, use a
majority of closed
questions. This
will allow
averages (means)
and proportions to
be calculated from
which inferences
can be made from
the sample to the
target population.
and are hence viewed as easier by respondents.
Copyright © Business Development & Research International 2010, All Rights Reserved
27
C. Basic Question Formats
There are five basic formats from which to structure questions in a survey:
1. Open:
“The job tasks I enjoy most are ___________________.”
2. Modified open:
“I was _______years old when I began my current job.”
3. Closed with ordered response choices:
“How would you rate your preferences for the following job tasks?”
(tick one answer for each item):
Enjoy
Neutral
Dislike
Writing
□
□
□
Editing
□
□
□
Organizing
□
□
□
4. Closed with unordered response choices:
“Which of the following job tasks do you like the most?” (circle one
letter)
A. Writing
B. Editing
C. Organizing
5. Partially closed:
“Which job task do you most enjoy doing?” (circle one letter)
A. Writing
B. Editing
C. Organizing
D. Other (please specify)_____________
Copyright © Business Development & Research International 2010, All Rights Reserved
28
In general, closed questions with ordered or unordered response choices
are the easiest to code for data processing. Open questions are the most
difficult. However, all question types can be useful depending upon what is
being measured (behaviors, attitudes, etc) and the kinds of information
needed.
D. Rating Scales
One powerful technique to use in a survey, especially when asking
attitudinal questions e.g. customer satisfaction, is to have respondents
rate the parameter on a given scale. The Likert Scale is the most well
known and is often used to measure qualities such as satisfaction and
importance, such as:
“How satisfied were you with the service you received during your
last visit to our company?” (Please tick one box)
Very
Somewhat
dissatisfied
dissatisfied
□
□
Neutral
□
“How important to you is the service
you received during your last visit to
our company?” (Please tick one box)
Very
important
□
Somewhat
important
□
Not really
important
□
Not at all
important
□
Somewhat
Very
satisfied
satisfied
□
□
Research Tip:
The survey designer will have
to decide whether to use an
odd or even number scale.
An odd number may allow
respondents to ‘sit on the
fence’ at the midpoint
whereas an even number
forces the respondent to
choose as there is no
midpoint. As a guide, a
7 point scale will allow
respondents to register a just
positive or just negative
response without resorting to
the central position.
Copyright © Business Development & Research International 2010, All Rights Reserved
29
If possible, the rating scale should be laid out vertically with the most
positive response uppermost. This format creates the least amount of bias.
If the rating scale must be horizontal, put the most negative response on
the left and most positive on the right. This counteracts the natural desire
which favours both the left side and positive answers.
Scales should be balanced on both sides of the midpoint, with equal
numbers of positive and negative options. Although 5 point scales are
most frequently employed, 7 or 9 point scales can also be used for greater
refinement. The wording used in both the question and the scale should be
clear, concise and unambiguous so that it carries the same meaning to all
respondents.
An alternative to labeling each option is just to label the two end and the
central options. This allows the respondent to decide for themselves on
their strength of feeling for the options in-between without “putting words
into their mouths”. This is called anchored end points. This can also be
used to create a semantic differential scale where two opposite meaning
words or phrases are used as the anchored end points.
Copyright © Business Development & Research International 2010, All Rights Reserved
30
9. How to Improve Response Rates
Not everyone to whom the survey is sent will respond. This may be due to
lack of time or interest, not being relevant to the person or the survey
looking too hard to fill in. Consequently, the sample size is affected by the
response rate, which is given by:
Response rate =
Number of completed surveys x 100%
Number of surveys distributed
Hence, if the response rate is 50% and a sample size of 100 is required,
then 200 surveys must be distributed to take into account the non
response. Face-to-face and telephone surveys are more likely to achieve
higher response rates than mail or internet surveys. However, the latter
are both cheaper to undertake and so more surveys can be distributed.
There are several ways in which the
response rate can be improved before
and during the survey. Firstly, advance
notification to the potential respondent
makes them aware that a survey is
coming and they are thus expecting it.
Explaining why the survey is being
undertaken, ensuring confidentiality and
saying what will happen as a
consequence also helps improve
Research Tip:
Estimating what response
rate will be achieved by a
survey is very difficult due to
the wide variation in types of
survey. However, the figures
below provide an initial
estimate of what response
rate might be expected:
•
•
•
•
Face-to-face
Telephone
Mail
Internet
60-90%
40-70%
30-60%
20-50%
response rates.
Offering an appropriate incentive to complete all the questions is often
used. This could range from entry to a draw for some high value article,
through including some lower value voucher or discount with the survey to
Copyright © Business Development & Research International 2010, All Rights Reserved
31
sending a report of the completed results. The survey objectives and
budget will influence what approach is adopted.
Providing an end date by which all responses must be in and sending
regular reminders to non respondents also increases response rates. This
includes calling back people who were originally not available or who
requested a later call.
Of those surveys that are returned, not all will be useable due to
incomplete answers or errors. The designer must determine if a partially
completed survey may be included in the results or not, remembering that
a non completed answer will reduce the sample size for that question.
Copyright © Business Development & Research International 2010, All Rights Reserved
32
10. Pretest Your Survey!
Page 5
The last steps before actual distribution of the questionnaire include:
1. A review by colleagues and potential data users, and
2. A pilot or pretest of the survey instrument to be used.
For comprehensive pretesting, a copy should be submitted to a
representative cross-sample of the population to be surveyed through the
specific mode chosen to conduct the survey. Some preliminary data
analysis (even if hand calculated) should be attempted to check both
design and coding procedures.
It often is useful to run two or more versions of the questionnaire to
determine which version will do the best job. This may include variations
on questions. It is also advisable to ask the pretest respondents about the
layout of the survey and the clarity of the questions.
In general, a pretest is administered to ensure:
1. Ease of administration of the survey,
2. Method of survey delivery to be employed works smoothly,
3. Instructions are understood,
4. Questions are easily understood,
5. All important questions have been asked, and
6. Data can be easily collected, coded and analyzed.
This final step before the main questionnaire is undertaken is often
omitted, either for time or cost reasons. However, it is vitally important
that it is done to ensure that your questionnaire is understandable by your
respondents. Any feedback from the pretest may then be incorporated
Copyright © Business Development & Research International 2010, All Rights Reserved
33
into the final version to ensure that the results obtained will fully satisfy
the survey objectives.
Copyright © Business Development & Research International 2010, All Rights Reserved
34
11. Checklist
What You Need To Do
• Determine if a new survey needs
to be undertaken – will existing
results satisfy your needs?
• Write down the problem to be
solved, the survey objectives and
the decisions which will be made
as a result
• Identify the target population
and pick an appropriate sample
• Determine budget and how the
survey will be conducted
• If using the internet, use the free
version of SurveyGizmo
• Ensure the questionnaire is
relevant and of interest to the
target population
• Ensure the layout and
instructions are clear, and
question sequence is logical
• Ensure the question words are
simple, familiar and clear
• Ensure the respondent is the
best person to answer the
question and ask about specifics
rather than generalities
• Choose open questions for
expansive answers and insight,
but closed questions for easier
data analysis of many questions
• Use rating scales to measure the
degree of feeling or attitude etc
• Improve response rates through
advance notification to potential
respondents, offering an
incentive and sending reminders
to non respondents
• Ensure you pilot or pretest your
questionnaire to confirm it is
clear and fulfils your purposes
• Analyse the results in depth
Beware!
• Check how old any secondary
data is, how relevant it is to your
issue and verify the source
• If specific survey objectives are
not set, you will have neither a
roadmap of what to ask nor
clarity of destination
• Small sample sizes will produce
statistically unsound results
• Not allocating sufficient funds to
ensure the results are actionable
• Internet survey packages which
do not allow data download
• Avoid asking too many questions
– keep to ‘need to know’ rather
than ‘nice to know’
• Don’t put demographic questions
at the beginning as this will put
respondents off
• Avoid colloquialisms and slang
•
Avoid asking two things in the
same question or asking leading
questions
•
Use no more than 2-3 open
questions as respondents view
them as more difficult and time
consuming to fill in
Avoid unequal scales which are
weighted too much to one side
Don’t assume everyone will
answer your questionnaire and
ensure your sample is large
enough to account for the likely
response rate.
Don’t miss out this step and go
straight into your main survey to
save time and costs
Not taking appropriate action!
•
•
•
•
Copyright © Business Development & Research International 2010, All Rights Reserved
35
Glossary
• Bias (error): Distortion or unreliability in survey results. All surveys contain
some bias. Bias is increased when the respondents (persons answering the
survey) are not representative of the population being questioned, when
questions are poorly written or misunderstood, and when the researcher uses
inappropriate techniques to analyze the data.
• Census: A study using all available elements (members) of a population.
• Coding: The assignment of numerical (or other) values to individual questions
and answers on a survey instrument (questionnaire) to facilitate
statistical analysis of the information.
• Data: The collection of observations and information resulting from the survey
process.
• Element: The basic unit about which survey information is sought (i.e., person,
business, household, car, dog, etc)
• Instrument: The tool or device used for survey measurement, usually
a questionnaire.
• Nonresponse: Unit nonresponse refers to the refusal of persons selected to be
sampled to participate in a survey (i.e., person does not return the mail
questionnaire). Item nonresponse refers to selected questions left unanswered
by the person responding.
• Population: The universe or collection of all elements (persons, business, et
cetera) being described or measured by a sample.
• Pretest or Pilot: An initial evaluation of the survey design by using a small
subsample of the intended population for preliminary information.
• Questionnaire: A measuring device used to query a population/sample in order
to obtain information for analysis.
• Response Items: The various answer choices provided on a survey instrument.
• Response Rates: The percentage of surveys/ questionnaires completed from
the total sample queried.
• Respondent: An element or member of the population selected to be sampled.
• Sample: Any portion of the population, less than the total.
Copyright © Business Development & Research International 2010, All Rights Reserved
36
• Survey: A process of inquiry for the purpose of data collection and analysis
using observation, polls, questionnaires, and/or interviews.
• Statistics: Descriptive measures based upon a probability sample.
Copyright © Business Development & Research International 2010, All Rights Reserved
37