1. business and industrial
2. company
3. annual report

HOW TO REVIEW CONSULTANCY REPORTS
Version edited 16/01/2013
TMEA and TMEA’s partners frequently contract consultants to conduct baselines, develop strategies,
deliver training, design policies, undertake research and write reports. Consequently, it is essential
to ensure that the outputs are relevant to your work, methodologically sound, and of a high quality.
To achieve this, you must carefully review each consultancy report and give appropriate feedback.
This report outlines key factors to consider when reviewing consultancy reports, illustrates this with
three (fictitious!) examples, and briefly discusses how feedback should be managed. It accompanies
the main TMEA monitoring guidelines, which can be found here.
Overall guidance:




Take time. Reading a report properly (not just
the executive summary and conclusion)
requires time and effort. Go somewhere that
you will not be disturbed, turn off your mobile
phone and email, and read through the full
report carefully.
Be respectful, but demanding. Consultants
are expected to possess experience,
professionalism and knowledge. However,
never assume that they are always correct.
There are many examples of sub-standard
consultancy reports – ensure that yours is not
one of them.
From bloodandmilk.org
Consult stakeholders. Give everyone involved
the chance to give feedback, including TMEA staff, partners, or government agencies. Set
deadlines for collecting feedback and ensure that they are met.
Use TMEA colleagues who have relevant experience and knowledge. Colleagues can review
consultancy reports, or suggest other sources of information. The K&R team may also be able to
help with issues like logic and clarity of reports, or research methods.
Step One: Did the Consultancy Achieve Its Objectives?
This is a crucial question. A fascinating, well-designed and thoughtful report
is useless if it does not meet your objectives.
Before reading the report, reread the original terms of reference. Remind
yourself exactly what you asked the consultant to do, and ask whether the
terms of reference really captured the purpose you had in mind.
While reading the report, consider the following points:

Completeness. Ensure that the consultant
requirement specified in the terms of reference.
fulfilled
every

Audience. Briefly think about the intended audience. Is the language and style of the report
appropriate for them? For example, jargon may only be comprehensible to specialists.
EXAMPLE ONE – Customs Reform Baseline
You have commissioned a baseline study into your customs reform project. The aim is to
establish the time it took to process imports before the start of the project, in order to provide
a comparison point for the time taken at the end. (For more on baselines, see the how-to guide
here.) The project has been running for a while, so you have asked the consultant to use
historical data to establish the baseline.
The initial draft report contains a glowing assessment of your work so far. Having spoken to
numerous stakeholders in the public and private sector, it concludes that all stakeholders
believe that it now takes less time to process customs goods. The report also suggests several
important areas for improvement, and gives valuable suggestions for future work.
Are you happy with this report? What potential issues might there be? How would you deal
with this? Take a moment to think about this, and then see our thoughts below.
____________________________________________________________________
Although the information provided was valuable (and encouraging), it did not address the issue
you specified. The aim was to get baseline data – not to evaluate the positive or negative effect
of the project. Moreover, the consultant appears to have relied principally on stakeholder
interviews, while quantitative analysis may have been more appropriate. Although their report
may contain valuable insights, it will not help establish the impact of your project.
In this case, it would be necessary to meet with the consultant to review the terms of
reference, and highlight areas where you feel information is missing. Explain how you plan to
use the report, and why the baseline is important. Ask to see a second draft before receiving
the final report.
Step Two: Is the Methodology Sound?
It is necessary to evaluate the strengths and limitations of the methodology. There is no perfect
In this case,but
the consider
consultantwhether
has not the
donetechnique
the job you
requested.
The aim wasfor
to get
methodology,
selected
is appropriate
the baseline
research
data
–
not
to
evaluate
the
positive
or
negative
effect
of
the
project.
Moreover,
they
were
questions. If you are uncertain, then ask a colleague or the Knowledge and Results team for further
asked to analyse data, but they appear to have relied principally on stakeholder interviews.
guidance.
Consequently, their findings cannot be compared against results from your planned end-ofThisproject
sectionevaluation.
briefly describes
three
broad
methodologies;
reviews,itqualitative
research,
Although
their
report
may contain literature
valuable insights,
will not help
you
and establish
quantitative
research.
the impact of your project.
Literature
The purpose
of the literature
is to
identify what
relevant
In this Reviews.
case, it would
be necessary
to meet review
with the
consultant
to review
the information
terms of
already
exists.
You
should
have
completed
a
brief
literature
review
before
commissioning
the
reference, and highlight areas where you feel information is missing. Give more detail about
consultancy,
but they
will often be
expected
and are
strengthen
the expected
methodology,
and
the waytoindeepen
which you
planningit.on using this information.
Ask them to submit a second draft for approval before moving on to the final draft.
It is important that the literature review does not simply repeat the literature. It should analyse it;
stating
emerge,
and highlighting
of consensus
disagreement.
so doing,
Youthe
maythemes
wish tothat
review
the terms
of referenceareas
to ensure
that youorwere
sufficientlyIn
clear
about it
your expectations.
should evaluate the quality of the documents reviewed. It should actively link the literature review
back to the question posed in the terms of reference, and to the conclusions that the report will
draw.
Quantitative research measures the situation in numeric terms. It often uses closed-ended
questions with limited potential responses, and employs statistical techniques to detect significant
differences between different groups of respondents. It often requires large samples, weighted to
represent the population that the study is interested in.
Good quantitative analysis should use a large enough sample size to justify their conclusions. A
survey of two people, for example, would not tell you very much. The study should justify the
sampling methodology, following national or international standards where appropriate. Any
assumptions that are made in the quantitative data analysis should be explicitly listed and justified.
For example, TMEA frequently assumes that a reduction in time to trade in East Africa leads to a
reduction in cost. Studies that rely on that assumption should make this explicit, say how much they
assume each day of delay costs, and reference their sources for this assumption. Limitations in the
analysis should be clearly explained.
Qualitative research seeks to uncover the context, perceptions and quality of, as well as opinions
about, a particular experience or condition. Data collection methods are likely to employ a more
participatory approach through the use of open-ended questions that allow respondents to expand
on their initial answers and lead the discussion towards issues that they find important. Samples
tend to be smaller.
Good qualitative analysis should explain who was interviewed, and why they were chosen. When
reading, consider whether they are appropriate for the report’s purposes. The consultant should
describe the instrument that was used, how it was designed, and how the results were analysed.
In practice, reports are likely to use elements of all three methodologies. The literature review is
essential to learn from previous work, and ensures that you contribute something new. Quantitative
data measures the extent of a problem, and provide a simple understanding of how it has changed.
It tells you what happened, when, and to who. Qualitative data will supplement this with insights
from partners, beneficiaries and staff, addressing the causes, sustainability, and impact of this
change. This explains how and why things happen
Whichever methodology is used, the consultant should be explicit about what they did, and how
they did it. They should be willing to provide you with tools used (such as questionnaires or
workshop guides), and the raw data from their analysis (either quantitative information, or
transcripts from interviews.) Moreover, they should be clear about the limitations of the research.
Perhaps most importantly, the analysis and recommendations should clearly link to the literature
review and primary research. It should explain the findings, make explicit the relevance to the
problem defined in the terms of reference, and explain the implications for your programme.
This section has only given a brief snapshot of ways to critique research. A short and readable guide
on quantitative research is available here and on qualitative research here. Although originally
designed for medical students, they are applicable to any research projects. See Annex One for
examples of specific methods that can be used.
Step Three – Other Factors to Consider
There are numerous signs of a high-quality report, and only a couple are listed here. They include:
 Spelling, grammar, and layout. The report should be professionally presented, with correct
spelling and grammar. Jargon should be avoided where possible, and the language should be
clear and accessible.
 Executive summary. Every report should start with an executive summary. This will be the only
section many people read.
 References. A good report should cite its references, in footnotes or a bibliography. Assertions
should be backed up with sources, so you can check where they get their facts from.
 Length. The shorter the report is, the more likely it is to be read. Sections irrelevant to the main
purpose should be cut out. If relevant but lengthy or specialised, then they can be put in annexes
(for example, detailed methodology descriptions, or questionnaires).
EXAMPLE TWO – Communicating Regional Integration
The Knowledge and Results team commissioned a report into the economic effects of regional
integration, designed for a wide audience. The executive summary contained the following
points:
 Kenya, Rwanda and Burundi are likely to have benefited from convergence to the CET
due to reductions in MFN tariffs. However, the results suggest that the degree of IIT in
the EAC region is relatively low
 Using an indicator of import similarity to gauge the degree of source switching between
preferential and non-preferential partners gives evidence of some trade diversion
occurring.
What are the potential issues with this report? How might you solve them? Take a moment
to think about this, and then see our ideas below.
________________________________________________________________________________________________________________________________________________________________
The report sounds like it is well-researched and detailed. However, it is not easily
understandable by a non-economist. The executive summary should be the most accessible part
of the report, since it is likely to be the most widely read section. The current terminology will
limit the audience of the report, and so reduce its impact.
Meet with the consultant to emphasise the importance of readability and accessibility. Highlight
areas in the report which you feel meet these criteria, to provide a model for the rest. Consider
providing an example of a report that successfully balances complexity with readability.
Review the final draft when it arrives, to ensure that they complied with this request. You could
consider summarising the report when you email it round, to help other readers understand the
main points. Alternatively, other methods of reaching wider audiences – like policy briefs or
short overviews – could be produced and disseminated.
Step Four – Give Feedback to the Consultant
When giving feedback to the consultant, consider the following points:




Feedback should be phrased constructively – highlight the positive aspects of the report as well
as those that you believe need to be adjusted.
It may be wise to ask for a second draft of the report before receiving the final version, to give
an additional opportunity for improvements to be made.
Follow up on comments to ensure that they are addressed in the final draft of the report – don’t
assume that they will be.
Finally, if the consultant does not deliver what you wanted, evaluate whether your expectations
were realistic. Perhaps you asked for too much, or the terms of reference did not specify exactly
what you wanted. If you have a bad experience with a consultant, think about how you can learn
from it.
EXAMPLE THREE – Advocating for Weighbridge Removal
Your team advocates for the removal of weighbridges in East Africa. You commissioned initial
research to help design your advocacy strategy. The consultant surveyed 400 truck drivers, and
statistically analysed differences between countries. They show which countries have the most
weighbridges, how long drivers spend at them, and how much this costs businesses. The report
is clearly well-researched, and the consultant explains their methodology clearly.
What are the potential issues with this report? How might you solve them? Take a moment to
think about this, and then see below for our answers.
_______________________________________________________________________
There are many positive aspects to this report. However, it may not provide new information.
Previous reports have highlighted that weighbridges are a serious concern in East Africa (such
as the EABC Business Climate Index, or the CDS Report.) It may be that the consultant (or you)
did not conduct a literature review before starting the research.
Moreover, the advocacy strategy requires not just evidence of a problem, but understanding
why the problem exists. Given the huge number of previous efforts to eliminate weighbridges
(for example, the EAC Time Bound Programme on NTBs), why haven’t they been eliminated?
Who is supporting them, and who else can influence the process? This would specify the target
group for your advocacy and key messages. It may have been more appropriate for the
consultant to use qualitative methods rather than quantitative.
Meet with the consultant to clarify the objectives of the consultancy, and explain exactly what
you need from the report. Emphasise that the data they collected can still be extremely useful,
but ask for additional information to meet the needs of the advocacy strategy.
Further Support
There are a large number of internet resources that discuss research methodologies. Simply search
for the terms you are interested in, or consult Wikipedia. A short and readable guide on quantitative
research is available here and on qualitative research here. Also consider reading MSF’s guide to
qualitative research.
Other TMEA resources include:
 TMEA MEL Guidelines
 How to Design a Results Chain
 How to Plan a Baseline
 How to Develop a Monitoring Plan
 How to Assess Contribution
 How to Find Sources of Information
 How to Monitor Through Observation
 How to Review Progress Reports
 How to Have Effective Meetings
Annex One
Qualitative Methods
Quantitative Methods
The table below shows some common research methods. In practice, combinations of these
methods are likely to be used. For example, a survey may contain both closed and open questions.
Method
Analysis
of
data
from
government
IT
systems,
e.g.
ASYCUDA.
Survey with
closed
questions
(with
a
prescribed list
of answers)
Focus group
discussions
Case studies
Benefits
 Accurate
and
objective
measurements.
 Data can be analysed in multiple
ways, e.g. examining time taken by
different types of trucks.
Limitations
 Data may be difficult to obtain.
 Data often contains errors or mistakes,
which can limit its use or require
cleaning.
 May require specialised skills to analyse.
 Data can be easily analysed to
establish percentages, statistical
analysis, etc.
 Often simple to collect.
 Responses will be limited to those listed
in the questionnaire, as respondents
must select from a list of options.
 This means it may be an inappropriate
method for complex or sensitive
subjects.
 Requires skilled facilitator.
 Some participants may feel shy or
unwilling to speak in a group.
 Data difficult to analyse.
 Group interaction can lead to
insights and data that may not
arrive from individuals.
 Allows for exploration of complex
questions
 Can provide rich information on
broad, complex questions.
 Useful for demonstrating impact.
 Information
generalizable.
 Subjective.
not
necessarily