Download Report

www.regulatoryreform.com
Session 12
How to Collect Data for RIA
Scott Jacobs
Managing Director, Jacobs and Associates
Regulatory Impact Analysis Training Course
Office of the Presidency
South Africa
1-5 September 2008
Copyright held by Jacobs and Associates. This material may not be reproduced without permission.
1
www.regulatoryreform.com
Data collection: indicator of good RIA
 You will usually need much highly specific data
that is tailored to the questions raised by the
specific regulation.
Copyright held by Jacobs and Associates. This material may not be reproduced without permission.
2
www.regulatoryreform.com
Plan ahead for data collection
 Good data collection strategies are essential.
 Plan in advance and develop an inventory of
data sources.
 Much data needed is held by the regulated
community. Data collection is a public-private
task. You must develop working relations with
private sector partners IN ADVANCE.
 Define standards of data acceptability IN
ADVANCE, as well as the quality control
process for data use
Copyright held by Jacobs and Associates. This material may not be reproduced without permission.
3
www.regulatoryreform.com
Identifying data needs
 RIA must start as early as possible to leave
time for identification and satisfaction of data
needs. This can be a time-consuming part of
the RIA process.
 EC RIA Guide: “The first step in the
preparation of the Roadmap will be to
determine what data are available, what
complementary data are needed, and how
they will be produced”
 But identification of data needs and collection
of data will continue through much of the RIA
process
Copyright held by Jacobs and Associates. This material may not be reproduced without permission.
4
www.regulatoryreform.com
Data needs are defined and satisfied
throughout RIA process
Problem Definition and Risk Assessment
1. Definition and refinement of the problem to ensure the broadest possible range of potential
solutions
2. Profiling the magnitude, risk levels, and distribution of the problem across Member states,
demographic groups, economic sectors, and sizes of firms, with trends in a relevant time
period (first data collection, risk assessment)
3. Establishment of a baseline
Options Selection and Impact Assessment
4. Initial identification of categories of economic impacts, based on profile and baseline
5. Initial consultation with sectoral experts in business associations in high-impact Member
states for validation, issue identification (second data collection)
6. Selection of regulatory options to be considered
7. Selection of method
8. Data collection on detailed benefits and costs of options through business surveys and
other data sources (third data collection)
Presentation of analysis and consultation
9. Analysis, comparison of options, and presentation of impact assessment to the stakeholders
(fourth data collection)
10. Refinement of impact assessment after consultation
Copyright held by Jacobs and Associates. This material may not be reproduced without permission.
5
www.regulatoryreform.com
Steps in identifying data needs
1.
Define and refine the problem to ensure the
broadest possible range of potential solutions

Definition of the problem is the foundation of the
analysis, since all options and scenarios
developed in the analysis will be judged on how
cost-efficiently they resolve the underlying
problem. The scope of the problem definition will
define the scope of data needs.
–
–
Some perishable foods are not transported in refrigerated
trucks (narrow data needs)
OR
Some perishable foods are not maintained at mandatory
temperatures during transport (wider data needs)
Copyright held by Jacobs and Associates. This material may not be reproduced without permission.
6
www.regulatoryreform.com
Steps in identifying data needs
2.
Profile the risk level, magnitude and distribution of the
problem across economic sectors and sizes of firms,
with trends in a relevant time period

A map of the distribution and extent of the problem across relevant dimensions
can be assembled to determine risks, the scope and trends of the problem.
Scope of profiling must be carefully selected to use limited resources to greatest
effect.

Profiling the problem can be done in terms of:
–
–
–
–
–
–
–
–
–
–
–
–
Level and severity of risk
Demographic distribution of risk
Geographic distribution of risk
Causal relationships between risk and economic activities
Number of businesses and employees potentially affected by regulation, and trends over time;
Value-added of sectors potentially affected by the regulation, and trends over time;
Profitability of the sectors potentially affected by the regulation;
Export performance of the sectors potentially affected by the regulation;
Market share in the products or services affected by the regulation, and trends over time;
Value-added of sectors potentially affected by the regulation, and trends over time;
Comparison with other country performance in these dimensions, to provide a international benchmark
to judge performance;
Sizes of firms.
Copyright held by Jacobs and Associates. This material may not be reproduced without permission.
7
www.regulatoryreform.com
Steps in identifying data needs
3.
Establish the baseline

Get an initial view of trends across time

Determine which baseline parameters are most
important to these trends.

Document the factors that could change the
baseline, such as likely evolution of the market,
changes in external factors affecting benefits
and costs, and changes in other regulations
Copyright held by Jacobs and Associates. This material may not be reproduced without permission.
8
www.regulatoryreform.com
Steps in identifying data needs
4.
Identify likely categories of impacts, based on
profile and baseline

Economic impacts of regulations fall into several categories, each
important under good RIA practice:
–
–
–
–
–

Benefits of regulations can be categorized as:
–
–

Budget or fiscal impacts.
Operating Costs.
Capital Costs
Dynamic Costs
Distributional effects
Direct (contributing to resolving the problem)
Indirect (all other positive effects)
This part of the assessment can conclude with a matrix mapping
out the regulation against the categories of costs and benefits, and
assigning each a priority level: Important, moderate, less important,
not applicable. This mapping can drive the data needs assessment.
Copyright held by Jacobs and Associates. This material may not be reproduced without permission.
9
www.regulatoryreform.com
Mapping regulatory costs - reform of the sugar
market (from EC IA guide 2005)
Copyright held by Jacobs and Associates. This material may not be reproduced without permission.
10
www.regulatoryreform.com
Steps in identifying data needs
5.
Initial and limited consultation with sectoral
experts in business associations in high-impact
area for validation and issue identification



Use expert groups to validate the assumptions made
with respect to the problem definition, the likely
categories of economic impacts, the profiling of the
problem, and the baseline projections;
Collect information on data priorities and identify data
sources;
Collect preliminary information on possible regulatory
options, and the potential impacts of each option.
Copyright held by Jacobs and Associates. This material may not be reproduced without permission.
11
www.regulatoryreform.com
Steps in identifying data needs
6.
Select regulatory options to be considered
 Due to the exponential increase in the data
needed as we expand options, we will generally
limit the choice of options to 3-4 in the impact
assessment, and explain why those were chosen.
 In general, consider, for inclusion in the cost
assessment, the feasibility and potential cost
implications of four kinds of options:
–
–
–
–
changing the stringency of the regulation or moving to
performance standards to estimate effects on costs per
unit of benefit;
information approaches, in which incentives are changed
by correcting information failures;
voluntary agreements and other forms of co-regulation;
economic incentives.
Copyright held by Jacobs and Associates. This material may not be reproduced without permission.
12
www.regulatoryreform.com
Steps in identifying data needs
7.
Selection of method

Data needs differ according to the analytical method. The method encompasses
choices about the scope of the analysis (what kinds of costs will be included), the
time period of the analysis (and how time will be valued), the means of counting
impacts, and the means of presenting impacts.

The analysis should generate policy-relevant results that are meaningful to
decision-makers. Costs should be compared to meaningful benchmarks, such as
investments or profits, and to a metric of benefit, such as unit of abatement of the
problem. Data collection should aim for such results.

To assess compliance costs, you can use straightforward cost estimation
techniques based on four kinds of data:
–
operational costs, over a relevant period of x years, of meeting various levels of
regulatory stringency or complying with various regulatory options;
–
capital costs (depreciation and net opportunity cost of invested capital), over a
relevant period of x years, of meeting various levels of regulatory stringency;
–
estimation of non-market values, such as foregone market opportunities, using
expert assessment, comparison with other markets, and historical trends;
–
estimated effects of foreseeable technological innovation on these costs over a
relevant future period.
13
Copyright held by Jacobs and Associates. This material may not be reproduced without permission.
www.regulatoryreform.com
Steps in identifying data needs
7.
Selection of method (continued)

Decide how you will use these data to generate the following kinds of
indicators:
–
–
–
–
–
Net present value of total costs, per sector (using the cost of capital as the
discount rate);
Total sectoral (or activity-wide) annual compliance costs
Annual per-establishment compliance cost, by size of firm
Annual per-employee compliance costs, by size of firm
Compliance costs as:
●
●
●



a percentage of turn-over;
a percentage of profits;
a percentage of annual investment in the sector.
Cost-effectiveness analysis to compare these costs to potential levels of
risk reduction. This will allow us to obtain the costs per unit of abatement
for sectors, firm size and regulatory options.
Sectoral models that assess affects on sectoral output by using inputoutput analysis that incorporates the effects of regulation on cost
variables.
For economy-wide effects, use more synthetic methods that project intersectoral effects and impacts of higher production costs on business
decisions about employment and investment.
Copyright held by Jacobs and Associates. This material may not be reproduced without permission.
14
www.regulatoryreform.com
Collect data
8.
Collect data on detailed benefits and costs of
options through a variety of means




Data collection is the single most costly and timeconsuming step of the RIA
Be opportunistic – use existing databases and
research to the maximum extent
Design new data gathering efforts on the margin to
fill in data gaps
Shift data costs to stakeholders through open
consultation techniques
Copyright held by Jacobs and Associates. This material may not be reproduced without permission.
15
www.regulatoryreform.com
Common data collection techniques (1)
Technique
Pros
Cons
Literature searches and
existing databases
Fast and cheap. Credible because it
draws on scientific expertise.
Unfocussed. Cannot usually
answer questions specific to the
regulation. Gaps in the literature
increase uncertainties.
Meta-studies (combining
several studies into one
big study)
Can compensate for weaknesses in
individual studies
Limited in the range of information
(cost data but little benefits data),
reliability is suspect, costs can be
high. Difficult to explore options.
Experts from science
committees, academia
and industry (e.g.
interviews)
Fast and cheap, can provide good
quality cost and benefit data in
response to specific questions. Can
explore alternative approaches.
Vulnerable to bias and to existing
knowledge which is not specific to
the regulation. Must diversify
sources.
Passive consultation
(publication for comments)
Fast and cheap for collection of
wide variety of data and ideas about
better regulation.
Vulnerable to bias and poor data.
Not conducive to dialogue. Data
gaps.
Copyright held by Jacobs and Associates. This material may not be reproduced without permission.
16
www.regulatoryreform.com
Common data collection techniques (2)
Technique Pros
Cons
Business surveys
Fast data collection, can be carried out
for targeted groups, can collect
information otherwise unavailable.
Limited in the range of information
(cost data but little benefits data),
reliability is suspect, costs can be
high. Difficult to explore options.
Focus groups
Business Test Panels can be fast and
provide a wider range of cost and
benefits data. Can explore alternatives
and options easily.
Data quality can be a problem.
Requires upfront preparation and
good relations with stakeholders.
Model enterprises that
“represent” the
affected population
Fast and cheap way to collect cost data.
Simplistic and vulnerable to errors
in assumptions. Must be widely
consulted.
Modeling:
Econometric modeling
(input-output, general
equilibrium models,
environmental impact
assessment models,
Microsimulation
models).
Best approach for estimating welfare
Costly and vulnerable to
changes for particular groups, and for
assumptions. Macro models lack
estimating second-order effects through detail on micro interventions.
the economy. IA on reform of Common
Unless existing model is available,
Market for Sugar used modelling
analysts have limited access to the
analysis to determine macroeconomic
tools and resources needed to
effects of different reform scenarios, as
adopt a modeling approach.
well as qualitative multi-criteria
Difficult to use economic models to
evaluation of broader impacts and
collect benefits data.
17
stakeholder views.
Copyright held by Jacobs and Associates. This material may not be reproduced without permission.
www.regulatoryreform.com
How can you compensate if you
cannot find enough data?
 In the absence of adequate valid data, clearly identified
assumptions are necessary for conducting the RIA.
 Transparency is even more essential to test the
assumptions. Analysis of the risks, benefits, and costs
associated with regulation must be guided by the
principles of full disclosure and transparency.
 Shift data costs to the stakeholders by asking
questions in the consultation period. Those affected by
regulations have the incentives to provide the data
necessary to complete RIA.
Copyright held by Jacobs and Associates. This material may not be reproduced without permission.
18
www.regulatoryreform.com
Other means of dealing with data
uncertainty
 Sensitivity analysis: estimate a range of possible
effects. High and low estimated benefits can be
interpreted as a range of potential effects. When we
lacked direct evidence on uncertain values, we can
deal with the uncertainty by choosing values that
generated lower-bound estimates of benefits.”
 Expert scientific judgement: present wide range of
scientific studies. Try to find a scientific consensus by
creating expert panels to review scientific evidence.
 Independent and external peer review of data and
analytic results
Copyright held by Jacobs and Associates. This material may not be reproduced without permission.
19
www.regulatoryreform.com
Guard against “data capture” by diversifying
data sources and consulting widely
 Since stakeholders will provide much of the needed
data, the risk is high that the RIA will be biased.
 This risk can be managed by diversifying data sources,
a check and balance approach.
 Data biases can also be detected by being completely
transparent. When data are weak, the more external
review that RIA receives, the better it is likely to be.
Copyright held by Jacobs and Associates. This material may not be reproduced without permission.
20
www.regulatoryreform.com
Set data quality standards
• Defining standards of data acceptability in advance, as
well as the quality control process for data use, can
avoid “junk RIA” and boost RIA credibility and reliability.
• The most common data quality standards:
–
–
–
–
–
Transparency in raw data and assumptions
Use of “best available data”
Reproducibility,
Acceptance by independent experts
Collected according to good statistical practices such as random
sampling
– Presentation of best estimates reflecting expected values (as
– distinct from “worst case” or conservative estimates)
Copyright held by Jacobs and Associates. This material may not be reproduced without permission.
21
www.regulatoryreform.com
Summary of data collection and
presentation practices for high quality RIA
1.
Plan ahead and create public-private relationships
2.
Map out data needs and collect data throughout the RIA
process in an iterative process
3.
Consider a variety of methods to collect scarce data, and shift
data costs through structured stakeholder consultation
4.
Use good data quality techniques. Carefully document data.
Leave a trail in the RIA that a careful reader can follow to
connect the input data with the outputs (i.e., the estimated
effects)
5.
Make weaknesses transparent and deal with uncertainties
openly
6.
Use diverse sources to guard against “data capture”
Copyright held by Jacobs and Associates. This material may not be reproduced without permission.
22