1. No category

Manual and Tools for Call Implementation
Contents
Objective of Manual
1. Call Planning
2. Call Preparation
3. Submission
4. Evaluation
5. Funding Decisions
6. After the Call
7. Attachments
Page 1 of 154
Objective of the Manual
The central goal of this online manual is to provide the ERA-NET community with a
sound basis for the implementation of transnational calls for proposals. You will find
general information, practical examples and templates.
These optional descriptions, recommendations and guidelines are based on
experiences and realistic scenarios, provided as an interactive compilation of modules
which reflect the different phases in the planning and operation of joint calls.
The contents are based on past and ongoing ERA-NETs and have proven useful for
specific purposes. The documents may be used as blueprints, references or inspiration
for future joint call activities.
There is no doubt that the ERA-NET instrument is suitable for flexible use;
consequently material provided here may be appropriate for various scenarios and
constellations whereas less appropriate for various other scenarios and constellations.
All information is based on the input from ERA-NET coordinators. This compilation was
developed by ERA-LEARN to simplify and standardize the execution of joint calls as
much as possible.
Page 2 of 154
This manual was first published by ERA-LEARN in March 2010. We invite you to check,
comment and provide us with feedback. Updated versions are planned on an annual
basis.
Latest update: March 2014
A printable version of this manual is currently under revision. Please note that the
printable version does not replace the online version, as it does not contain any
hyperlinks or templates.
A printable checklist for call implementation is also provided.
The PLATFORM project, a network of KBBE related ERA-NETs, held a master class on
"best practices and recommendations for effective and cost-efficient call management
in bio economy related ERA-NETs" in June 2013. Recommendations from the call
management experts can be downloaded here.
Page 3 of 154
1. Call planning
The following information regarding the implementation of joint calls refers to the
aforementioned experiences of various ERA-NETs with coordinators in different
countries and covering different research topics.
In the past realized joint calls have shown that – respecting the thematic focus or
type of funding mode chosen – their implementation comprises the following stages:
1.1 Call process and administration
1.1.1 Call administration
1.1.2 Overall time frame
1.1.3 Submission of proposals
1.1.4 Evaluation of proposals
1.1.5 Funding modes
1.1.6 Financial commitment
1.1.7 Funding decision
1.1.8 Start of funded projects
1.1.9 Monitoring
1.1.10 Administrative conditions and necessary documents
1.1.10.1 Sample of Memorandum of Understanding
1.2 Timing of the call
1.2.1 Call schedule
1.2.2 Overlaps with other calls
1.2.3 Fine tuning of call milestones
1.2.4 Plan necessary meetings
1.2.5 NETWATCH call calendar
1.3 Scope of the call
1.3.1 Method for definition of scope
1.3.2 Overlaps of topics with other calls
1.3.3 Identification of target groups
1.3.4 Definition of targeted projects
1.3.5 Submit your data to Netwatch
Contents Page
Page 4 of 154
1.1 Call process and administration
1.1.1 Call administration
One of the first steps before preparing a joint call will be the definition and
implementation of the basic structures for call management.
It has proven useful to install a "call secretariat" as a central office which organises
the operative steps involved in the call administration; if the joint call is carried out
under a specific Work Package (WP) in your ERA-NET it will be appropriate to put the
WP leader in charge of the call secretariat. Depending on the experience and expected
workload this call secretariat could be assisted by representatives from the funding
agencies that are participating in the call.
The workload and time frame of activity (installation period) of the call secretariat
needs to be concisely defined and thoroughly planned in advance, assuring the strict
functionality which applicants expect from a funding body even if the ERA-NET
consortium is not a formally established legal entity.
The usual workload will also include follow-up activities and covers all
or parts of the following duties:
• Prepare the necessary call documents
• Provide the required call information & promote the call
• Provide a system for submission of proposals and distribution to funding
organisations
• Coordinate the evaluation of proposals
• Prepare a system for monitoring the call implementation
• Prepare a system for monitoring the funded projects
Additional administrative considerations required in ERA-NET Plus
calls:
•
•
•
•
Rules of 5: participants (funding organizations) from 5 Member States or
Associated Countries; Minimum total budget of the joint call 5 M€; Maximal
duration 5 years
Selection of the topic for an ERA-NET Plus with adequate budget allocations
Projects co-funded by national/ regional research funding agencies and the
European Commission (EC)
Impact of the ERA-NET Plus Grant Agreement on the individual contract with
Page 5 of 154
•
•
•
•
•
•
beneficiaries
Financial contribution from EC of max 33% of the call budget – no
management or coordination costs for the duration of the action directly
funded by the EC, although financial audits and reporting obligations
are binding only the beneficiaries (funded projects)
Agree on the use of top-up funding and a distribution among the
partners (Consortium Agreement, CA)
Use of a mixed mode as a financial model in order to ensure the selection
list for projects to be funded can follow the ranking list
Change in the composition of the project consortium members is limited
until the call publication
Any request for the addition or removal of a beneficiary shall be submitted
to the EC no later than the publication of the joint call
Implementation and management of the joint call must involve a 2-step
procedure (mostly pre-proposals, full proposals) and international peer
review evaluation (based on excellence criteria).
Additional administrative considerations required in ERA-NET Cofund
calls:
• Compulsory for each Grant Agreement (GA): Implementation of a co-funded
joint call with EC, one co-funded call per Grant Agreement
• Duration of the GA: 5 years
• Participants (funding organizations) from at least three Member States or
Associated countries
• Type A ERA-NET Cofund agreement: Implementation of a single co-funded
joint call – financial support paid to third parties (research projects) in cash
• Financial EC contribution of max 33 % of the co-funded call budget
• No management or coordination cost directly funded by the EC,
although financial audits and reporting obligations are binding only the
funded projects
• Type B ERA-NET Cofund agreement: In addition to the co-funded joint call
with EC, additional ERA-NET activities including additional joint calls with the
Members States (MS) funding
• Financial support to research projects as well as coordination cost for
additional activities (fixed calculated unit costs/ beneficiary/ year)
• EC contribution of max 33 % in both
• Type C ERA-NET Cofund agreement: For a single co-funded call in kind
contribution possible in exceptional cases (mentioned in the call text)
• with governmental research organizations carrying out the chosen
projects by themselves
Page 6 of 154
• non-reimbursed expenditure, Horizon 2020 rules
• Implementation and management of the EC co-funded joint call:
• Must involve a 2-step procedure
• International peer review evaluation on step 2 is based on the following
criteria: 1) excellence, 2) impact and 3) quality and efficiency.
• Evaluation of proposals on step 2 must involve at least three
independent experts per proposal and will lead to binding ranking list of
fundable transnational projects
• Minimum eligibility criteria for a project: two independent entities from
two MS or Associated Countries.
• Preferable funding mode is a mixed mode (or a real common pot) to
ensure the selection of the best projects
• The use and distribution of top-up funding among the funding
organizations should be agreed in a CA.
Please note: The conditions for call implementation of the EC co-funded joint call do
not apply to additional calls of the ERA-NET Cofund without EC top-up funding.
1.1.2 Overall time frame
In order to agree on a common timeframe at the ERA-NET-network level the
funding agencies have to match and/or adapt their individual programme timelines,
considering the following issues:
• Legal constraints (minimum time required by national/regional law for each
phase of the call)
• Operational constraints of programme managers (minimum time required for
performing each of the single tasks)
• National/regional programme schedules
• Expected number of proposals and the time required to manage the workload
• Evaluation procedures are the most influencing operational and legal factors
on the call timeline, since they represent the most critical constraints set by
all agencies of the network
• Timing of funding decisions
The resulting minimum time frame is the basis for a definite schedule to be further
elaborated during the preparation phase.
In case of ERA-NET Plus Action and ERA-NET Cofund, the timing of the co-funded call
has to be planned in coherence with the assessment and decision making process of
the EC upon submission of the ERA-NET Plus /Cofund proposal. Thus, one should take
into consideration:
Page 7 of 154
Timing of the call to be launched by the ERA-NET-network in relation to the
submission/ evaluation/ negotiation of an ERA-NET Plus /Cofund proposal
with EC and the starting date of the contract. In the co-funded ERA-NET
Cofund call, the consortium must inform the EC 30 days earlier about
launching the call, and the first step of the call (for pre-proposals) must be
open for at least 60 days.
• Timing/ compatibility of reporting periods for the Grant Agreement
(between the EC and the ERA-NET consortium of funding organizations) and
the individual contracts between ERA-NET Plus and ERA-NET Cofund
participants as well as their beneficiaries
•
1.1.3 Submission of proposals
The ERA-NET consortium has to establish a method for collecting proposals from the
applicants and for distributing the submitted proposals to the respective funding
organisations.
Options for such proposal submission systems are in place.
It is a core task of the call secretariat to operate the submission system smoothly.
In case of the ERA-NET Plus and co-funded ERA-NET Cofund call, a 2-stage evaluation
process is compulsory.
1.1.4 Evaluation of proposals
The ERA-NET consortium has to agree on a method for selecting proposals for
funding, as their core activity.
Options and potential scenarios for the evaluation phase are described in this
manual.
In ERA-NET Plus and co-funded ERA-NET Cofund calls, a 2-stage process resulting in a
joint ranking list must be used.
Page 8 of 154
1.1.5 Funding modes
All programme owners participating in a joint call have to agree on a way in which to
fund successful transnational proposals. The question of which funding mode to
implement is a central issue of every call. The final decision for a funding mode that is
used in a specific call will depend on the agencies in the ERA-NET consortium and the
respective flexibility of national/ regional programmes. It has strong implications on
the necessary evaluation system(4.3) and its required outcome, and often also on the
funding contract and funding administration.
In ERA-NET Plus calls, where the projects are co-funded by the national/ regional
funding agencies and the EC, the mixed mode of funding is the one that most ERANET Plus consortia decided to use. In the co-funded ERA-NET Cofund calls, the
mixed mode or the real common pot are the recommended funding modes. In these
co-funded calls with EC, a sound financial mode (real common mode or at least a
mixed mode) have to be used to comply with the requirements for joint proposals
evaluation, ranking and selection in order to be able to benefit from the top-up
funding of the EC. The best research projects are co-funded by national/ regional
funding agencies and the EC.
The modes of funding can be distinguished into the following categories:
Real common pot
Suitability:
This funding mode is suitable for participating funding organisations which
wish to engage in a transnational joint call with an agreed research theme,
with evaluation undertaken by an international expert committee, and where
funding decisions are based on a joint ranking list.
The participating funding organisation accepts that funding decisions are
made by the designated joint decision-making body to ensure funding of the
best quality proposals, in accordance with joint standard rules and
procedures, and irrespective of nationality or place of residence.
Benefit for participating funding programmes:
Participation in a real common pot will enable national funding organisations
to jointly fund the best quality proposals, and excellent resident and nonresident researchers taking part in these proposals.
Participating funding programmes will benefit from expert international
evaluation of joint call proposals, from the development and operation of
transnational rules and procedures.
Page 9 of 154
Commitment:
Funds will have to be earmarked and committed to participation in
transnational calls through a jointly agreed common budget, irrespective of
the national/regional affiliation of applicants.
National/regional funding organisation will, through contribution to the real
common pot, fund the best quality proposals irrespective of nationality,
whereby international expert committe evaluation and subsequent funding
decisions by the designated decision-making body (e.g. Steering Committee)
retain these proposals for funding.
The national/regional funding organisation does not retain control of
funding decisions and funding, and will, depending on the funding decisions,
potentially be funding non-national and non-resident researchers according
to the committed budget. The real common pot requires the commitment of
program owners for cross-border funding and thus requires a system to
administer the distribution of funds at the ERA-NET level.
Administrative efforts and benefits:
Administrative coordination with other national funding organisations will be
necessary to establish joint call procedures, and administrative effort is
needed in order to ensure efficient operation of joint call decisions and joint
funding, in accordance with joint standard rules and procedures.
Cross-border funding is essentially involved. National legal provisions may in
some countries restrict or disallow cross-border funding.
Virtual common pot
Each participating funding organisation will fund its own successful
participants. Funding will not be available from one participating funding
organisation for successful participants from other participating funding
organisations, there is no cross border funding involved.
Evaluation of proposals is undertaken by an international expert committee,
whereas funding decisions and funding is undertaken by individual national
organisations, in accordance with their own standard rules and procedures
Suitability:
For participating national funding programmes which wish to engage in a
transnational joint call with an agreed research theme, with evaluation
undertaken by an international expert committee/evaluation panel, but
which also wish to retain control of funding decisions and funding, in
accordance with own standard rules and procedures.
Page 10 of 154
This funding mode enables national funding organisations to retain
autonomy and control of their own national budget and of own funding
decisions, and does not entail funding of non-nationals and/or non-residents.
Participating funding programmes will benefit from expert international
evaluation of joint call proposals, program owners will benefit from the
internationalisation of their programme portfolia.
Commitment:
National funds will have to be earmarked to guarantee effective participation
in joint calls.
National funding organisation will fund own nationals or residents, where
both international expert committee evaluation and subsequent national
funding decisions retain these proposals for funding.
Even if the contribution of a guaranteed budget is essential for the virtual
common pot, national funds may be selectively increased according to the
national/regional demand of the evaluation result.
Administrative efforts and benefits:
Administrative efforts will be necessary for operation of national funding
decisions and funding, in accordance with own standard rules and
procedures.
No cross-border funding is involved.
Mixed mode
This model is the minimum condition for implementing an ERA-NET
Plus and co-funded ERA-NET Cofund call, but can be used in the
traditional ERA-NET scheme call between Member States as well.
ERA-NET Plus and co-funded ERA-NET Cofund call may receive a European
Commission (EC) financial contribution to top up the call budget. The
topping up may reach 33 % of the total cumulative funding of the joint call
budget provided by the ERA-NET partners.
Suitability:
This mode is designed for ERA-NET-networks wishing to engage in a
transnational joint call with a pre-defined research theme, with evaluation
undertaken by independent international peer review, and with financial
topping up by the EC.
Page 11 of 154
The participating funding organisation accepts that funding decisions are
made to ensure funding of the best quality proposals, irrespective of
nationality, in accordance with joint standard rules and procedures.
The participating funding organisations must agree on a joint ranking list for
funding of projects, and must formally commit to finance the successful
projects.
The rules and minimum requirements for funding organisations to
participate in the EC co-funded ERA-NET Plus and Cofund calls are
given in chapter 1.1.1.
Benefit for participating funding programmes:
Participation in an ERA-NET Plus and co-funded ERA-NET Cofund call
will enable national funding organisations to jointly fund best quality
proposals and excellent non-resident researchers. Participating funding
programmes will benefit from expert independent international peer review
of joint call proposals.
Commitment:
National funding organisation must formally commit funds to finance the
successful projects of the agreed joint ranking list according the funding
committed to this call.
National funding organisations will, through their contribution to the ERANET Plus or co-funded ERA-NET Cofund call, fund the national/regional
participants of the best quality proposals, where international peer review
evaluation and subsequent joint funding decisions by a designated joint
decision-making body retain these proposals for funding. In addition, the
top-up funding can be used to close the gaps of funding within the ranking
list (i.e. project participants for which no more national/regional funding is
available). An agreed share of top-up funding is to be allocated to fill these
funding gaps irrespective of nationality. Moreover, funding organisations
may also commit to fund non-national and non-resident researchers.
National funding organisations do not retain control of funding decisions and
on funding of the agreed share of top-up funding dedicated to fill funding
gaps.
Administrative efforts and benefits:
Administrative effort is necessary for operation of joint call decisions and
funding, in accordance with joint standard rules and procedures.
Page 12 of 154
Cross-border funding may be involved. National legal provisions may in
some countries restrict or disallow cross-border funding.
Distribution of funding
Usually, the real common pot is distributed centrally and the virtual common
pot is operated by the single partners. The same applies for the two
separate shares of the mixed common pot.
1.1.6 Financial commitment
Reliable commitments of all participating funding organisations are crucial factors for
a smooth operation and a successful call administration. Budgets must be secured
before the call is launched. It would be a major problem if one agency dropped out of
a running call, so this is to be avoided. All partners and particularly key players
(holding the biggest potential of applicants) of an ERA-NET should commit an
adequate budget to the call in order to enable a reasonable success rate.
Commitment of ERA-NET consortium partners
It is up to each programme owner (partner organization) to decide on the respective
financial contribution to a call. The written financial commitment from each of the
partner organisations is recommended. Budgets should be reasonable with respect to
the expected submissions.
All partners must consider the availability of their committed funds in regard to the
jointly agreed timeline, i.e. the budget availability of the national/regional
programmes must be connected to the ERA-NET call. Flexible funds will be a welcome
feature. Usually the funding commitment is expressed in a MoU.
A Memorandum of Understanding (MoU) describes the targeted cooperation of funding
organisations. It summarises the chosen call procedures including the methods for
selecting and funding successful proposals. Partners formally accept these procedures
and declare their commitment.
The nature of the document may vary and even the degree of liability is variable. It
may range from a fully elaborated legal document to a simple set of commonly
agreed minutes of a telephone conference.
Page 13 of 154
A sample MoU may contain all or a selection of the following items:
•
•
•
•
•
Purpose of the MoU & duration of the cooperation
Governance of the call
Method for evaluating & selecting proposals
Funding mode, budget plan for the funding of projects per partner
Commitment letters: what amount of funding is budgeted by each funding
agency
In case of ERA-NET Plus and co-funded ERA-NET Cofund call, the funding agencies
need to ensure that the cash-flow from the coordinator to the participating funding
organizations and their beneficiaries will fluently take place.
1.1.7 Funding decision
Before a call is launched it has to be clear how funding decisions will be taken to
finally fund successful proposals. All call partners need to have full transparency of
the decision making process in order to avoid any complications or surprises during
the call execution.
The evaluation process will produce a list of proposals suitable for funding, leaving
the actual funding decisions as the final step.
The following scenarios can be distinguished:
1. Common centralised decision: formal decision making is based on
the (binding) ranking list provided by the evaluation panel or body.
In case of ERA-NET Plus and co-funded ERA-NET Cofund call, a
decision on a ranking list is obligatory.
2. Common centralised recommendation & decentralised funding
decisions:
In many cases the final step of the funding decision is not taken jointly
by the ERA-NET consortium but individually by the national/regional
programme owners. The outcome of the evaluation process at the ERANET level is a jointly agreed list of proposals that are recommended for
funding. The panel or body that produces the common list of proposals
suitable for funding or ranking list issues a recommendation to the
Page 14 of 154
decision makers at the national/regional level. The national/regional
funding organisations are invited to respect these recommendations
and transform the recommended proposals into funded projects.
This requires a setting which enables a reliable implementation of the
common recommendations, which should be binding agreements for
the final funding decisions. Any contradicting decisions at the
national/regional level should be avoided. Thus, the final decisions
should be restricted to a formal character as much as possible.
This setting has to arrange for all substantial national/regional
peculiarities at the earliest possible stage in order to avoid
disappointing results. In any case potential weaknesses have to be
resolved before the ERA-NET consortium agrees on the common list of
recommended projects. Unilateral revisions of the ranking list will
cause major confusion.
1.1.8 Start of funded projects
The earliest possible starting date of any successful proposal will depend on the final
funding decisions on funding of all transnational project partners involved.
In the case of decentralised funding decisions the decision making process of every
single project participant influences the overall coordination of project enrolment. A
large discrepancy between the deadline of the ERA-NET call and the corresponding
national/regional funding decision will cause delays in the start of successful
proposals.
It is therefore necessary to know the procedures and timing of the involved
national/regional programmes throughout the ERA-NET consortium and make them
fully transparent.
The ERA-NET partners should then arrange for timely decisions within their joint call
time frame to allow for a quick start and release of funds for the transnational
projects. In addition other formal aspects should be known and effectively
administered, such as the necessity of a project consortium agreement prior to
signing a contract and/or transferring of instalments.
Page 15 of 154
1.1.9 Monitoring
One of the follow-up activities of a joint call is the monitoring of its efforts and results.
The ERA-NET consortium can agree on the implementation of systems for common
transnational monitoring, distinguishing between the monitoring of the call impact and
management and the monitoring of funded projects.
1. Monitoring of funded projects.
While there will be procedures for reporting and monitoring at the
national/regional levels, it is up to the ERA-NET consortium to agree on common
parameters for joint monitoring at the ERA-NET level, or, as currently most
often used, to collate national monitoring results for a joint monitoring at the
ERA-NET level.
Even if the monitoring of funded projects starts only at the end of the a call it is
necessary to identify the targets and indicators in advance as this may affect
the method and details for collecting information from the applicants already at
the time of submitting proposals.
Details on monitoring of projects are described in section 6.1
In case of the co-funded ERA-NET Cofund call, the coordinator of the ERANET Cofund reports to European Commission (EC) about each project selected
for funding after the end of the evaluation. This includes data on each project
partner (principal investigator) as well as abstract of the project proposal. After
the funding period of the projects (approximately three years), the EC is
provided with a new set of data including an overview of the results from each
project partner.
1. Monitoring of call implementation
It is unlikely that any involved national/regional programme will routinely cover
this aspect. However, especially for ERA-NET that are planning more than one
call it is recommended to analyse the efforts and feed the acquired knowledge
("lessons learned") back into the execution of future calls.
This type of monitoring can be done decentralised by the national/regional
organisations or centralised by the call secretariat applying online
questionnaires, telephone interviews or other methods.
These monitoring activities must target:
o analysing of internal call procedures
Page 16 of 154
o optimising call management
o implementation of user-friendly systems
o creating a reasonable and attractive funding instrument for the
target group
Details on monitoring of call implementation are described in section 6.2
The European Commission (EC) also wants to monitor the processes of the
co-funded ERA-NET Cofund calls. Therefore, the consortium must submit to
the EC at the end of the evaluation a) the joint selection of the funded projects,
b) the ranking list(s) of the projects, c) the observes report about the whole
evaluation process, and d) a formal commitment on availability of funds for the
selected projects from each funding organization participating the call.
1.1.10 Administrative conditions and necessary documents
A final important step in preparing a call is the confirmation of commitment of all call
partners.
A well-established instrument for that purpose is a Memorandum of Understanding
(MoU), which describes the targeted cooperation of funding organisations.
It summarises the chosen call procedures including the methods for selecting and
funding successful proposals. Partners formally accept these procedures and declare
their commitment.
The nature of the document may vary and even the degree of liability is variable. It
may range from a fully elaborated legal document to a simple set of commonly
agreed minutes of a telephone conference.
A sample MoU may contain all or a selection of the following items:
•
•
•
•
Purpose of the MoU & duration of the cooperation
Governance of the call
Method for evaluating & selecting proposals
Funding mode, budgetary commitment, funding decisions
Page 17 of 154
1.1.10.1 Sample of Memorandum of Understanding
Memorandum of Understanding
Partner organisation: .......................
National/regional funding programme: .......................
is willing to participate in the EXAMPLE ERA-NET Call 2010.
[ ] We agree with the process and schedule described in the call announcement version "Call 2010Announcement-version 1-final.doc"
[ ] There will be an adequate budget available for project funding.
The final funding decisions will be made by the partner organisation according to national/regional
procedures and regulations.
All information related to the projects will be kept confidential and not used for any other purpose than
evaluation of the application, making a funding decision and monitoring of the project.
For this collaborative activity Mr./Mrs. ....................... will be our contact person; contact data:
.......................
.......................
.......................
.......................
.......................
Legally authorised representative:
Name .......................
Function .......................
Date .......................
Signature .......................
Page 18 of 154
Memorandum of Understanding
1.
2.
3.
4.
5.
6.
7.
8.
Preamble (rational)
Aims of the joint call
Purpose of the MoU
Duration of the MoU
Call secretariat and additional bodies
Funding instruments
Details about the call and selection process
Annexes (glossary, timeline, etc.)
Draft texts
1. The aim of the [Name of the ERA Net] programme is to promote [Research Field] between [Target
Group] by supporting transnational research networks in this field, consisting of European
scientists.
It is hereby agreed that the Parties shall launch a transnational programme [Name of the ERA Net]
which will be jointly organized and financed by the participating European organisations, according
to their respective legal framework and regulations.
2. The joint programme will be implemented by a Call for Proposals. The Call will be advertised
simultaneously by the Parties in accordance with the procedure for their national programmes in
their respective member states. It will be published by the partners and, centrally, on the common
website [Website Address]. The aim of the Call is to enable scientists in different countries to
build an effective collaboration on a common transnational research project, based on
complementarities㻌and㻌sharing㻌of㻌expertise.
The programme offers funding for research networks for a [specify period or any other information
that may be of special importance for the ERA Net] period. The projects have to be established
between European research groups from at least two partner countries [Name of the ERA Net].
[A more detailed text in this section, on the specific aims of this ERA Net]
3. This Memorandum of Understanding (MoU) is a mutual statement of commitment among the
Parties. It serves to keep record of the commitment of the Parties which agree to make every
reasonable effort to fulfil the intentions expressed in the joint call as well as in the financial
framework. Details on the implementation of the joint call are outlined in the separate documents
[specify or list documents if already agreed].
4. The Memorandum shall come into force on the date of its signature by all Parties [specify when
this should be realized at the latest if agreed upon]. It shall cover the entire funding period of the
[Name of the ERA Net] grants [Possible: including any wind-up period to be agreed upon later].
The partners have thus to be aware that the agreement to their participation in a joint call
includes their financial commitment for the whole funding period of the projects which might be
longer than the duration of the [Name of the ERA Net] network as a whole.
5. The central coordinating body for the [Name of the ERA Net] call are, [Name of coordinating
body]. This central coordinating organisation is responsible for the overall organisation of the call
and the management of the joint international review process according to the rules agreed upon
by the [Name of the ERA Net] Programme Steering Committee (PSC), and will be assisted by the
PSC.
Page 19 of 154
The [Name of the ERA Net] Programme Steering Committee (PSC) comprises the nominated
representatives from each funding body signing this Memorandum. The PSC is responsible for
defining and organising the call, the application procedures and the review of proposals. The
nominees are entitled to join the review panel meeting of the scientific evaluation committee (SEC)
of [Name of the ERA Net]. However, each national funding organisation will make the final
decision on all national applications submitted to them taking into account the rating by the
scientific
evaluation
committee
(SEC)
of
the
[Name
of
the
ERA
Net].
The Scientific Evaluation Committee (SEC) is constituted to evaluate the proposals submitted in
response to this joint transnational call and to propose a final ranking of these proposals. The
members of the SEC will be nominated by the Programme Steering Committee (PSC). Some SEC
members should be drawn from non participating countries, in addition. The SEC members are
internationally recognized scientists chosen for their scientific or technical expertise. The
members of the SEC may not represent the nominating parties or adopt national considerations. In
order to avoid any conflict of interest, the SEC members must not apply to this call.
6. All decisions concerning the call procedures will be taken by the Programme Steering Committee
(PSC). The scientific assessment of applications will be undertaken by external written peer review
and the Scientific Evaluation Committee (SEC).for the funding decisions of the national partner
organisations.
In brief, the evaluation process is designed to involve three steps: First, a check for eligibility is
done by the joint call coordinator together with national partner representatives. The second step
includes external written peer review by international experts covering all fields of research
addressed in the call. Finally, in a review board meeting of the Scientific Evaluation Committee
(SEC) the applications are discussed, the best proposals are selected and final funding
recommendations are given by the reviewers. The SEC's decisions will form the basis
7. [Name of the ERA Net] collaborative grants may include the following funding resources:
• Research costs including and excluding additional costs as set out in the national
programmes.
• Research Mobility Schemes
• Seminars, Workshops (including research training elements)
Since the Call for Proposals will be published both by the partners in their respective partner
countries, and centrally at the [Name of the ERA Net], the national calls have to clarify which
funding instruments apply to each funding organisation. Some partners might use funding
instruments that are not used by others, e.g. paying salaries for scientists. Financing will be
provided in a [funding mode], [in case of virtual common pot: i.e. each partner will only finance
activities from its own country].
8. [This section may include all the details that are already agreed or it may include the methods fro
finding agreements on the particular details. This may include sections about the submission
methods of proposals, the eligibility and exclusion methods of applicants, selection criteria,
governance of the call, the evaluation procedure and more]
9. [Annexes may include documents that are already agreed like a glossary, a timeline, a draft
common call text, promotion material, strategic research agendas, etc.]
Page 20 of 154
1.2 Timing of the call
1.2.1 Call schedule
The overall time frame of a call is determined by the single steps involved in the call
procedure.
The following milestones need to be arranged:
•
•
•
•
•
•
•
•
•
Launch of call
1st deadline (if 2-stage submission procedure)
Availability of eligibility checks & evaluation results 1st stage
Feedback to applicants and invitation to 2nd stage
2nd deadline (or the only one if 1-stage submission procedure)
Availability of eligibility checks & evaluation results 2nd stage
Availability of ranking list / recommendations
Dates of funding decisions
Any meeting required for the call execution
1- or 2-stage submission procedures
Particular impact on the time frame arises from the choice between 1- or 2-stage
procedures: your call process may involve 2 steps (pre-proposal and full proposal) or
simply one (full proposal). In either case, only one single deadline for applications
will be effective for the overall eligibility of the proposal: At pre-proposal stage (for 2
stage procedure) or at full proposal stage (for 1 step procedure).
In case of ERA-NET Plus and co-funded ERA-NET Cofund call, a 2-step submission and
evaluation procedure is mandatory.
The timing of the co-funded ERA-NET Cofund call must be planned so that the
European Commission (EC) will receive information about launching the call 30 days
before the actual launch. Furthermore, the 1st stage submission should be kept open
at least for 60 days.
1st stage (optional) of submission
The first step is usually introduced to submit/receive a pre-proposal in order to keep
efforts low for both applicants and agencies. Based on the pre-proposals, non-eligible
applicants are sorted out and the 1st stage evaluation is carried out. All agencies
agree on a list of consortia that are invited to the 2nd stage. Depending on the
Page 21 of 154
number of the pre-proposals and the budgets committed, the number of the invited
full proposals can be limited to approximately twice the number of those that can be
funded.
2nd stage of submission or the only one (mandatory)
The 2nd stage of the submission (or the only one) is mandatory. Full proposals (and
national/regional funding applications, if necessary) are submitted/received. Proposals
are evaluated against the published criteria according to the agreed evaluation
procedures.
Apart from being a legal requirement for many national/regional programmes, a 2stage submission procedure provides the funding organisations with a filtering
function for limiting the number of full-proposals to be dealt with and for increasing
the overall proposal quality. A 2-stage submission also facilitates input from the
national/regional programmes at an early stage in the overall workflow. A 1-stage
submission procedure however will reduce the total time frame, despite relatively high
efforts imposed on the evaluation of proposals that finally might be rejected on simple
grounds that could have been spotted already in the pre-proposal.
1.2.2 Overlaps with other calls
The timing of a call will have to take into account other calls aiming at the same
targets. Similarities may occur in respect to:
•
•
•
•
Eligible organisations
Participating countries
Thematic focus
Other focus (e.g. SMEs)
1. The planning of the timeline does not need any specific coordination between
and among calls if the targets of your ERA-NET call are quite specific.
2. An overlap of the corresponding phases of different calls shall be avoided if the
targets are clearly overlapping to a large extent. In this case the applicants will
interpret these as similar calls and base their choice for a particular call on their
own criteria. The same target group may however appreciate staggered options
for submitting their proposals, therefore a call may benefit from being in
antiphase with a similar call.
3. On the other hand an overlap of the time frame with other calls may even be
desirable in order to avoid double submissions or double funding. The added
Page 22 of 154
value of this strategy could be the most adequate allocation of proposals or
even a re-routing of proposals received in the "wrong" call to the correct one,
provided the two call managements cooperate effectively.
One aspect of call planning in which mutual influence of calls cannot be considered is
the agreement on a time frame before the data of other calls become available, or if
the developed timeframe of an ERA-NET is the only feasible option.
The planning of call time frame usually requires trade-offs between the agencies
involved, and any later change of this time frame may be a step back in the process.
Forward planning and a timely entry into the NETWATCH call calendar database are
encouraged and will contribute to a better coordination.
1.2.3 Fine tuning of call milestones
Apart from boundary conditions imposed by the funding programmes, various
issues need to be considered when deciding on the milestones. These include the
effective promotion to the target group, the setting up of adequate project
consortia, or holiday periods.
Typical examples for positioning call milestones are given below.
Call X
Call Y
launch
15.12.2007
02.03.2009
interval (days)
75
65
1st deadline (if 2 stage
procedure)
28.02.2008
06.05.2009
interval (days)
11
14
availability of eligibility
checks & evaluation
results 1st stage
10.03.2008
20.05.2009
feedback to applicants &
invitation to 2nd stage
11.03.2008
28.01.1900
Page 23 of 154
Call Z
interval (days)
49
51
2nd deadline (or the
only one if 1 stage
procedure)
28.04.2008
10.07.2009
interval (days)
50
95
availability of eligibility
checks & evaluation
results 2nd stage
17.06.2008
13.10.2009
dates of funding
decisions
01.09.2008
01.11.2009
start of funded projects
02.09.2008
02.11.2009
total time frame (days)
110
160
Useful tools for fixing exact dates include interactive calendars such as
http://holidaycalendar.swisspost.com/en/index.html
1.2.4 Plan necessary meetings
The administration of a call will necessarily require the exchange of information
among all agencies, such as the results of national/regional eligibility checks, the
monitoring of funded projects or similar. This can be done through web-based
proposal management tools and/or meetings (physical or telephone/video
conferences). Experience shows that personal meetings are still a relatively efficient
way of reaching agreements, especially for large ERA-NET consortia, and that
unanimous decisions can be reached.
Communication could be required at the following call milestones:
• Eligibility checks at the programme level
• Agree on evaluation results after the 1st stage of a call and invite consortia to
2nd stage
• Agree on evaluation results after the 2nd stage and issue ranking list or list of
recommended proposals
Page 24 of 154
• Inform on the time frame for the initiation of each national/regional contracts
• Monitoring of call implementation
• Monitoring of funded projects
The meetings are ideally planned already during the setting up of the time frame in
order to ensure the availability of all call participants.
Between the milestones, communication will be established through email, telephone
and/or web-based tools.
1.2.5 NETWATCH call calendar
The NETWATCH call calendar contains dates and scope of calls launched by ERA-NETs
and is placed on the public part of NETWATCH. It is available to the RTD community
and provides an interactive display of calls (ideally including also calls that are
planned but not yet launched) with respect to timing, topics and participating
countries. Thus, it is a coordination and promotion tool for your call and its use is
highly recommended.
The calendar can only display data that are actively provided by the ERA-NETs.
Coorinators are responsible for the data entry and the accuracy of the information. In
any case the calendar can only provide basic information, complemented by links to
the individual ERA-NET websites where further details should be available.
The call calendar also provides basic data on the call scope, such as target groups and
targeted projects.
Page 25 of 154
1.3
Scope of the call
The participating funding partners should agree and commonly define the target
group and the call topic. Different methods can lead to an early stage to a
comprehensive arrangement. A common inventory of background material like central
scientific publications, national or international policy reports or strategies, reports
about past conferences or similar meetings concerning the topic can be useful to
develop a common understanding of the call topic and the envisaged projects.
Identification of topics by organising stakeholder consultations may be an additional
method.
The use of an existing strategic research agenda (SRA) or the development of such a
document (green book, joint guiding paper, etc.) for a specific research field or a
technical challenge may be very helpful. A jointly developed fact sheet, in the sense
of a continuously developed basic concept paper, may be helpful as well to
systematically align the common work regarding target group and topic.
While defining the scope of an ERA Net it could be of importance for programme
managers and owners to mirror the envisaged topics with current national priorities of
national programmes identifying overlaps or gaps.
The development of (draft) common benchmarks at this stage of the work may also
lead to an intensive reflection about the scope and the envisaged goals.
The connection to an existing national or European strategic research agenda or policy
might be the easiest approach. Most ERA-NETs select call topics with a clearly defined
scope. Only very few work with entirely open call topics.
In some cases it may be justified to have slightly different goals as the national
programmes and policies are not fully compatible. In such cases the differences
should be known by every consortium partner and the collaboration between funded
projects should be envisaged anyway.
1.3.1 Method for definition of scope
A clear definition of the call scope will be helpful for the ERA-NET consortium as well
as for applicants. The RTD community will appreciate transparency of invited topics,
eligible consortia and targeted projects.
A common understanding of the scope is essential for applicants as well as for funding
organisations. Especially if the call scope cannot be supported by all agencies, i.e.
only parts of a call can be supported at national/regional level, it is necessary to
clarify the details.
Page 26 of 154
Quite often there will be a trade-off between a common focus and the number of
funding programmes that can support a specific target. A narrow scope of a call will
usually mean that only a limited number of funding programmes will fit into it. A
wider scope on the other hand will fit to a larger number of programmes but might
also attract a larger number of submissions including a number of proposals that are
not very specific with regard to the call scope.
Various methods of defining the scope of a call have been applied in different ERANETs, including:
• Matching of individual national/regional programme priorities with European
strategies and common targets at ERA-NET level
• Exploiting existing documents on research priorities such as the strategic
research agenda (SRA) of one or more technology platforms (ETP). A list of
ETP can be found here. Especially for calls targeting applied RTD projects a
close cooperation with an ETP will support the target group through an
adequate choice of topics and the appropriate timing of calls.
• Joint programming board composed of experts in the field: In addition to
ETPs a group of international experts exchanging views and visions may also
be helpful for designing the call. Here, ERA-NETs may also liaise with JPIs.
• Analysis of previous calls and response from target groups: The target group
may have shown preferences for specific areas in previous calls, which will be
worth being explored in detail.
1.3.2 Overlaps of topics with other calls
Again the overlap with other calls needs to be checked in order to provide the target
group with a clear setting.
As already explained the full set of call features has to be taken into account when
considering the overlap of calls. This includes the targeted type of projects
Possibilities to check the thematic focus of other calls include:
• Cordis
• NETWATCH call calendar
• FP7 National Contact Points>
Page 27 of 154
1.3.3 Identification of target groups
Critical mass of applicants available
It is important to anticipate the potential of applicants interested in your call in order
to design the workflow and time frame accordingly.
Important partner countries on board
In many ERA-NET calls it has proven unproductive to launch calls without the
European key players on board. Experience has shown that partnering of applicants
often works via these key players, and it may therefore be helpful to include major
key players/countries in the call before launching it.
Analysis of response to previous calls
If you are not designing your first ERA-NET call, the analysis of previous calls is
helpful in determining the call topic, since you rely on information how previous calls
have been received by applicants. Questionnaires can be spread to stakeholders.
1.3.4 Definition of targeted projects
Before launching a call the ERA-NET consortium has to agree on the type of projects
that will be supported. Clarification and transparency on what is eligible will avoid
misunderstandings also on the part of the applicants.
Eligibility criteria
Basically, eligibility criteria are necessary to assure equal formal conditions for all
applicants.
Phases of research
The ERA-NET consortium will define which type of research will be addressed. The
choice between basic and applied RTD will naturally emerge from the focus of
participating programmes.
In the case of applied RTD particular attention should be paid to the question of
commercialisation of project results. It needs to be clarified to the applicants if the
participation of industry partners is a requirement at ERA-NET level and/or at
national/regional levels.
Page 28 of 154
The choice of the funding mode may depend on the phases of research that are
supported in the call.
Size of consortia
The minimum size (number of participants) of eligible consortia has a decisive
function in your calls. It has been shown that small consortia are a preferred
constellation for ERA-NET calls, which bridges the gap between pure national schemes
and programmes which require larger consortia, for example FP7.
Duration of projects
When deciding on the eligible duration of projects the ERA-NET consortium should
take into account the follow-up activities, such as the monitoring of funded projects
as discussed in Section 6.1.
In particular it has to be clarified how any activities beyond the duration of
the ERA-NET or ERA-NET PLUS contract will be carried out and how the
necessary resources will be secured, because in no case can funding money
from an ERA-NET project be set aside for monitoring activities after the
project end.
1.3.5 Submit your data to network
The objectives of the call together with additional data as far as they are decided
(target group, budget, events after funding decision, any other key data) should be
published on Netwatch, where the information will be connected to the call calendar.
In the past years CORDIS (Community Research and Development Information
Service) became a one-stop-shop for information about ongoing funding activities for
researchers. Netwatch will become the same standard for transnational activities so it
will outreach to stakeholder.
Page 29 of 154
2 Call Preparation
2.1 Call documents
2.1.1 Supporting documents
2.1.1.1 FAQ Template
2.1.1.2 Feedback Letters
2.1.2 Check content of call announcement for coherence with the proposal form
2.2 Promotion of the call
2.2.1 Introduction/Background
2.2.2 ERA-NET specific activities
2.2.2.1 Common Call
2.2.2.2 Call Brochure
2.2.2.3 ERA NET Newsletter
2.2.2.4 Success Stories
2.2.3 National activities
2.2.3.1 National Annocuments
2.2.3.2 National Mailing List
2.2.3.3 National Events
2.2.3.4 Conferences/ Fairs
2.2.3.5 individual Consultuing
2.2.4 Promotion via target group
2.2.5 Providing services to applicants
Contents Page
Page 30 of 154
2.1 Call documents
2.1.1 Supporting documents
For implementing a call several documents must be prepared and the call must finally
be promoted efficiently. Below you will find a short description of these documents
and promotion tools with an exhaustive toolbox for most of these items.
1) Call announcement, call text (national/regional or ERA-NET level)
The call announcement is a brief description of the basic call parameters, including
participating countries, eligible applicants, dates, deadlines, call scope & title,
description of projects and consortia to apply, and criteria against which the projects
will be evaluated.
The call announcement will be published at the ERA-NET level, but it may also be
necessary to publish it at the national/regional level (if a parallel national/regional call
has to be launched). It is a binding document for both agencies and applicants.
The call text may be a simple flyer or a longer document supporting the call
announcement with more specific details of the above mentioned kind of information.
Usually this instrument is used at both ERA-NET and national/regional levels to inform
the applicants on expected projects, when they have to be submitted, and how they
will be evaluated.
A preliminary version can be published as a pre-announcement in order to notify
applicants of a forthcoming call with larger anticipation. You then must hint at the fact
that the announcement is preliminary and that single (and which) issues might
change.
Download Template
2) Guidelines for applicants
The guidelines for applicants contain all information concerning the joint call and
detailed descriptions of the application process so that (in the best case) no queries
arise. For a generic template see section 3.2.1. Additionally, national/regional
requirements that are not the same for all call participants shall be explicitly
mentioned, as outlined below (see also for an example).
Page 31 of 154
National requirements:
In addition to the general ERA-NET guidelines any national requirements that each
participating country may request, should be clearly stated in the guidelines. The level
of such additional requirements should be sought to be kept at a minimum level. It is
important that such requirements are known to the applicants in advance (but not
after the submission of the proposals). The names of the national/regional contact
point should also be published, with a recommendation to contact these persons prior
to submission of proposals in order to clarify the general requirements, but also the
national/regional requirements and other national/regional issues of importance.
3) FAQs
Although questions may be avoided by thoroughly drafted guidelines, there will often
be queries from applicants. A collection of frequently asked questions (FAQs) is a
practical and effective tool for providing answers to major concerns or apparently
poorly defined and/or ambiguously expressed sections of your guidelines, or general
answers to the call.
FAQs serve as a general documentation at the ERA-NET level and help avoid
misinterpretations and diverging answers at the national/regional level. Templates are
provided in section 2.1.1.1.
FAQs may also be helpful for improving the call implementation itself, as apparent
misunderstandings can be sorted out directly in the process.
4) Proposal Forms
It is important to clarify the formal procedure for submission in the guidelines: how
many forms have to be submitted at the ERA-NET level and at the national/regional
level, who has to submit which document.
Applicants will submit their proposal in forms provided by your ERA-NET. Usually
these forms can be downloaded from the website or the call secretariat. In these
forms you request all data from the applicants which you need to have available for
the selection process. You are advised to define the full set of required information
well in advance, because it will be difficult to collect any information at a later stage.
Follow-up activities such as statistical analysis and monitoring have to be taken into
account.
In some ERA-NETs macros have been used to extract these data into a database,
other ERA-NETs have limited space to avoid lengthy applications. Some ERA-NETs are
using webbased application forms, or a combination of webbased applicant data entry
and an upload facility for the proposal description.
Page 32 of 154
The following Proposal Forms can be used as a template:
• Example 1: NEURON
• Example 2: MNT-ERA.NET
• Example 3: ETB
5) Feedback letters
Applicants will appreciate a harmonized feedback during the process. This will start
with the submission (confirmation of receipt). After each round of evaluations the call
secretariat is advised to communicate results, recommendations and/or invitations to
consortia or coordinators.
In all these letters you should precisely explain to which extent the information is
binding, since the ERA-NET as such is not necessarily taking any formal decisions. For
examples see section 2.1.1.2
6) Evaluation checklist & reporting forms
The list of evaluation criteria has to be mutually agreed among all agencies and will
be applied at the ERA-NET level. Usually evaluation criteria are also published in the
guidelines; however, you also can publish only larger domains of criteria (while the
detailed sub criteria are a working tool for evaluators and experts).
Evaluation reporting forms are designed for collecting results from the evaluators;
ideally, they are provided in a form that makes it easy to collate the results in a table.
The design will depend on the chosen evaluation process.
The jointly agreed evaluation process has to be described and provided to the
(external) referees and evaluators.
7) Guidelines for evaluation
This is a document providing evaluators and experts with the necessary information
(criteria, thresholds, dates and deadlines) to perform their task. They will depend on
the evaluation process.
8) Description of evaluation meeting
Depending on your evaluation process, you must plan and describe how the
meeting of evaluators is organised.
9) Non disclosure agreement (NDA)
All information contained in a proposal is confidential. Usually, program managers
have already signed a NDA, but if external experts / evaluators are involved, you
should supply a form they can sign.
Page 33 of 154
2.1.1.1 FAQ Template
The following FAQ can be used as a template:
1. Scope of the Call
Q 1: What kinds of proposals are expected?
A 1: The aim is to motivate and launch transnational, innovative R&D projects
related to XXX; specifically: XXX
2. Eligibility
Q 2: Who can apply?
A 2: The call is open for industry, universities and research centres. Eligibility
may vary as it will be dependent on the rules of the relevant national funding
programmes. Prospective partners are strongly recommended to discuss
eligibility with their national/regional funding agency:
• Country/Region
• Participating Funding Agency
• Accessible Programme(s)
Q 3: What is the minimum size of the Consortium?
A 3: The Consortium must consist of a minimum of: Sample: ‘three partners
located in two different countries. Each project partner must be prepared to
enter into an individual contractual agreement with an ERA-NET funding agency.
[*Note: The contractual agreement with the funding agency will define the
roles, responsibilities and activities of the partner within the proposed project.
The rules of each funding programme will determine whether an individual
partner may be entitled to receive funding for their role and activities within the
proposed project].
Q 4: Can partners outside the ERA-NET countries/regions participate?
A 4: This Call is primarily directed at those nations/regions that are partners of
the ERA-NET consortium. The list of participating funding agencies is detailed in
the Call Guidelines. Additional project partners can join the consortia as a third
party if they can provide their own source of funding.
Q 5: Why should we participate in an international consortium?
A 5: The ERA-NET Call gives you the opportunity to participate in an
International Project under your own national/regional rules and in Programmes
with which you are already familiar. Sample: ‘This Call also gives you the
chance to exploit international technology and market expertise and allows you
to integrate into international value chains. Participation will allow you to work
under a well-defined framework with partners that would otherwise not be
readily accessible to you'.
Page 34 of 154
Q 6: Can applicants be involved in more than one project?
A 6: Yes, it is possible to participate in more than one project.
Q 7: Is there a maximum level of project costs that should originate in any one
country within a project?
A 7: Applicants are encouraged to build consortia with partners from several
countries. Although there is no firmly defined split of funding between countries
within a project, it is recommended, as a guide, that a maximum of 70% of the
total project costs should originate in any one country.
3. Application Process
Q 8: Where can I find all the necessary forms for submitting a proposal?
A 8: The ERA-NET uses an on-line submission system. Access to the system
(and its user manual) is via the web link. Users must register on the website
prior to gaining access to the submission system. Call Guidelines may be
downloaded from the website. National/Regional Programme Managers will also
provide relevant information.
Q 9: Will I receive confirmation of receipt?
A 9: The project coordinator will receive an automatic email confirming receipt
of the submitted proposal.
Q 10: Does every partner have to supply a proposal?
A 10: The project coordinator must submit the ERA-NET Pre-Proposal and Full
Proposal using the electronic submission tool. Each project partner has to apply
for funding from their respective national/regional agencies. The information
detailed in the Pre-Proposal will meet the requirements of some funding
agencies. However some other funding agencies will require the submission of
further documents and/or supporting information. Each partner is requested to
contact their local funding agency in order to determine what additional
information is required and when it needs to be provided.
Q 11: Is it permissible to supply a proposal in a national language?
A 11: The Pre-Proposal and Full Proposal forms must be submitted in English.
The national/regional documents must also be provided to the relevant funding
agencies according to the existing national/regional rules.
Q 12: Is it necessary to provide a Consortium Agreement with the proposal?
A 12: An outline for the Consortium Agreement must be submitted with the Full
Proposal form. The Consortium Agreement must be signed before the first
national/regional funding instalments are transferred.
Q 13: What are the timescales for the Call Process?
Page 35 of 154
A 13: The timescales for the Call Process are given in the table below:
Q 14: Can I request an extension of the deadline to submit either a PreProposal or Full Proposal?
A 14: No. The Call deadlines for Pre-Proposals and Full Proposals are fixed. The
deadlines for the national/regional proposals may vary a little. It is advised to
check the deadlines for the national/regional programmes with the appropriate
funding agency.
4. Evaluation
Q 15: How are Proposals evaluated?
A 15: The national/regional agencies make their own evaluation of the
respective funding applications, based on the role and activities of the
national/regional partners viewed in perspective of the overall transnational
project. The ERA-NET Management Group will assess whether the eligibility
criteria are fulfilled. Based on the outcome of the individual national/regional
evaluations and the ERA-NET Pre-Proposal evaluation, selected projects will be
invited to submit Full Proposals. Full proposals will be evaluated by international
experts in accordance with the requirements of the Commission. Projects
receiving a positive evaluation (i.e. those over the threshold) will be ranked in
order of overall score. The ERA-NET will prioritise the funding of each project
based on its position on the ranking list. The number of projects that will receive
funding will be subject to the availability of funds.
Q 16: Will I get feedback on the project submission?
A 16: The ERA-NET Call Office will send feedback on the evaluation to the
project coordinator after all funding decisions have been made. Feedback on the
proposal will be provided also by the relevant national/regional agencies.
5. Funding
Q 17: Who is funding this call?
A 17: The funding comes from the national/regional funding agencies so each
project partner is responsible for applying for funding from the relevant agency.
Q 18: What exactly is funded?
A 18: The level of funding and the costs covered are dependent on the rules of
the relevant programmes. Partners are recommended to discuss this with the
appropriate funding agency.
Q 19: Is there bonus funding available for transnational cooperation?
A 19: Only some programmes offer a bonus for transnational cooperation.
Partners are recommended to discuss this with the appropriate funding agency.
Q 20: Is there bonus funding available for the role of Project Coordinator?
Page 36 of 154
A 20: There may be bonus funding available. However this will depend on the
rules of the relevant funding agency.
Q 21: Is every partner required to submit a funding application?
A 21: No, partners may use their own financial means if they wish but they
must confirm this source of funding at application stage.
Q 22: What happens if a partner fails to be awarded funding by their respective
national/regional agencies?
A 22: One of the eligibility requirements of the ERA-NET is that all partners
either receive funding from their respective national/regional agencies or use
their own financial means. If a partner is not awarded funding by their
national/regional agency and is unwilling to use their financial means, then the
Consortium will not be invited to submit a Full Proposal.
Q 23: What happens if a project is awarded funding by their respective
national/regional agencies but is not approved by the ERA-NET?
A 23: If the national/regional agencies wish to fund the project, they may do
so. However the project will not be considered as an "ERA-NET project".
6. Project Implementation
Q 24: How long does it take before the projects can actually start?
A 24: It is expected that funding decisions will not be made earlier than XXX.
Q 25: Is it a problem if a project starts before the final decision by the
national/regional agencies?
A 25: It is recommended that the project should only start after the funding
decisions have been made.
Q 26: What happens if a partner leaves the consortium?
A 26: The issue over what happens if a partner leaves should be covered by the
Consortium Agreement.
7.
Reference the "Guidelines" section for relevant information on the call process, eligible
projects and consortia, funding rules or evaluation procedures.
Reference the "Submission System" section for instructions on how to complete the
Application Form, the process of accessing to the Submission System for the first time
and subsequently, and also the process of submitting the Application Form.
Page 37 of 154
2.1.1.2 Feedback Letters
Feedback to applicants: after stage 1
Dear applicants,
We have assessed your pre-proposal for the EXAMPLE ERA-NET Transnational Call 2010.
Number: 2010-0
Acronym: xxx
Based on the information given in your pre-proposal, we recommend the submission of a full
project proposal and corresponding submissions of national/regional funding application
However, there are some conditions/recommendations that must be fulfilled in the full
proposal phase.
In any case all applicants are strongly advised to contact their local funding organisations for
detailed information on regulations and potential weaknesses of your pre-proposal.
Please find all agency contact details at http://.....
The full proposal has to be submitted by the coordinator using the appropriate form available
at http://.....
We remind you that all partners must submit the application for funding to their local
agencies.
When preparing full project proposals, please pay special attention to the detailed description
of:
- state of the art and technical challenges
- targeted innovation
- work packages, timing, milestones, results and deliverables
- management of consortium and role of partners
- exploitation of results and market situation
- resources
Please observe the recommended length of the full proposals.
Deadline for full project proposals and national funding applications is 5 July 2010 12:00 CET.
Sincerely,
Your EXAMPLE ERA-NET team
www.example-era.net
Page 38 of 154
Feedback to applicants: after stage 1
Dear applicants,
We have assessed your pre-proposal for the EXAMPLE ERA-NET Transnational Call 2010.
Number: 2010-0
Acronym: xxx
Based on the information given in your pre-proposal, we do not recommend the submission of
a full project proposal and corresponding submissions of national/regional funding application.
All applicants are strongly advised to contact their local funding organisations for detailed
information on regulations and potential weaknesses of your pre-proposal.
Please find all agency contact details at http://.....
Sincerely,
Your EXAMPLE ERA-NET team
www.example-era.net
Feedback to applicants: after stage 2
Dear applicant,
Thank you for your application to the EXAMPLE ERA-NET Call 2010.
The Transnational Co-ordination Team (TCT) met on 29 September 2010.
The TCT has reviewed all proposals submitted and is now communicating its views and opinions to
the relevant national and regional funding agencies for their consideration.
We would like to remind you that the final decisions regarding funding (or not) rests with the
national/regional funding agencies and not with the TCT. Once all funding agencies have made
their decisions your funding agency will communicate the results to you.
Applicants are reminded of the need for a unanimous decision of all involved funding agencies for
transnational collaborative projects to proceed.
Sincerely,
Your EXAMPLE ERA-NET team
www.example-era.net
Page 39 of 154
2.1.2 Check content of call announcement for coherence with
the proposal form
As soon as all documents have been drafted they must be checked for coherence and
soundness. Call documents (guidelines, proposal form, etc.) must reflect the call
targets outlined in the announcement. The following issues need to be checked:
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Topical priorities
Target group
Countries involved
Call budget
Project consortium description (eligible applicants)
Role in the project
Commercialization
Transnational added value
Excellence
Innovation
Duration
Fit to scope
Compliance with the call topic
Other call related issues (e.g. researcher training)
Page 40 of 154
2.2 Promotion of the call
2.2.1 Introduction/Background
ERA-NET Joint Call publication can typically be carried out as follows:
• By common call announcement (supplemented by national/regional call
specifications)
• By call announcement made by one agency or the ERA-NET call secretariat
for all participants
• By each funding organisation separately
Typical Call Promotion activities therefore have been reviewed and sorted into the
following categories:
• ERA-NET specific activities (s 2.2.2)
• National / regional activities (s 2.2.3)
• Promotion via target group (s 2.2.4)
Each category should be considered in the context of relevance to any new Call
Promotion activity.
CHECKLIST for CALL PROMOTION
You will need to have all (or a selection) of the following available at CALL
LAUNCH
•
•
•
•
Call announcement: call text, flyer, website, E-mail format
List of participating countries and information on national/regional programmes
Guidelines for applicants, Application forms
FAQs
Page 41 of 154
2.2.2 ERA-NET specific activities
• Some ERA-NETs provide maximum prominence to Call announcements by
positioning them on specific 'Call for Research' pages on the ERA-NET
website. These are used to host information about the Calls and to link to
information about the various projects already funded or underway.
• The Common call announcement published on the ERA-NET website will
include a short descriptive text with details of the countries participating and
the deadlines for proposal submission. It will provide links to the Call flyer /
brochure and further information.
• The Call flyer (brochure) includes a description of the ERA-NET, the benefits
of participation in the Call, the Call focus, eligibility to participate and details
of the procedure and evaluation with Call deadlines. Funding agency and
further information links will be provided.
• ERA-NET newsletters are routinely published on the website and targeted to
subscribers. Newsletters may be used to report scientific workshops and/or
Joint Call research projects. They typically describe the aim of the ERA-NET,
provide some background information e.g. relating to the strategy of the
initiative and highlight the number of national research funding organizations
involved.
• Press releases are surprisingly few and typically utilize the Call brochure
information. They may however provide a context to the current Call e.g. in
relation to plans for future Calls.
• Dissemination of call results
• As the ERA-NET develops Success stories of previous Calls and information on
existing projects become available. Success stories should be actively
disseminated. The Success stories generally include title and project goals,
details of the project participants (institution and country) as well as the
contact information for the project coordinator. Ideally, also project outcomes
are featured.
Page 42 of 154
2.2.2.1 Common Call Announcements
ERA-NET Logo and Branding
• SAMPLE 1: ERA-SME
The announcement appears on the Home Page with
Brochure and Help Desk. It includes a description
participating, the deadline for proposal submission and
following summary footnote could be used as standard on
links to the Call
of the countries
a short text. The
all Calls:
"The ERA-NET [name] project is an EU initiative covering research and
innovation programmes from [number of] countries/regions and funded by
the 7th R&D EU Framework Programme under the ERA-NET scheme. The
projects will be funded by the respective national funding programmes".
• SAMPLE 2: M-ERA.NET
The announcement appears on the Call-specific page with links to the Call
Brochure, the participating countries and funding information and the
process description. It includes links to the Guide for proposers, FAQs, a
proposal form and the deadline for proposal submission. The advantage of
the ‘Call for Research' page is that it directly links Success Stories
demonstrating successful trans-national cooperation between national and
regional funding programmes to the Current Call.
[M ERA-NET]: CALL STRUCTURE - Funding Agencies and Programmes participating in
the ERA-NET Call by Country/Region using a graphic map – click here to get a
detailed description of the participating funding organizations and their relevant
Page 43 of 154
2.2.2.2 Call Brochure
ERA-NET Logo and Branding
The following template can be used:
1
What is an ERA-NET?
2
What is this ERA-NET?
3
The benefits of participation in the Call
Sample: 'offers participants a simplified introduction to the initiation and participation in
R&D projects at the European level. Individual project partners seek funding from their
national/regional funding agencies in accordance with the relevant programme rules.
The added value is the opportunity to easily access international know-how, to gain
technology and market expertise and to integrate into international value chains'.
4
Call focus
5
Who is eligible to participate?
Sample: 'The Call is open to industry, universities, institutes and research centres.
Project consortia must consist of a minimum of three partners from at least two
different ERA-NET countries. Funding is via national/regional funding programmes.
Applicants must verify both their eligibility and the rate of financial support with the
relevant national/regional funding agency'.
6
Procedure and evaluation
Sample: 'Application is through a two-step process. Following the Call announcement,
prospective participants must discuss their proposal with the relevant national/regional
programme and funding agency.
The first deadline is for the Project Coordinator to submit a mandatory Pre-Proposal. At
the same time each project partner is also required to apply for funding from their
national/regional funding agency. The information detailed in the Pre-Proposal will meet
the requirements of some funding agencies. However some other funding agencies will
require the submission of further documents and/or supporting information. Based on
evaluations at national/regional levels the ERA-NET will invite selected applicants to
submit a Full Proposal.
The second deadline is the submission of a Full Proposal including a full project plan by
the Project Coordinator to the ERA-NET. It will organise an independent evaluation of
the Full Proposals according to the following criteria:
• Scientific and/or technical excellence
• Quality and efficiency of the implementation and the management
Page 44 of 154
• Potential impact
This evaluation process will be completed by a panel of international experts. The
outcome of this independent evaluation will be a ranking list of proposals. Based on this
ranking list the ERA-NET will provide recommendations for funding to the respective
funding agencies.
For this Call the total available budget is anticipated to be in the region of €XX Million'.
7
Further information
Detailed information on the Call including forms, guidelines etc. can be found in the
website at www.eranet.org
8
Call deadlines
Opening of the Call
+2 months - Pre-Proposal deadline
+5 months - Invitation to submit Full Proposal
+3 months - Full Proposal deadline
+3 months - Funding decisions
9
Funding Agency Information
Country / Region
Funding Agency
Website Contact
Topics
Eligible Project Partners
Page 45 of 154
2.2.2.3 ERA-NET Newsletter
Newsletters should be branded according to standard format including:
Front page:
•
•
•
•
ERA-NET Logo and Branding (primary-top)
European Commission logo (base)
FP7 logo (base)
European Research Area logo (base)
Reverse page:
• Participating organisations logos (base)
Standard text can include:
"The ERA-NET [name] project is an EU initiative covering research and innovation
programmes from [number of] countries/regions and funded by the 7th R&D EU
Framework Programme under the ERA-NET scheme. The projects will be funded by
the respective national funding programmes".
The newsletter should reference 'further information can be found on our web page at
http://www.eranet.org'
SAMPLE: NEURON
Page 46 of 154
2.2.2.4 Success Stories
The following template can be used:
1
Short Title
2
Full Title
3
ERA-NET name
4
Project Description
5
Participating Partners
• Institution
• Country
6
Project duration
7
Total Project Cost
8
Coordinator
9
Contact details
• E-mail:
• Tel:
2.2.3 National activities
In most ERA-NETs the common Call announcement is made by one agency or by the
ERA-NET secretariat for all participants (as described under ERA-NET Specific
Activities). This announcement is supplemented by the activities of the participating
agencies in their local networks.
• Web-based activity will vary according to national requirements. Where a
participant country has a clearly identified National/Regional Programme
Website (e.g. Tekes) it will include Call related topics within the programme
description with launch date, total volume of funding, duration and
collaborative arrangements nationally/regionally within the format governed
by the local provider. The Call announcement will typically appear as a News
item. Countries/regions without e.g. specific theme-focused programmes will
typically also announce the Call under News.
• The Call announcement published on the National/Regional Programme
Website will include comments specific to the national/regional funding
allocation, a local contact point and details of any local events. It may refer to
Page 47 of 154
the submission procedure at national/regional and international levels and will
highlight local deadlines.
• The local contact for the ERA-NET usually includes a remit for both
national/regional and international activities in a given thematic area. This
relationship governs the 'targeting' of potential applicants at national/regional
level and usually points to the application of a National/Regional Mailing List
to the ERA-NET Call. The mailing list is managed by a nominated Programme
Owner and it is cross-referenced with national/regional and FP7 databases at
regular intervals.
N.B. Regular updates through the mailing list can include pre-marketing of
the Call, an important step in raising awareness of the opportunity in a
targeted manner.
• National/Regional Events targeted to relevant audiences can be used as the
forum for policy discussions, for dissemination of information of the European
Commission / The European Research Area, and for dissemination of project
outcomes. National/Regional Conferences/Fairs will also provide opportunities
for ERA-NET exposure at e.g. high-profile annual events targeting indigenous
technology sectors. Benefits include the contextualisation of the ERA-NET Call
in keynote speeches and the opportunity to network with multiple
national/regional and international peers.
• The decision to participate in the ERA-NET Call will be based on client needs.
These will be determined most efficiently one-to-one by Individual Consulting.
Programme managers will typically have in-house processes and structures to
increase the effectiveness of their organisations in delivering effective
services for clients. These may include organisational support for ERA-NET
partner searches.
2.2.3.1 National Announcements
SAMPLE 2: TEKES
The following summary text could be used as standard on all Calls:
"the National Programme has reserved X million euros to finance new research
projects for the year 2010. Funding will be directed to trans-national co-operation
projects, mainly inside the ERA-NET [name]".
It may also include: CHECKLIST
•
•
•
•
Research themes
Call budget
Eligible countries
Maximum duration of projects
Page 48 of 154
•
•
•
•
•
•
•
•
•
•
•
•
Call timetable including closing date
Local Information Session programme and presentations
Local contact details
Links to further information under:
• ERA-NET website
• ERA-NET Member Organisations
• ERA-NET Reports and Success Stories
Documents for download
Institutional Endorsement Form
Pre-proposal template (see below)
Web Portal Instructions
Important Information for National Applicants
Call Announcement
Guidelines for Applicants
Guidelines for Reviewers
2.2.3.2 National Mailing List
SAMPLE: The Irish Nanotechnology Association
Irish interests in nanotechnology are shared within the Irish Nanotechnology
Association which was established in 2002 by Enterprise Ireland to encourage the
development of nano-materials and processes by Irish Industry. All interested parties
receive information relating to their areas of focus and while doing so, can obtain
overview information of local academic research, industry, instrumentation and patent
activity.
The mailing list is managed by a nominated Programme Owner and it is crossreferenced with national and FP7 databases at regular intervals.
The local contact for the ERA-NET will arrange for the distribution of the Call
information through the FP7 National Contact Points as well as other relevant
networks.
Page 49 of 154
2.2.3.3 National Events
The following content can be used as a template:
1
Seminar Title
2
Rationale
e.g. ‘to give information on the mid-term status and the first R&D
projects of the ERA-NET, to launch a new Call for funding and to
provide an overview of the related activities in Europe'
3
ERA-NET today and plans for the future
4
Strategic national measures linking with ERA-NET
5
Overview of related national research
6
ERA-NET funded projects [flash presentations]
7
Call Announcement [with evaluation description]
7
Future role of the ERA-NET in FP7 and the ERA [European
Commission speaker]
8
Other ERA-NETs of interest
9
Related activities e.g. European Technology Platforms
Page 50 of 154
2.2.3.4 Conference/Fairs
While much of the planning of a large national conference may not be accessible to
the local ERA-NET contact, the following activities could be pursued:
• Exhibition stand: ERA-NET funded projects – success stories and current Call
brochure
• Podium presentation: ERA-NET coordinator or local participant
• Informal meetings: pre-schedule ‘science café' interactions between ERA-NET
participants and international delegates on the fringes of the conference
• Follow-up individual meetings on-site: organise transport for ERA-NET
participants and international delegates to meet local partners.
2.2.3.5 Individual Consulting
Some activities which could be used during individual consulting:
1
Assessment
2
Expertise / Development Feedback on potential access to new
technology and current product/service
offerings:
better
strategic
focus
and
direction.
3
Information /
Introductions
Facilitation
of
visits
of
international
collaborators; detailed group and individual
itineraries
abroad;
adding
value
to
Conference-Trade Show offer by approaching
potential partners in advance.
4
Financial Assistance
Travel support
Detailed discussion with the client (and
possibly others involved in their research
and development strategy): as a result, the
ERA-NET contact is in a position to
significantly challenge the plans of the client
based on a close understanding of the
existing Call and future opportunities.
Page 51 of 154
2.2.4 Promotion via target group
• There are multiple opportunities to publicise ERA-NET Calls at European level
given the plethora of ongoing ERA, FP7 and related activities. Web-based
routes will include CORDIS, relevant European Technology Platforms and
other relevant ERA-NET websites. While the promotion through the CORDIS
database will position the Call in the context of European funding
opportunities in general, the European Technology Platform will involve
promotion through the RTD community itself.
• The new NETWATCH web portal will be another route for exploitation here.
Current plans include a Call Calendar which will be a core element in the
transparency of Call Promotion.
• Call announcement to European and international level bodies/networks will
include presentation or exhibition at international meetings and conferences
as well as publication in major European journals, including the Official
Journal of the European Union.
CORDIS, relevant European Technology Platforms and other relevant
ERA-NET websites
•
•
•
•
The ERA-NET Scheme on CORDIS
ERA-NET projects
ERA-NET and FP7 Calls
Information on all European Technology Platform activities
NETWATCH web portal
• ERA-NET Learning Platform and NETWATCH
Presentation or exhibition at international meetings and conferences
The Launching event for the ERA-NET Learning Platform and NETWATCH System took
place in Brussels on 31 March and 1 April 2009. It will be an annual single entry point
to the ERA-NET community and policymakers.
Publication in major European journals
Including the Official Journal of the European Union
Publication on web-sites of major European initiatives featuring calldata of any kind
• INSME
Page 52 of 154
2.2.5 Providing services to applicants
Many ERA-NETs provide additional services to applicants, mostly with regard to
partnering, which has been identified to rank among the biggest difficulties faced by
applicants, and concerning intellectual property rights (IPR). These services are not
directly call-related, but might influence the call impact through support of the
applicants.
• Finding a partner for European cooperation might often pose problems to
applicants, particularly for not experienced ones and those involving in
cooperation for the first time. Within applied research, particularly if industry
is involved, there are often partnering events. These can be promoted via the
website of the ERA-NET.
Moreover, some ERA-NETs have developed electronic partnering tools, which are
particularly dedicated to the topics of the ERA-NET and where potential applicants can
promote either their project ideas and/or provide their company profile.
Examples of and links to such partnering sites are found below.
http://db-ictagri.eu/pub/iproject_intro.php
https://www.etbsubmission.eu/partner/
• Providing information / guidance on the negotiation of IPR between project
partners has become an increasingly important service in the relation between
ERA-NETs and applicants. ERA-LEARN provides an introduction to IPR issue for
various levels, and an exhaustive collection of useful links in a separate tool,
from which respective services can be derived.
Page 53 of 154
3. Submission
3.1 Submission of pre/full proposals
3.1.1 Submission of pre/full proposals examples
3.1.1 Overview of principal ESS function
3.2 Submission of national/regional funding application forms
3.2.1 Guidelines for applicants
3.3 Distribution of submitted proposals to involved funding organisations
Contents Page
3.1 Submission of pre/full proposals
There are several methods to implement proposal submission, ranging from hard
copies or discs to electronic versions which are sent to either agencies
(decentralised) or a single office (centralised). The most appropriate submission
system of your ERA-NET will depend on your project goals, i.e. whether your focus is
on other joint activities than calls and your ERA-NET will (eventually) launch one
single call or whether your focus is on launching several joint calls. In practice
electronic methods have replaced other forms of submission.
1. Proposal submitted by E-mail or drop box
In case you envisage only a single call and/or if you expect only few proposals per
call, you can avoid deploying efforts and resources in establishing complex submission
systems. A first step towards electronic proposal submission would be submitting files
by e-mail:
• To the agencies involved, (ensuring that single applicants (participants of a
project consortium) submit identical copies of their proposal); or
• To a central office with the advantage of having all data collected on a
single site (the office must then further distribute the proposals to the
agencies/evaluators).
In any case, you must make sure that security/confidentiality issues are considered
(e.g. secure e-mail/website). Compliance of submission with cut-off date and call
closure time must be checked.
Page 54 of 154
A web based drop-box system (access by user key and password) has been
successfully applied to handle some 30 proposals. In terms of security issues, it is
considered to be safer than E-mail submission.
2. Proposal submitted to submission systems
In case you focus on joint calls, you are recommended to go for a fully developed
(though still easy enough to implement) solution, which is a centralized electronic
submission system. Examples of such systems are presented in section 3.1.1
The following description of function and suggestions are based on research of
submission systems of existing ERA-NETs that have been successfully employed.
• Functional submission system requirements
The main function is the proposal upload. A submission system must be
hosted at a secure site and should allow access for authorized users only. An
average submission system is expected to handle easily 20-100 proposals
with 5-15 partners involved in the call.
Many currently implemented submission systems offer additional functions
(mainly used in the evaluation process) such as:
• Automatically generated notification to applicants,
• Download of the proposals by respective funding organizations, thus
providing a distribution function of proposals
• Information-hub with restricted access only for authorized individuals,
which is mainly used for evaluation purposes, but also for
monitoring. It includes up- and download functions and a database
of proposal and evaluation data.
• Automatically generated call statistics.
These systems are termed Electronic Submission and Evaluation systems (ESS)
• Basic technical requirements
• A safe site to upload proposals, including the function of
automatically generated notifications to applicants
• A selectively accessible data-hub for various purposes (evaluation,
project follow up and monitoring [if a system is in place])
• System provenience
Most submission systems in place have been developed by and are being
maintained by an ERA-NET consortium member. Most are hosted at the ERANET webpage or on that of an agency (the other ones being hosted by
Page 55 of 154
subcontractors). Often the submission system is derived or adapted from a
submission system already in use for national/regional calls; sometimes the
system development has been subcontracted.
An analysis has shown that almost all currently applied submission system
are highly recommended by their users, but hardly available for commercial
use. Recently, some funding agencies provide their ESS for external use. The
majority of ERA-NETs considers their system convenient for agencies and
applicants, respectively.
Apparently, the establishment of an ESS system or the adoption of an existing
system did not bring about difficulties for any of the ERA-NETs, and individual
systems have been developed repeatedly with almost the same functions.
• Recommendation for access to ESS
There is a quite uniform common standard of ESS in place. There is no
indication that establishing an ESS with the above requirements and
specifications was difficult to reach (technically) nor particularly expensive.
Developing or adapting an ESS thus appears to be no major issue in
implementing different call procedures. Apparently many ERA-NETs found
access to an existing ESS through one of the agencies of the ERA-NET
consortium because of the availability of a system within this agency, or
because of its involvement in another ERA-NET.
Therefore, most probably one of the ERA-NET consortium partners will have
access / institutional bonds to an ESS, which is likely to be adaptable to
individual needs of your ERA-NET. There are few systems commercially
available, mostly in ERA-NETs where the system runs independently from an
agency, or where the development has been subcontracted. Sometimes,
successful subcontracted solutions have been integrated into the systems of
national/regional agencies, the sector is strongly performing.
Meanwhile, also external access to an ESS is offered by several experienced ESS
providers, and there is at least one system developed by ICT-AGRI, which will be
made available as open-source for direct use or further development by individual
ERA-NET consortia.
An overview of essential ESS functions, the context of the ESS within a call, and some
profiles of ESS providers are presented in section 3.1.2.
See also PLATFORM recommendation "Best practices and recommendations for
effective and cost-efficient Call management"
Page 56 of 154
3.1.1 Submission of pre/full proposals examples
ESS Flow Charts examples:
Page 57 of 154
Page 58 of 154
ESS Portal Examples:
Page 59 of 154
Page 60 of 154
3.1.2 Overview of principal ESS functions
Availability, capacity and costs of existing electronic submission systems (ESS)
Basic settings
A secure website on a server is provided by the ESS provider. The ESS allows
applicants to upload proposals and also allows agencies for (technically) managing
international peer review evaluation. The final deliverable is a numerical ranking list
across all submitted proposals (based on scores, a potential consensus meeting by a
panel is not included here).
Default scenario of call/ESS tasks
• Call with 1-step procedure, 5-30 participating programs, up to 500 proposals
• Proposal reception (online or online + PDF) and automatically generated
notification to applicants
• Access to the proposals by respective funding organizations, thus providing a
distribution function of proposals (eligibility / evaluation). Allowing for
assignment of proposals to individual evaluators.
Page 61 of 154
• Information-hub with restricted access only for authorized individuals
(agency, expert/evaluator, evaluator panel), which is mainly used for
eligibility / evaluation purposes (optionally also for monitoring):
• Download of proposals
• Upload of review sheet / evaluation results
• Outcome: Ranking list of evaluated proposals
• Monitoring and Automatically generated call statistics (optional)
BUSINESS
MODELS OF
ESS
ACCESS AND TECHNICAL REQUIREMENTS FOR USERS
• ESS is hosted by provider and offered as complete service (including
operation) => ESS full service provider
• ESS is hosted by provider and offered to be operated by ERA-NET call manager
(agency) => ESS provider for external use
• ESS is available for charge / for free, will be hosted, adapted and/or operated
by customer => ESS for internal use
The former two solutions require an internet access only, the latter option will require
a server, a basic IT knowledgeable person, and might also require specific knowledge
of the programming language of the acquired ESS (in case of privately programmed
ESS)
Variables not included into the relationship between ESS provider and user
• Eligibility / Evaluation details: only the technical step of handling the evaluation
is included into the ESS scenario, users must develop an own
scenario/mechanism or follow the default of the system
• Evaluators (selection, appointment and costs of experts/evaluators)
• Concept / Organization of a consensus meeting
• Monitoring / Automatically generated call statistics
The context of the ESS and its environment is presented in section 3.1.2.1.
More features of ESS and ESS Providers: ESS providing the above features
will also have the following specifications
• Capacity (handling proposals): easily 250, up to 10.000-15:00 possible
• Capacity (bottleneck deadline): As proposals tend to be submitted shortly
before deadline, a possible adjuvant to many expected proposals is staggered
deadlines (e.g. for sub-calls) as a technical feasible solution
• Capacity (evaluation handling): unrestricted, but over 500 proposals are a
challenge for the call coordinator
• Adaptability: Lead time (order-to-product): 2-3 months
• Adaptability: Costs (non-profit): 1-2 (-4) PM
• Trouble shooting technical problems in the interaction of ESS with applicants
and evaluators is usually provided
Page 62 of 154
Capacity of ESS
Basically all ESS systems do not have technical limitations and can operate whatever
kind of call and handle a sufficient number of proposals. Limitations are only expected
(but can be solved in advance by a capacity increase) in the system stability (e.g.
receiving 500 proposals within the last hour before deadline). Specific call- and
capacity settings can be developed from the default scenario. The ESS capacity (in
terms of technical effectiveness) appears to be not a limiting factor in any case.
Availability (and business models):
Currently, a number of systems are available; some examples are listed below; for a
more detailed description check section 3.1.2.3:
ESS provider
Availability
1. ICT –AGRI *
From 2014 onwards
2. AiF **
From 06/2014
3. DLR **
Available
4. JÜLICH **
Available
5. VDI/VDE **
Available
6. e-Intelligent **
Available
* ESS for internal use
** ESS full service provider & ESS provider for external use
Adaptability of the ESS
Users may follow the default of the system or must develop an own
scenario/procedure. All systems are adaptable to specific requirements, e.g. to
include a 2-step evaluation procedure and/or to add a panel meeting. Modifications
will be done by the ESS provider in mutual accordance with the user, except for the
"ESS for internal use" model, where potential modifications have to be negotiated
with the seller / done by the user.
The roles of the call manager/participating agencies and the ESS provider and the
workflow of the interaction are presented in section 3.1.2.2.
Page 63 of 154
Cost factors
Aspects
ESS (basic system)
ESS modifications (for specific features)
ESS & operation during call
Cost (rough
estimate)
Timeline to
transfer/ set-up
Availability
(of provider)
No production
costs*
Ready
Yes
1-2 (-4) PM
30-60 (-100) days
Yes
≤ 1 PM
During call
Yes
Operating evaluation (full service)
Dependent on call conditions
On request
Handling per proposal (eligibility / 1 step
evaluation, full service)
Dependent on call conditions
On request
Evaluation results (exporting database)
No costs charged
Yes
Evaluation summary (numerical ranking)
No costs charged
Yes
* except (potentially) for "ESS for internal use"
The call specifics need to be known in order to develop a complete costfunction. Tentative (and roughly estimated) figures are ranging between 2 and 4 PM
(and respective costs of approximately €20.000 – 40.000) charged by the provider for
providing the ESS, doing (reasonable) modifications from the default, and for
technical assistance during the process.
Page 64 of 154
Additional costs and efforts to be considered (not related to the ESS itself)
• Development of the call specifications and procedure
• Definition of the order to the ESS provider (will be the most influential aspects
for a provider cost estimation)
• One-step v. two step process (the latter approach means more work and has an
extended timeline)
• The number of proposals appears to be less important (except for full services)
but will indirectly influence evaluation costs (both for handling and fees of
evaluators)
• Eligibility and evaluation efforts
3.1.2.1 Function and Environment of ESS
Page 65 of 154
3.1.2.2 Role of and Interaction between ESS Providers
Page 66 of 154
3.1.2.3 ESS provider and specification of their services online
1. ERA-NET ICT-AGRI (DASTI, DK)
ICT-AGRI (Information Technology and Robotics in Agriculture) introduced the "Meta
Knowledge Base" for their own purposes. It collects metadata of projects and partners
and provides it on-line for about 1100 users at the moment. Through Member login
one can join forces in a secured setting. There are opportunities to create private or
public project-proposals. The tool also serves for matchmaking and users can share
and search for project profiles. It is also offering to enter specifications of
participating programs and has other functionalities.
This content management system offers an ESS within the much broader system and
is being transformed into an open source system for matchmaking and on-line
submission and evaluation of applications. It will be an open source system to be used
by all ERA-NETs. Moreover, national regulations (funding schemes; themes; eligible
funding) can be provided automatically.
The ESS works in principle like the EC submission system. Electronic application forms
as well as a tool for on-line evaluation are available. The system also provides a tool
Page 67 of 154
to automatically calculate budgetary items, which is supportive to the discussion
about optimizing the number of funded proposals, so this tool can optimize ranking of
equally ranked proposals according to the availability of funds. Apart from the default
scenario, all modifications are feasible.
The system will be made available for free by 2014; users will have to host it on their
own servers. Users may adapt/modify the system according to their own
specification; the programming does not require a specialist, but a skillful IT person. A
manual and a user forum will be provided; the latter will be moderated by DASTI.
It is estimated that very low costs will be needed to operate the system: You will need
≤ 1PM to get acquainted with the system and probably another 1 PM for making the
adaptations (if required).
Contact: Iver Thysen [email protected]
2. ESS from AIF (DE)
This ESS is employed in the sustainable ERA-NETs Cornet and ERA-SME. It can handle
up to 1500 proposals and provides all essential features as described in section 3.1.2,
also further modifications are feasible. Currently it is available only for networks in
which AIT is a partner, but it will be available for external use and as a full service by
06/2014.
Contact: Cvetkovic Viktoria [email protected]
3. ESS from DLR (DE)
DLR has vast experience in ERA-NETs and also as ESS provider. The ESS cannot be
externalized, but is available for external use and as full service. It is available as a
default and all feasible modifications of the user will be implemented by in-house
personnel. Providing ESS is a regular product of DLR (the agency is non-profit).
Contact: Olaf Heilmayer [email protected]
4. ESS from JÜLICH (DE)
JÜLICH is involved in many ERA-NETs (and one JTI) for which it provides the ESS. In
addition, JÜLICH also acts as a ESS provider (for external use and full service), but
the system cannot be externalized. Apart from the default system, JÜLICH has vast
experience in adapting specific solutions far beyond the usually requested corefunctions (e.g. partnering site, monitoring system, etc). Modifications are done by inhouse personnel and/or by subcontractors (according to the required skills).
Contact: Tinois Nicolas [email protected]
Page 68 of 154
5. ESS from VDI/VDE (DE)
VDI/VDE operates ESS in several ERA-NETs and is also established as an ESS provider
(for external use and full service). All modifications from the default system can be
implemented and VDI/VDE has a dedicated and experienced IT team so that specific
solutions can be implemented very quickly and at short-term.
Contact: Peter Hahn [email protected]
3.2 Submission of national/regional funding application forms
Usually a common proposal is submitted to the central ERA-NET call secretariat by the
project consortium coordinator on behalf of the project consortium. This proposal will
be evaluated according to the ERA-NET evaluation system in place.
It might be required that, for legal reasons, one or more of the involved agencies
require an additional, separate national/regional application. This may be the same
proposal (e.g. as a signed hard copy), or a short online application in the national
language, or an additional form etc.
These national/regional requirements should be explicitly stated in the guidelines for
applicants.
3.2.1 Guidelines for applicants
The following Guidelines for applicants can be used as a template:
1. INTRODUCTION
•
•
•
•
•
Description of acronym for the ERA-NET
Objectives of the ERA-NET
Previous activities of the ERA-NET
The objective of the Call
How the Call will be evaluated and managed jointly
2. SCOPE AND OBJECTIVES
• Detail the objective of the Call
• Consortia requirements
• Total available budget and its source
Page 69 of 154
3. CALL STRUCTURE
• Table 1. Funding Agencies and Programmes participating in the ERANET Call
• Country/Region
• Participating Funding Agency
• Accessible Programmes
• Best practice: use of graphic map – click to get a detailed description of
the participating funding organizations and their relevant programmes.
• Call Process
• Figure 1 to illustrate the schematic workflow of this one-stage or twostage Call
4. CALL ANNOUNCEMENT
•
•
•
•
•
•
•
•
•
Objectives
Topics
Funding rules
Eligible consortium structure
Confidentiality
Proposals and any information relating to them shall be kept confidential
amongst the ERA-NET funding agencies. Proposals shall not be used for any
purpose other than the evaluation of the applications, making funding decisions
and monitoring of the projects. International experts are required to sign a
confidentiality agreement prior to evaluating proposals. Projects submitted to
the second evaluation stage will be required to include a publishable summary
of their proposal. If the project is offered funding, this information will be
published on the ERA-NET website. All other project details are kept strictly
confidential.
Consortium Agreement
A Consortium Agreement between the project partners will be required. In order
to accelerate the selection and contract offer process, an outline of the
Consortium Agreement should be submitted with the Full Proposal. Funds may
not be released to the consortium by some funding agencies until a signed
Consortium Agreement is available. Models for Consortium Agreements can be
obtained from national and regional funding agencies or from the EC IPR
Helpdesk: http://www.ipr-helpdesk.org.
Project budget
National/regional
programme
specific
rules
apply.
Contact
your
national/regional funding agency for further information. It is recommended
that the budget for each partner does not exceed the typical national/regional
limits for the respective partners. The national/regional agencies can provide
detailed feedback on possible eligibility criteria.
Exchange of personnel
Project duration
Page 70 of 154
Project duration is limited to a maximum of X years.
• Time schedule
• Call opens [00/00/2009]
• Cut-off date for Pre-Proposals +2 months (6.00 pm Brussels time)
• Invitation to submit a Full Proposal +5 months
• Cut-off date for Proposals +3 months (6.00 pm Brussels time)
• Selection and funding decisions +3 months
• Start of project funding
5. SUPPORT FOR PROPOSERS
• Helpdesk
The Helpdesk consists of the participating funding agencies. All funding agencies
participating in the Call have been informed about the coordinated Call
procedures and will provide assistance to project proposers in case of questions.
• Frequently Asked Questions (FAQ) are listed in the website.
6. PROCESS OVERVIEW AND SELECTION CRITERIA
• Pre-Proposal
The Project Coordinator submits a Pre-Proposal via the on-line submission tool
available at the ERA-NET website. A guide to the electronic submission system
is available on the website. An email confirming submission of the Pre-Proposal
will be issued automatically to the Project Coordinator. The information detailed
in the Pre-Proposal may not meet the requirements of all the ERA-NET funding
agencies. Thus project partners must contact their local funding agency to
identify any additional national/regional requirements.
• National/Regional Evaluation
The national/regional agencies make their own evaluation of the respective
funding applications, based on the role and activities of the national/regional
partners viewed in perspective of the overall transnational project. As part of
the eligibility assessment it is necessary for all the individual national/regional
evaluation criteria to have been fulfilled for any given project.
• Eligibility criteria for Pre-Proposals
For the eligibility assessment of Pre-Proposals the following criteria will apply:
• Compatibility with the scope of the Call
• Eligibility of the consortium
• Relevance to national/regional programmes and strategy
• Added value through transnational cooperation
• Is the project clearly described [aims, work programme, outcomes]
• Clear, realistic and defined plan for the exploitation and dissemination of
results
Based on the outcome of the Pre-Proposal evaluation, selected projects will be
invited to submit Full Proposals.
Page 71 of 154
• Full Proposal
The Project coordinator submits a Full Proposal via the on-line electronic
submission tool. A submission receipt will be issued automatically to the Project
Coordinator.
• Common Evaluation
The Full Proposal is evaluated by international experts in accordance with the
requirements of the Commission.
• Evaluation criteria for Full Proposals
• Scientific and/or Technical Excellence Threshold (3/5)
• Quality and efficiency of the implementation and the management
Threshold (3/5)
• Potential Impact Threshold (3/5)
• Overall Threshold (9/15)
• Funding Decisions
Eligible projects (i.e. those over the threshold) will be ranked in order of overall
score. The ERA-NET will prioritise the funding of each project based on its
position on the ranking list. The number of projects that will receive funding will
be subject to the availability of funds.
7. FUNDING
• Contract
Funding contracts are dealt with directly between the project partners and
their national/regional funding agencies. Each project partner will be
responsible for the necessary reporting to their funding agency according to
national/regional rules in order to obtain and maintain funding during the
lifetime of their portion of the project.
• Start and Instalments
A pre-condition before transferring the first funding instalments is the
existence of a Consortium Agreement (including IPR) between the project
partners. As the funding contracts may, in principle, become effective at
different dates, the individual contracts do not need to start, receive funding,
or be reviewed at exactly the same time. However, it is recommended that all
the partners start (and also finish) the project at the same time.
8. PROJECT MONITORING
Reporting and monitoring mechanisms should be applied to ensure accountability.
These are:
National/regional project monitoring of the project (carried out by the funding
agencies in the various countries/regions);
ERA-NET consortium monitoring.
National/regional project evaluation
Page 72 of 154
The progress of each individual contract will be monitored through specific project
evaluation processes established by the respective national/regional agencies.
• Project monitoring
The ERA-NET will monitor the progress of funded projects. Project co-ordinators
will be required to provide a short annual report outlining the progress of the
project and the milestones/deliverables obtained.
• Change in active projects
Actions to be taken in the case of major changes in an active project (e.g. a
change in consortium composition, etc.) must be addressed in the Consortium
Agreement. Members of the consortium must be aware that any changes in its
composition may have an impact on funding. The funding agencies involved
must be immediately informed about the changes by the project coordinator.
9. DISSEMINATION
• Promotion of the ERA-NET Projects
Project partners are required to refer to the ERA-NET in publications,
exhibitions, lectures and press information concerning results of the ERA-NET
projects. To demonstrate the added value of transnational cooperation projects,
results from the Call shall be disseminated. This process can be tackled via
different channels, e.g.:
• High level conferences with relevant stakeholders to inform widely about
the project results;
• Publication of a short outline of funded projects on ERA-NET and
national/regional websites;
• Press conferences and workshops.
This information may also be used by the ERA-NET for further dissemination.
However, further details of projects are strictly kept confidential. They should
only be published with the prior agreement of the project partners.
10. FURTHER INFORMATION
For further information reference
national/regional funding agency.
the
ERA-NET
ANNEX 1 – LIST OF ERA-NET CALL DOCUMENTS
•
•
•
•
•
•
ERA-NET Flyer
Guidelines
Frequently asked questions (FAQ)
Pre-proposal Template
Pre-Proposal Evaluation Form
Guide to Electronic Submission System
Page 73 of 154
website
or
contact
your
3.3 Distribution of submitted proposals to involved funding
organisations
All submitted proposals must eventually be forwarded to the evaluators, usually either
via the involved funding organizations or directly to the experts and/or the evaluator
panel, depending on your evaluation system in place.
All ESS developed by different ERA-NETs so far have implemented an upload and
download page ("selectively accessible data-hub"), where evaluators can download
proposals and upload evaluation results.
Page 74 of 154
4. Evaluation
4.1 Introduction
4.2 Expected results from evaluation
4.3 Evaluation procedures to be agreed on
4.3.1 Centralized evaluation
4.3.2 Decentralized evaluation
4.4 Evaluation criteria
4.5 Forms and documents
Contents Page
4.1 Introduction
ERA-NETs have already implemented more than 280 joint calls (period 2004-2012).
Various procedures have been selected by the participating funding agencies, each
tailored for specific ERA-NET consortia, constellations and targets.
The large variety of procedures already in use by ERA-NETs makes it a challenging
task to identify common standards for the evaluation & selection of transnational RTD
projects, because the choice of applicable and appropriate procedures may depend on
the individual setting, e.g. basic research or applied research, the chosen funding
mode or national programme requirements.
In any case the overall target of the evaluation must be the selection and funding of
the best proposals through transparent systems that are convenient for funding
organisations and applicants. Also, for the sake of transparency, it might be good to
recruit an independent expert to monitor and report about the evaluation procedure.
To achieve the common goals of an ERA-NET, commitment of the participating
organizations to the agreed evaluation procedures is essential for the successful
evaluation process and optimal outcome.
In the case of ERA-NET Plus calls, EC has imposed some specific requirements:
International Peer Review process is mandatory, and the primary evaluation criterion
is excellence. The 2-step procedure, which is mandatory in ERA-NET Plus actions, will
open in chapter 4.3 Procedures – Centralised evaluation .
Page 75 of 154
The co-funded ERA-NET Cofund call continues further the principles of an ERA-NET
Plus call:
• 2-step submission and evaluation procedure
• step 1 call: needs to be open at least for 60 days, EC to be informed 30 days
in advance
• step 1 evaluation: review of proposals either at national or transnational level
• step 2 evaluation: a single international peer review process
• each proposal in step 2 is evaluated with the assistance of at least three
independent experts
• evaluation criteria: 1) excellence, 2) impact, and 3) quality and efficiency of
the implementation
• selection of the projects follows the ranking list (binding ranking)
• only proposals with identical scores at the threshold of available funding may
be selected according to the availability of funds in order to maximise the
number of selected projects
• minimum criteria for the projects: two independent principal investigators
from two different EU Member States or Associated countries
It must be pointed out here that the conditions for call implementation of the EC
co-funded ERA-NET Cofund call do not apply to the additional calls of the ERA-NET
Cofund without EC top-up funding.
You will find detailed descriptions and material that have proven valuable in
various cases. A range of issues need to be considered, including:
• Call launched by ERA-NET or ERA-NET Plus or co-funded ERA-NET Cofund call
(with EC)?
• Outcome of evaluation: common funding decision or joint recommendation?
• Evaluation criteria
• Evaluation procedure (1-step or 2-step procedure)
• Assessment against national rules – inclusion of national requirements
• Methods for the selection of experts
• Check whether there is one common proposal form or a separate
national/regional funding application
• Organize evaluation meetings appropriately
• Confirm the available funding before evaluation
See also PLATFORM recommendations "Best practices and recommendations for
effective and cost-efficient Call management"
Page 76 of 154
4.2 Expected results from evaluation
The most appropriate evaluation procedures will be chosen depending on the type of
the expected evaluation result.
Evaluators are either concerned with compiling a list of proposals suitable for funding
(i.e. outcome is a recommendation to the decision makers, possibly without any
ranking of proposals) or they are also concerned with ranking the proposals according
to the defined criteria (i.e. outcome is a recommended ranking list that in the
case of ERA-NET Plus and co-funded ERA-NET Cofund call will be a binding ranking
list; see below).
In certain cases it may be preferable to come up with an alternative system instead
which leaves room for the required flexibility, e.g. categories such as "recommended
for funding" and "recommended for funding, if funds are still available" vs. "not
recommended for funding", i.e. a colour code (green-yellow-red) or similar.
Categories can also be used to add on and clarify the recommended ranking list.
It is of central importance to know at which step the national / regional criteria
(either as program criteria or as criteria of the Research Councils) are taken into
account. Experience shows that it is preferable to collect decisive input from the
national/regional organizations as early as possible and to carry on at a collective
level in order to have a smooth and coherent progress later in the evaluation
phase. After this eligibility check, the participating organizations should commit
themselves to the joint evaluation procedures agreed earlier.
In the case of ERA-NET Plus and co-funded ERA-NET Cofund call, a 2-step
procedure, a mixed funding mode (or a real common pot), and a binding ranking
list are used to enable the funding of the best proposals. The requested and available
budgets should be outlined as well.
It is recommended to have all budgetary modifications and amendments completed
and prepared at the evaluation meeting, since any later alterations in the national
budgets influences the EC contribution and, therefore, the funding available. Thus,
communication of the ERA-NET Plus and co-funded ERA-NET Cofund call results
to the EC is essential. At the end of the evaluation, the consortium/coordinator of the
co-funded ERA-NET cofund call must submit to the Commission the following
documents:
• The ranking list(s) of the projects
• The observers' report on the evaluation
• The joint selection of the projects to be funded
Page 77 of 154
• From each consortium partner (funding organization) participating in the cofunded call, a formal and duly signed commitment on availability of funds for
the selected projects
• Information on each project selected for funding, including data on each
participant (principal investigator) as well as abstracts of the project
proposals.
4.3 Evaluation procedures to be agreed on
Irrespective of the chosen workflow, a typical evaluation includes the following items:
a) Eligibility check
1. Check eligibility (must always include check for national requirements/
conformability and should be done before the proper evaluation phase)
2. Assess roughly the compliance with and/or suitability/applicability to the
scope of the call
b) Basic tasks during the evaluation
1. Assess project quality
2. Compile, match and agree on results
3. Communicate results to applicants
c) Basic elements to carry out the evaluation
1.
2.
3.
4.
Proposals
Evaluators
Resources
Evaluation meetings (physical or virtual)
d) Basic modalities and conditions
1. Process design and required outcome
2. Amount of flexibility applied
3. Absence of conflict of interest
4. Evaluation scores or other systems
5. Evaluation criteria: national requirements and joint ERA-NET criteria
Page 78 of 154
The most appropriate workflow will be selected by each ERA-NET based on the
following aspects:
Issue
Eligibility
National rules
Method of implementation
Eligibility check (both centralized & desentralised evaluation)
How to integrate national rules into the overall evaluation? (usually the call text is
linked with separate appendixes)
Centralised evaluation
• assessments by independent
experts and/or a panel
• common expert panel/ scientific
steering committee
Decentralised evaluation
• assessments by separate experts
(if applicable)
Evaluation
result
Scoring and ranking of proposals (focus
on scientific excellence), and if helpful,
classification also in categories
Classification in categories
(focus on project feasibility and scope e.g.
innovation aspect)
Steps of
evaluation
1 step or 2 step procedure (consider legal constraints and the amount of potential
proposals; evaluation steps 1 and 2 can be integrated in a single procedure based on
a single proposal)
Type of
evaluation
Special cases
• evaluations according to ERA-NET's
rules
Which criteria cannot be assessed by external evaluators?
How are they assessed?
The Funding mode (common pot model) will bias this decision:
While real common pot and mixed mode are usually associated with centralized
evaluation and will have to produce a ranking list, decentralized evaluation is often
used when the virtual common pot model is applied (the latter common pot model is
open to any type of evaluation, however).
While it is not possible in the frame of this manual to identify all possible concepts of
running an evaluation it is possible to describe some of the most common schemes:
Centralised evaluation carried out by common expert panel/ scientific councils
Decentralised evaluation carried out by national programmes/research councils
Page 79 of 154
4.3.1 Centralised evaluation
Centralised evaluation is run by the call coordinating office of the ERA-NET, and the
evaluation is done by external reviewers and/or an expert panel/a scientific
committee. This is a typical evaluation system for reaching a joint ranking list based
on scientific excellence. Most activities are carried out at the ERA-NET level.
Usually there are no constraints on the feasibility of performance (basic science). It is
a standard procedure in the basic sciences where expertise is obtained by external
reviewers. It is completely appropriate to use this system for all real common pot and
mixed mode models. It also complies with the step 2 rules for ERA-NET Plus and cofunded ERA-NET Cofund call.
For an ERA NET Plus and co-funded ERA-NET Cofund call, two evaluation steps are
required, but this is advisable to use also in basic science ERA-NETs where a high
number of proposals are expected.
First, the national eligibility and the fulfilment of all national requirements (i.e.,
eligibility of the project participants / proposals, financial liability check, etc.) will be
assessed by the funding agencies (= eligibility check).
The outcome of the first evaluation step will be a list of proposals invited to submit
full proposals. It is important that only the proposals that are fundable by all national/
regional funders are invited to step two. The external reviewers will assess the
scientific quality of the proposals as well as if the projects fit in the scope of the call. A
first balancing of the call budgets available with the requested budgets is advisable.
In the second evaluation step, international peer review process is used to select
the best proposals according to excellence criteria. Evaluations from at least three
independent experts per proposal are required in the ERA-NET Plus and ERA-NET
Cofund call. In the following panel discussion meeting, consensus evaluations are
formed and a binding ranking list of fundable proposals is produced. The requested
and available budgets should be outlined as well. The binding ranking list forms the
basis for proposal selection and national/ regional funding decisions. Only proposals
with identical scores at the threshold of available funding may be selected according
to the avalailability of funds, in order to maximise the number of selected
projects. Adjustments to the ranked project selection list are not allowed after the
evaluation meeting.
In the ERA-NET Plus and Cofund call, reviews from at least three independent
experts per proposal are required in the second step of the evaluation. Furthermore,
the evaluation results have to be communicated to the EC (see Chapter 4.2). The
consortium must appoint an external observer, who will follow the whole evaluation
Page 80 of 154
process and the ranking meeting especially, and based on the observations produce a
report for the EC. In addition, it is recommended that a representative of the EC
participates the evaluation meeting.
Preparation:
Before selecting the experts, the ERA-NET consortium must agree on a definition of a
potential conflict of interest (nationality of evaluators, economic competitors,
organization overlap, family relationships, close collaborations etc.). The ERA-NET
consortium decides how to assign proposals to individual evaluators.
Resources need to be considered: costs and payment of evaluators, availability for
evaluation meetings, evaluation reports, monitoring of the projects.
Page 81 of 154
Appoint evaluators:
A joint expert data-base should be compiled via exploitation of national expert
databases, European databases, national and international thematic networks.
Research funders participating in ERA-NETs have the possibility to request access to
the Commission's database of experts for research activities and can use this pool of
more than 90.000 qualified experts for evaluations. How to access the Commission's
data base of experts is described here.
Before the evaluation, a panel of experts must be constituted. The ERA-NET
consortium has to agree on the composition of the panel with respect to
representation
of
countries,
experience
of
evaluators,
thematic
and/or
interdisciplinary expertise, and gender aspects.
Evaluation scores and/or categories:
Allocating scores to criteria will lead to a clear ranking list of proposals. Ranking lists
usually imply that the best ranked projects are funded. Therefore a common
understanding of the evaluation criteria is essential (e.g. via briefing of evaluators or
guidance for scoring in a side document)
Optional: Pre-proposals (1st step of 2-step evaluation procedure):
Required for ERA-NET Plus and ERA-NET Cofund calls, optional for other ERA-NET
calls.
Assessment of pre-proposals:
A pre-proposal phase enables the agencies to identify project consortia that are
unlikely to succeed in the full proposal evaluation. Pre-proposals can also facilitate a
tailored and timely search and assignment of evaluators.
Handling of the pre-proposals after eligibility check:
• evaluation of proposal quality and suitability to the call
Only eligible proposals are considered in this step. Non-eligible proposals will be
rejected and this decision will be communicated via the call secretariat. The
participating organizations have to make sure they have the legal prerequisites to
reject a proposal at this stage.
Assessment of pre-proposals could be done remotely. Even with limited details
available in the pre-proposal it should be possible to assess the excellence and
innovation.
Page 82 of 154
National budgets:
It will be helpful to confirm the budgets prior to the evaluation in order to estimate
how many projects could be funded. The best and most transparent solution is to
collect the commitments of the funding organizations before launching the call. The
call text may include links to the separate commitments by different funders.
Consensus meeting: compiling results of pre-proposal assessment(s):
The call secretariat compiles the outcome of the assessments and coordinates the
elaboration of a list of project consortia that are invited to submit a full proposal. This
can involve external reviewers or the experts of the funding organisations.
Feedback to applicants and invitations to step 2:
Applicants will appreciate quick feedback. Invitations can be sent to successful project
consortia and should include guidance for any requirements and recommendations for
step 2. Non-successful applicants should also receive clear feedback from the call
coordination office of the ERA-NET, probably referring to more detailed information by
the participating agencies (in order to decentralize the workload). In any case the
content of the feedback must be consistent and mutually agreed by all partners
involved.
For the latter case, it will be helpful to arrange for a comprehensive collection of
available background information on the proposals and evaluation results among
agencies in order to ensure coherent communication of the ERA-NET with individual
project participants.
Full proposals (2nd step of 2-step evaluation procedure):
Assessment of full proposals (and national funding applications, if
applicable):
The full proposals contain the complete project information. This is the common basis
for centralised evaluations. It is therefore recommended to provide a joint ERA-NET
proposal form.
The applicants and/or project consortium leaders must confirm that the proposal or
parts of it has not been funded in another call – double funding is illegal.
The eligibility of submitted full proposals will be checked (i.e. alterations to the preproposals are considered, new information becomes available).
Eligible proposals will be assigned to evaluators according to their specific expertise.
The evaluation will follow strictly the evaluation criteria stated in the call text and in a
Page 83 of 154
separate instructions for reviewers (optional), and for this purpose full proposal
evaluation forms will be provided. Evaluators will send their completed evaluation
forms to the call secretariat.
Consensus (panel) meeting: compiling results of full proposal assessment:
Panel of experts will assess a peer review of all proposals. They will discuss on the
basis of the separate evaluation reports and prepare a consensus report for each
proposal. Finally, a ranking list of proposals is produced. Here, a link to a
recommendation of PLATFORM ERA-NET call managers is included.
The program owners (decision makers) will agree on the final (in the case of ERA-NET
Plus/ co-funded ERA-NET Cofund call: binding) list of projects suggested for funding
according to the ranking and available budgets and introduce the respective projects
to their national decision making process.
After this, the call secretariat collects the outcome of the assessment and
communicates the results to the applicants.
Communication of evaluation results to project coordinators:
Applicants will appreciate quick feedback informing about results and timelines. The
ERA-NET consortium must decide what to include in the notification (e.g. summary of
the evaluation results and/or anonymised peer-reviewers' comments).
It will be helpful to arrange for a comprehensive collection of available background
information on the proposals among agencies in order to ensure coherent
communication of the ERA-NET with individual project participants.
Special cases - to be considered individually:
This is to establish a procedure if some of the project participants drop out (e.g.
because they have been identified as non-eligible) - Is the project consortium still
viable? Do applicants get the opportunity to substitute the non-eligible participant,
etc?
It needs to be agreed beforehand if the pre-proposal assessment results in a definite
yes/no decision or if there is some room for flexibility with respect to potential
improvements towards the full proposal step, ie. if project consortia should be allowed
to step 2 despite of major alterations, and/or whether recommendations can be put
"under condition" (e.g. condition to revise/reduce a specific work package). It is up to
the ERA-NET consortium to decide whether or not to allow a rebuttal process.
See also PLATFORM recommendations "Best practices and recommendations for
effective and cost-efficient Call management"
Page 84 of 154
4.3.2 Decentralised evaluation
Decentralised evaluation is carried out by national programmes. This is a good
practice model for calls in which project feasibility with regard to the financial stability
and/or the focus on e.g. innovation aspect rather than an absolute scientific
excellence is a main criterion. It is frequently used in the applied sciences and has
clear benefits for the assessment of the feasibility of projects since often only national
agencies have access and internal knowledge of the applicants.
Within this scheme there is a variable degree of activity at national or ERA-NET levels,
and the joint outcome is often a consensus on projects to be funded but without
ranking priorities.
Decentralised evaluation is not recommended in ERA-NET Plus calls, where scientific
excellence is the primary evaluation criteria and a binding ranking list has to be
produced.
Page 85 of 154
Preparation:
Before selecting the experts, the ERA-NET consortium must agree on a definition of a
potential conflict of interest (nationality of evaluators, economic competitors,
organization overlap, family relationships, close collaborations etc.).
ERA-NET consortium needs to agree about the common evaluation criteria and the
evaluation form to be used by the national programmes/research councils.
Resources need to be considered: costs and payment of evaluators, availability for
evaluation meetings, evaluation reports, monitoring of the projects.
Appoint evaluators:
Programmes will have their usual pool of experts available.
In addition research funders participating in ERA-NETs have the possibility to request
access to the Commission's database of experts for research activities and can use
this pool of more than 90.000 qualified experts for future evaluations. Access to the
Commission's data base of experts is described here. Resources need to be
considered: costs and payment of evaluators, availability of evaluators (evaluation
meetings, evaluation reports, monitoring of the finally funded projects).
Evaluation scores or categories:
Ranking lists usually imply that the best ranked proposals are funded and are thus
ideally suited to real common pot or mixed mode models.
However in ERA-NETs where decentralized evaluation is used, it may be preferable to
come up only with categories such as "'recommended for funding" and "recommended
for funding, if funds are still available" vs. "not recommended for funding", a colour
code (green-yellow-red) or similar. This leaves room for some flexibility in decision
making: A categorisation will allow the funding of some (but maybe not all) of the
good proposals according to the availability of funding and is therefore appropriate for
the virtual common pot model. Funding can thus be optimally distributed to projects
with good quality. A common understanding of the evaluation criteria is necessary
(e.g. via briefing of evaluators, calibration at the beginning).
National budgets:
It will be helpful to check the budgets prior to the evaluation meeting so that you
know how many proposals could be funded.
Page 86 of 154
Optional: Pre-proposals (1st step of 2-step evaluation procedure):
The check of formal eligibility of proposals with respect to the ERA-NET call criteria
could either a.) be done by the call secretariat before the actual evaluation or b.) be
part of a decentralised assessments. In addition, the eligibility with respect to national
programme criteria will be assessed as part of the overall evaluation.
It has proven useful to do it the way b) for two reasons: balanced efforts across the
ERA-NET consortium (as not only the secretariat is involved, which usually is not too
familiar with all national requirements) and timely rejection of non-suitable proposals
(less effort for evaluation and quicker information to applicants).
Non-eligible proposals will be rejected and this decision will be communicated to the
call secretariat and a summary be provided to all involved agencies.
Assessment of pre-proposals:
A pre-proposal phase enables the funding organisations to identify project consortia
that are unlikely to succeed in the full proposal evaluation.
Handling of the pre-proposal check after eligibility check
• evaluation of proposal quality and suitability to the call
Even with limited details available in the pre-proposal it should be possible to assess
the scientific and technical excellence and innovation, the quality of project consortia
and the (economic) potential of applicants to reach the project goals, which is
particularly essential in terms of financial feasibility in the case of applied RTD
projects.
Consensus meeting: compiling results of pre-proposal assessments:
The call secretariat collects the outcome of the assessment by the national
programmes and communicates this to all involved agencies.
For each proposal, all funding organisations involved agree if a project consortium
should be invited to submit a full proposal for the 2nd step or not, in accordance with
the agreed ERA-NET goals. It needs to be clarified in advance how to deal with
diverging national results, and how recommendations / decisions are made.
Feedback to applicants and invitations to step 2:
Applicants will appreciate quick feedback. Invitations can be sent to successful project
consortia and should include any requirements and recommendations for step 2. Nonsuccessful applicants should also receive clear feedback from the ERA-NET.
Instead of a single communication to the proposal coordinator it could be beneficial to
Page 87 of 154
inform all participants of a project consortium in order to emphasise the necessity of
contacting respective funding organisations for detailed feedback.
Templates are provided in section 2.1.1.2.
It will be helpful to arrange for a comprehensive collection of available background
information on the proposals among agencies in order to ensure coherent
communication of the ERA-NET with individual project participants.
Full proposals (2nd step of 2-step evaluation procedure):
Assessment of full proposals (and national funding applications, if applicable):
The full proposals contain the complete project information. This is the common basis
for decentralised evaluations organised by the ERA-NET consortium. It is therefore
recommended to provide a joint ERA-NET proposal form. (Broken Link)
In addition, national funding organisations may have to insist on separate national
application forms according to programme requirements.
Full proposals (and national applications, if required) will be evaluated according to
programme requirements, which can include in-house evaluations or external experts
and/or panels. Eligibility is checked as well.
Double evaluation of proposals needs to be avoided by choosing a suitable method for
integrating evaluation of ERA NET full proposals and evaluation of national funding
applications.
The applicants or project consortium leaders confirm that the proposals have not been
submitted to other calls at the same time – double funding is illegal.
Consensus meeting - compiling results of full proposal assessment:
The call secretariat collects the outcome of the assessment by the evaluators and
communicates the results to all involved agencies. Usually, a meeting (could be a
personal meeting, a telephone or video conference or a virtual meeting) is scheduled.
For each proposal, all funding organisations involved have to agree if a proposal
should be rejected or recommended for funding. In this process, it may be necessary
to explain the individual national results in detail.
The ERA-NET consortium finally assembles a list of proposals which are recommended
for funding by the ERA-NET vs. not-recommended for funding.
All proposals on this list that are considered of high quality must have the support of
all funding organisations involved and budget has to be secured. This list is forwarded
to the decision makers (programme owners) and is considered the ultimate binding
basis for the national funding decisions.
Page 88 of 154
Communication of result to coordinators:
As soon as the ERA-NET has issued the final outcome of the evaluation to the decision
makers, all individual programmes/funding organizations will take their funding
decisions. Since this process will vary from country to country due to decision making
dates it has proven useful to inform the applicants about the end of the ERA-NET
process. Applicants will appreciate quick feedback.
Templates are provided in section 2.1.1.2
This feedback shall inform about the fact that all further steps are taken at the
national level and leaves all communication duties to the national organisations
according to their regulations. The ERA-NET consortium decides what to include in the
notification (e.g. summary of the evaluators comments, etc.)
It will be helpful to arrange for a comprehensive collection of available background
information on the proposals among agencies in order to ensure coherent
communication of the ERA-NET with individual project participants.
Special cases - to be considered individually:
Procedure if some of the project participants drop out (e.g. because they have been
identified as non-eligible or the involved funding programmes are running out of
budget) - is the project consortium still viable? Do they get the opportunity to
substitute the non-eligible participant, etc.
It needs to be agreed if the pre-proposal assessment results in a definite yes/no
decision or if there is some room for flexibility with respect to potential improvements
in the full proposal, i.e. if project consortia should be allowed to step 2 despite of
major alterations, and/or whether recommendations can be put "under condition"
(e.g. condition to revise/reduce a specific work package).
It is up to the ERA-NET consortium to decide whether or not to allow a rebuttal
process.
Page 89 of 154
4.4 Evaluation criteria
Evaluation criteria applied in a Joint Call should be based on the standard criteria used
by the participating national and regional funding programmes and/or be enhanced by
criteria from other transnational instruments, such as FP7, particularly including
transnational aspects.
The ERA-NET will have to decide which criteria to use at which step, when to
implement a formal eligibility check and how to carry out a comprehensive evaluation
which comprises national/regional as well as transnational views. Depending on the
selected workflow (e.g. 1 or 2 step call) different sets of the criteria may be applied at
different steps.
The publication of criteria and thresholds (if applicable) will increase the transparency
of the evaluation process. A list of criteria should be a basic part of guidelines for
applicants, 3.2.1
Check of Formal Eligibility:
The eligibility check will take into account 2 aspects:
Eligibility of project participants:
• Are the involved project participants formally eligible according to the
national/regional AND the Joint Call criteria?
• Preliminary estimation, is the scope of the call correctly addressed by the
proposal and in conformity with the definitions of the call text?
Eligibility of a proposal:
• Submission of all requested documents in time?
• Application form filled in correctly/appropriately?
• Project structure eligible (size and structure of project consortium, duration and
costs of project)?
• Thematic focus of proposal within the scope of the call (including respective
national/regional priorities)?
• Same proposal submitted to other calls as well? (if applicable)
Evaluation of project quality:
Eligible proposals will be assessed against defined criteria. Criteria will be measured
through scores or other system (e.g. categories). Any different weighting of criteria
must be published.
When coming to a conclusion it could be worthwhile (especially in step 1 of a 2-step
process) distinguishing between deficits which can be corrected and definitive
shortcomings.
Page 90 of 154
Examples for criteria are:
• Scientific (technical) excellence and targeted innovation
• Feasibility of the proposal described by a detailed work plan
Example: Is the project description adequate to obtain the desired results
(objectives, resources, deliverables)?
• Resources/implementation
Example: Is the cost calculation reasonable and well balanced?
• Quality of the project consortium & management
Example: Are the project participants experienced and well qualified? Why is
this combination of project participants most suitable for carrying out the
project? Is the management structure adequate? Is the partnership wellbalanced?
• Impact: exploitation of results, dissemination
Example: Is there a clear exploitation strategy? How will results be published
(incase of basic science)? Is the expected project outcome sufficiently close to
the market (in case of applied RTD)? Is there economic impact for each of the
project participants
• Transnational benefit & added value
Example: Is it necessary to perform the project at transnational level, i.e.
what is the added value of collaboration for the project? How does each
project participants benefit from the proposed transnational cooperation? Are
there medium - to long-term advantages? Handling of IPR issues?
Respecting the transnational nature of Joint Calls, the added value /
transnational benefit should be of high priority. In addition, particularly in
applied science projects, in which potential beneficiaries are supposed to cofund the project, the financial stability of the applicants is an important
criterion for project viability.
Page 91 of 154
4.5 Forms and documents
4.5 Forms and documents
Several documents must be prepared before the evaluation can be carried out. Below
you will find a short description of these documents with an exhaustive document
collection for most of these items. Their design will depend on the chosen evaluation
process.
Guidelines for evaluators
These documents provide evaluators and experts with the necessary information
about the evaluation (process, criteria, thresholds, dates and deadlines) to perform
their task properly. If scoring of criteria is involved, the guidelines must include
precise instructions for allocating scores.
>> Download Template 1
>> Download Template 2
Non-disclosure agreement (NDA)
All information contained in a proposal is confidential. Usually, employees of funding
organisations have already signed a NDA, but if external experts/evaluators are
involved, it is recommended to supply a form they can sign.
The evaluators have to confirm that there is no "conflict of interest". A clear definition
of such a conflict is required.
>> Download Template 1
>> Download Template 2
>> Download Template 3
Checklists of evaluation criteria
This list of criteria has to be mutually agreed among all agencies of the ERA-NET and
will be applied at the ERA-NET level.
Larger domains of criteria may be broken down into detailed sub-criteria.
For transparency, this list is also published in the guidelines for applicants.
>> Download Template 1
>> Download Template 2
>> Download Template 3
Individual Evaluation reporting form
Evaluation reporting forms are designed for collecting results from the evaluators;
ideally, they are provided in a form that makes it easy to create a result table.
Evaluation results (scores, yes/no decision) will be accompanied by detailed
comments explaining the evaluator's conclusion.
Page 92 of 154
Deadlines for returning the complete results shall be highlighted on the form.
>> Download evaluation reporting form
Proposal evaluation summary
This summary report compiles the external and national evaluations of the proposal,
together with the management group recommendations.
>> Download Project Proposal Evaluation Summary
Summary of the project
This summary compiles all information of the project needed for the evaluation
meeting.
>> Download Template
Consensus evaluation statement
Based on the individual evaluation reports, the evaluation panel/ scientific committee
discusses about the proposal, prepare of a consensus evaluation statement and give
the final score as well as ranking (if needed).
Final evaluation report
The final outcome of the evaluation, i.e. a list of proposals recommended for funding
(ranked, grouped or both), is accompanied by the final evaluation report, which lists
the common decisions and justifications.
>> Download Template
Page 93 of 154
5. Funding decisions
5.1 Introduction
5.2 Funding decisions
5.3 Challenges of contract preparations
5.4 Distribution and monitoring of funds
Contents Page
5.1 Introduction
From the programme managing point of view, the project initiation phase is the step
following the evaluation.
The funding decisions, concluding the contracts and the installments of funds will
eventually lead to launch the projects.
An effective operation will therefore be a marker for the quality of the respective ERANET (as perceived by applicants).
5.2 Funding decisions
National/regional decisions generally follow internal regulations except for cases in
which the ERA-NET has commissioned this task to a Call Steering Committee (CSC) or
comparable constitution (usually if a real common pot model is applied). The fact that
cross border collaboration between nationally funded projects is facilitated represents
the most important deviation from this internal regulation. A coordinated project start
within a common time frame is crucial to allow effective trans-national collaboration.
The national/regional funding decision of each program owner is the required input for
ERA-NETs applying the virtual common pot model. Usually, each national/regional
program / research council has an internal board of decision makers in charge to
officially approve the funding of projects recommended by evaluators.
Decisions should follow in strict consent with the results of the joint evaluation and
the jointly made recommendation of projects. An excellent communication within the
ERA-NET consortium, an open use of information about national decisions and a
timely finalization of the process are necessary to control the activities and to start
Page 94 of 154
the projects and their research activities in an appropriate and simultaneous manner.
For ERA-NETs applying a real common pot or mixed mode model operated by a
central call secretariat the required input of national/regional program owners is the
formal transfer of decision making to the Call Steering Committee of the ERA-NET,
and assignment of freedom to operate the following process to the call secretariat. In
this case the decision making is the result of the panel meeting and no further
coordination of decisions is necessary.
See also PLATFORM recommendations "Best practices and recommendations for
effective and cost-efficient Call management"
In ERA-NET Plus and co-funded ERA-NET Cofund calls, the most frequently
applied financial model is the mixed mode.
This funding mode helps to ensure that selection decisions can strictly follow the
ranking list of the evaluated proposals and that the participating funding
countries/beneficiaries maintain their initial commitments. The final decision on which
projects are to be funded in an ERA-NET Plus/co-funded ERA-NET Cofund call
determines the amount of budgets to be spent by the funders, and that influences
also the EC contribution. Beneficiaries of these co-funded calls (funding organizations)
need to consider that they are required to pre-finance the final payment from their
national/ regional budgets prior to receiving the final payment from the EC.
The final EC contribution is open to variations and can, in principle, only be
determined at the end of the ERA-NET Plus or ERA-NET Cofund activity as the costs
claimed by and granted to the projects can vary during their running time.
Page 95 of 154
5.3 Challenges of contract preparations
In the case of a virtual common pot, separate contract agreements are concluded
between the involved funding organisations and respective applicants.
Each funding organisation should therefore see for an efficient and timely preparation
of agreements (e.g., MoU, Consortium Agreement). In order to reach a maximum of
harmonisation between the different national funding rules and procedures, the
agencies should achieve agreement on the following points – in consistence with their
applicable national/regional provisions preferably before the process starts: (i) agreed
(usually identical) duration for all participants of a project, (ii) requirements for
additional joint reporting, (iii) rules for IPR distribution and the exploitation of results
- usually established in a Consortium Agreement between the project participants,
and (iv) the necessity of a Consortium Agreement for contracting.
In the case of a Real Common Pot (or the joint funds of a Mixed Mode), the contracts
for individual project participants are exclusively based on jointly agreed provisions of
the ERA-NET, and the contracts are concluded accordingly by the secretariat.
You will find more information on how to draft a CA and how to secure IPR as part of
the IPR information (http://netwatch.jrc.ec.europa.eu/web/lp/learningplatform/toolbox/intellectual-property-rights/ipr-and-ca) in the ERA-LEARN toolbox.
See also PLATFORM recommendations "Best practices and recommendations for
effective and cost-efficient Call management"
Page 96 of 154
5.4 Distribution and monitoring of funds
In the virtual common pot model funds are usually provided directly by the respective
national/regional agency to successful applicants.
Consequently, monitoring of funds, too, is effected by the agencies in this direct interrelationship. For trans-national project cooperation it is important that all involved
project participants will receive their funds at project start, as the delay in funding
one participant might cause the delay of the project and thus involves the other
participant. To assure a quick release of funds is part of the agencies' commitment to
the ERA-NET.
In the case of a Real Common Pot the programme owners provide the agreed
financial contribution to the central secretariat. Funds are then distributed to the
respective projects and the project is monitored by the secretariat. In the case of a
mixed mode, program managers provide the financial contribution directly to the
successful project participants from their respective countries - plus the maximum
funding contribution of joint funds agreed within the ERA-NET consortium to the call
secretariat.
Joint funds are subject to monitoring by the call secretariat, which surveys the proper
use of joint funds in projects that have been assigned joint funds. In the mixed mode
model, the call secretariat could also host the entire monitoring process. In addition
to joint funds, agencies have to provide a joint budget that covers the cost for the
secretariat.
Specific payment modalities of ERA-NET Plus involve two pre-financing phases, in
some cases interim payments and a final payment.
The ERA-NET Cofund involves two pre-financing phases: 1) the 1st pre-financing (10
% of the EC contribution) after signing the Grant Agreement and 2) the 2nd
prefinancing (80 % of the EC contribution) after the 1st reporting period just after the
project evaluation and selection process.
Therefore in EC co-funded calls, the preparation of an overview of
payments/reimbursement to the beneficiaries by the coordinator is advisable.
Page 97 of 154
the
1. Filling of gaps first regardless of national return/fair share, followed then by a
proportional redistribution of the remaining EC contribution among the funding
agencies. This means that the EC contribution will be given primarily to those funding
agencies that have run out of national resources to cover their remaining national
contributions requested in the projects recommended
To better fill the "gaps" only a certain fixed percentage of the EC contribution flows
into a joint "balancing pot" – any remaining EC contribution will be shared
proportionally among the funding agencies as under.
Download an example: "Distribution of funds in ERA-NET Plus"
2. The EC contribution will be distributed proportionally among the funding agencies
based on the respective national/regional contributions. This leads to fair national
return regardless of the "gaps", however, this usually results also in the fact that less
projects can be funded and/or that additional efforts have to be made in order to
adjust the project budgets during the contract negotiation phase.
Page 98 of 154
6. After the call
6.1 Monitoring of funded projects
6.1.1 Report templates
6.1.1.1 Template A1
6.1.1.2 Template A2
6.1.1.3 Template B1
6.1.1.4 Template B2
6.2 Monitoring of call implementation
6.3 Call results
6.4 Dissemination of call results
6.5 Analyzing impact of joint calls
6.5.1 Analysing the impact of the joint call: Example 1
6.5.2 Analysing the impact of the joint call: Example 2
6.5.3 Analysing the impact of the joint call: Example 3
Contents Page
6.1 Monitoring of funded projects
Designing and implementing a system for common reporting and monitoring
of the individual projects financed by joint calls may be a challenging task.
Each participating funding agency will have their specific reporting and
monitoring requirements. In order to gather information on the progress and
the final outcome of the trans-national/trans regional project as a whole,
there is a need for common reporting from the project in addition to eventual
national reporting from each project participant to its national funding
agency.
The common reporting and the national reporting should be coordinated as
much as possible to the benefit for the project participants.
This can be achieved by collecting indicators at ERA-NET level and comparing
these indicators with information that is already collected routinely at the
national/regional levels.
Page 99 of 154
In the last years an impressive variety of successful ERA-NET call procedures have
been established. However, only a few ERA-NETs have developed systems for
common reporting from the projects financed from the joint calls.
These guidelines are intended to be a help for ERA-NETs in setting up a system for
common reporting of the projects. In the preparation of the guidelines, we have
gathered information from ERA-NET coordinators on the reporting systems already in
use. Lists of indicators and templates for periodical and final reporting from projects
financed by joint calls are found in Section 6.1.1. These templates can be used by
ERA-NETs, or adapted to fit your specific needs.
A system of common reporting must be seen in relation to what is required as
national reporting by the different funding agencies participating in the joint call.
Double reporting and increase of bureaucratic burdens for the project participants and
the funding agencies should be avoided as much as possible. The funding mode
chosen for the joint call will have an important impact on how the reporting and
monitoring system for the call can be made.
If a common reporting system comes in addition to national reporting, it is important
that the common reporting and monitoring focuses in particular on the added value of
the cooperation in the project, and with a strong emphasis on the total outcome if the
national reporting is focusing on each project participant´s work. Indicators of this
added value of cooperation may vary according to what kind of research is supported.
The importance of this added value of cooperation is reflected in the example
templates in the toolbox within 6.1.1. ERA-NETs may want to modify this to their
specific needs by using its own indicators.
Most of the joint calls in ERA-NETs have so far been funded according to the principles
of virtual common pot (s1.1.5). As all project participants in the financed projects will
have separate contracts with their national or regional funding agencies, progress
reports and final reports from each project participant to their funding agency will be
a part of the obligations in the contracts. These separate reports will often cover only
parts of the total project, often related to the specific tasks that each project
participant is doing in the project. In order to monitor the progress and final outcome
of the project as a whole, there is a need for systems of common reporting also for
joint calls with distributed pots/virtual common pots. Some ERA-NETs have such
common reporting in place, in addition to the national reporting from each project
participant to their respective funding agency.
For joint calls where there is only one contract – between the project coordinator and
the ERA-NET secretariat or call coordinator – the reporting will be common for the
whole project. This will be the case for joint calls with real common pot (s1.1.5), but
also calls with mixed mode (s1.1.5) can have such a system. That means that the
participating funding agencies will leave the task of collecting reports to a call
secretariat acting on their behalf.
Page 100 of 154
Common reporting and monitoring should take into account the requirements in the
participating funding agencies. If a common reporting and monitoring system can
include all requirements of all participating funding agencies, in principle a common
system could replace the national systems. The information requested will in most
cases be very similar from all participating funding agencies, and possibilities of
making one single reporting system that covers the needs of all the participating
funding agencies should be explored. If such a total integration is not possible, parts
of the common reports, such as Publishable Abstract and Publishable Summary, can
be used by each project participant in his/her national reporting. In order to make the
reporting easier for the project participants, an annex to the reporting template
explaining the reporting requirements of each can be prepared by the ERA-NET.
Harmonisation of reporting and monitoring requirements should not be restricted to
ERA-NETs and national systems. Many of the participants in projects financed by joint
calls in ERA-NETs will also be participants in FP7 projects, or will be participating in
such projects at a later stage. There may be good reasons to work for harmonisation
also with the FP7 reporting and monitoring system. The reporting and monitoring
requirements accepted by the Commission should be sufficient for most of the
participating funding agencies. Harmonization with FP7 requirement will also ensure
easier access to comparable statistics in ERA. Finally – when an ERA-NET evolves into
ERA-NET+ or other kind of joint programming including Commission R&D funding the FP7 requirements will have to be fulfilled. It is consequently rational to establish
common reporting systems which at a minimum meet FP7 reporting requirements.
Common reporting of jointly funded projects will be of benefit for all the
participating funding agencies in an ERA-NET. Common reporting will give
documentation of the outcome of successful projects and demonstrate that
participation in joint calls will give good value for money.
Setting up a system for common monitoring and reporting has also a cost
aspect. One of the agencies in the ERA-NET will collect and analyse the
reports on behalf of the whole ERA-NET. As long as the ERA-NET is a project
with funding from the Framework Programme, the cost of common reporting
should be included in the cost of running the ERA-NET. The coordinator of the
ERA-NET, or a joint secretariat or call coordinator should have a budget that
cover these costs.
In many cases, the projects financed by joint calls in an ERA-NET will run
after the funding of the ERA-NET project has ended. Provisions for securing a
budget for the secretariat functions after the ending of the EC-funded ERANET project should be included in a Memorandum of Understanding (MoU) or
other legal text, where the ERA-NET consortium agrees on how these costs
shall be paid for.
Page 101 of 154
6.1.1 Report templates
The following reporting templates can be used for common reporting of transnational
projects financed from joint calls. We are describing 2 different scenarios.
A) National reporting between each project participant and the
respective funding agency
The two first templates are intended for joint calls where there is national reporting
between each project participant and the respective funding agency. There is one
template for periodic reporting (template A1) and one for final reporting (template
A2). These can be seen as a minimum for information that can be collected jointly for
the project where full reporting is taking place nationally.
B) Reporting is centralised between the project coordinators and a call
coordinator or call secretariat
The other two templates are intended for joint calls where all the reporting is
centralised between the project coordinators and a call coordinator or call secretariat.
The templates for periodic reporting (template B1) and for final reporting (template
B2) are more detailed than template A1 and A2.
Parts of these two more detailed templates – such as list of milestones and
deliverables and even the financial part of the reporting - can also be used by ERAnets with national reporting.
6.1.1.1 Template A1: Reporting template – optional periodic
report
For use in ERA-nets with separate contracts between each project participant
and their funding agency, and where there is reporting between each project
participant and their funding agency.
1. Identification of project and participants
PROJECT FULL TITLE
Project acronym:
Project number:
Periodic report: Number
Page 102 of 154
Period covered:
from:
to:
(Reporting period: The ERA net has to decide the reporting periods for the
funded projects. For 3-year projects, we suggest a first report after 18
months and the final report at the end of the project)
Duration of project:
Project coordinator:
Name, title and organization of the representative of the project's
coordinator:
Tel:
Fax:
E-mail:
ERA-NET website address:
Identification of project participants / beneficiaries (includes PIC
code + official national registration number)
(1) < project coordinator >
(2) < name, organisation, country/region >
(3) < name, organisation, country/region >
(4) < name, organisation, country/region >
(5) < name, organisation, country/region >
(6) < name, organisation, country/region >
2. Short description of activities and intermediate results
Describe the activities and intermediate results. Special emphasis should be
given to the collaboration between the partners, and the result of the
collaboration.
(The ERA-net can define the maximum number of words)
Page 103 of 154
3. Problems/challenges/deviations from proposal/work plan
List and comment problems, challenges and deviations pertinent to progress
in the project. Explain any deviations from proposal/work plan and impact
on other tasks, as well as on available resources
Describe corrective actions adopted or proposed for deviations from tasks
Please also use this section to summarize any changes you propose to your
project, compared to the original proposal/work plan)
(The ERA-net can define the maximum number of words)
4. Dissemination activities in the period in question (including list of
publications where applicable)
Optional annex: National reporting requirements
(The ERA-net can fill in the information on national reporting requirements
for each country participating in the joint call, as information for the project
participants)
Country
Translation
into national
language?
Separate
national
activity
report (or
summary)?
Specific
national
requirements
for data and
outputs
Page 104 of 154
Where to find
information on
financial
reporting
requirements
(web-page,...)
Intermediate
report prerequisite for
releasing
funding?
6.1.1.2 Template A2: Reporting template – final report
For use in ERA-nets with separate contracts between each project participant and
their funding agency, and where there is reporting between each project participant
and their funding agency.
1. Identification of project and participants
PROJECT FULL TITLE
Project acronym:
Project number:
Period covered: from: to: (the full project period)
Duration:
Project coordinator:
Name, title and organization of the representative of the project's
coordinator:
Tel:
Fax:
E-mail:
ERA-NET website address:
Identification of project participants/beneficiaries (includes PIC
code + official national registration number)
1) < project coordinator
(2) < name, organisation, country/region >
(3) < name, organisation, country/region >
(4) < name, organisation, country/region >
(5) < name, organisation, country/region >
(6) < name, organisation, country/region >
Page 105 of 154
2. Final publishable summary report
Final publishable summary report of the key findings and impacts written for
a non-technical audience, which "the ERA-net" may use to communicate
results to user. It should also be possible to use in national reporting (The
ERA-net should define the number of pages and font size)
3. Description of activities and final results
List major objectives of the project. Describe briefly whether the objectives
of the research have been achieved and outline the principal outcomes of
the work and their significance to the field. (The ERA-net should define the
number of pages (3-10?) and font size)
4. General description of the cooperation over the duration of the project
Factual description, specifying the input of each participant. Describe the
added value of doing the work in a transnational project, compared to the
description in the project proposal
(The ERA-net should define the number of pages (1-2?) and font size)
5. Impact statement
Please give a short description of impacts resulting from work.
Work force statistics,
Gender Aspects,
Synergies with transnational partners,
Interdisciplinarity
6. Exploitation and dissemination measures
Patents, new cooperations, new products, employees, turnover
7. Explanation of the use of resources (final financial report)
List any deviations in participant's use of resources pertinent to the project
as a whole,
Describe corrective actions adopted for any deviations.
Page 106 of 154
Optional annex: National reporting requirements
(The ERA-net can fill in the information on national reporting requirements
for each country participating in the joint call, as information for the project
participants)
6.1.1.3 Template B1: Reporting template – periodic report
For use in ERA-nets with centralised reporting between project coordinator and ERAnet call coordinator/secretariat. Some elements of this template may also be included
in template B2.
1. Identification of project and participants
ERA-NET Grant Agreement No:(prefilled by ERA-NET)
ERA-NET acronym:(prefilled by ERA-NET)
PROJECT FULL TITLE
Project acronym:
Project number:
Periodic report: Number
Period covered:
from:
to:
(Reporting period: The ERA net has to decide the reporting periods for the
funded projects. For 3-year projects, we suggest a first report after 18
months and the final report at the end of the project)
Date of submission:
Project coordinator:
Name, title and organization of the representative of the project's
coordinator:
Tel:
Fax:
Page 107 of 154
E-mail:
ERA-NET website address:
Identification of project participants/beneficiaries (includes PIC
code + official national registration number)
(1) < project coordinator >
(2) < name, organisation, country/region >
(3) < name, organisation, country/region >
(4) < name, organisation, country/region >
(5) < name, organisation, country/region >
2. Publishable summary
Short description of activities and significant results
This description should be written for a non-technical audience (The ERA-net
can define the maximum number of words)
3. Work progress and achievements during the period
Project objectives for the period:
Work progress and achievement by work package
Work package 1:
1. A summary of progress towards objectives for each task
2. Significant results
Work package 2: (etc)
4. Milestones and deliverables
Milestones (The milestones here are examples)
Page 108 of 154
Partner
responsible
Milestone
Date
(dd/mm
/yyyy)
Progress
Comments
Project start-up
Project start-up meeting
Project Kick off meeting
1 st Status report (18 months)
...
...
Project finalization meeting
Final report
Project end date
Deliverables
Deliverable name
Partner responsible Date
(dd/mm/yyyy)
Progress
Comments
Please indicate whether the planned deliverables and milestones for the
period, as described in the proposal, have been completed, delayed or
readjusted (Progress column).
Page 109 of 154
5. Deviations from proposal/work plan
List and comment deviations pertinent to progress not covered in the tables
above. Explain any deviations from proposal/work plan and impact on other
tasks, as well as on available resources
Describe corrective actions adopted or proposed for deviations from tasks
Please also use this section to summarize any changes you propose to your
project, compared to the original proposal/work plan)
6. Dissemination activities in the period in question (including list of
publications where applicable)
7. Project management
Summary of management of the project
Comments and information on co-ordination activities during the period in
question, such as communication between project participants, cooperation
with other projects in the ERA-NET etc.
8. Financial report
Please provide an explanation of personnel costs, subcontracting and any
major direct costs incurred by each beneficiary, such as the purchase of
important equipment, travel costs, large consumable items, etc. linking
them to work packages.
There is no standard definition of "major direct cost items". Beneficiaries
may specify these, according to the relative importance of the item
compared to the total budget of the beneficiary, or as regards the individual
value of the item.
These can be listed in the following tables (one table by participant):
Page 110 of 154
Table 1: Personnel, subcontracting and other major Direct cost items for Beneficiary
1 for the period
Work
Package
Item description
Amount
Explanations
Salaries of 2 postdoctoral students
and one lab technician for 18
months each*
Ex: 2, 5, 8,
11, 17
Personnel costs
235000 €*
5
Subcontracting
11000 €*
Maintenance of the web site and
printing of brochure*
8, 17
Major cost item 'X'
75000 €*
NMR spectrometer*
11
Major cost item 'Y'
27000€*
Expensive chemicals xyz for
experiment abc*
Remaining direct costs
15000€*
TOTAL DIRECT COSTS
363000€*
The entries in italics are examples and purely for illustration
Table 2: Personnel, subcontracting and other major Direct cost items for
Beneficiary 2 for the period
Work Package
Item description
TOTAL DIRECT COSTS
Page 111 of 154
Amount
Explanations
Financial statements – summary financial report
Please submit a separate financial statement from each beneficiary together with a
summary financial report which consolidates the statements of all the participants in
an aggregate form, based on the information provided each participant.
6.1.1.4 Template B2: Reporting template – Final report
For use in ERA-nets with one single contract between project coordinator and ERA-net
call coordinator/secretariat. Some elements of this template may also be included in
template B1.
1. Identification of project and participants
ERA-NET Grant Agreement No:(prefilled by ERA-NET)
ERA-NET acronym:(prefilled by ERA-NET)
PROJECT FULL TITLE
Project acronym:
Project number:
Period covered:
from:
to:
(the full project period)
(Reporting period: The ERA net has to decide the reporting periods for the
funded projects. For 3-year projects, we suggest a first report after 18
months and the final report at the end of the project)
Date of submission:
Project coordinator:
Name, title and organization of the representative of the project's
coordinator:
Tel:
Fax:
Page 112 of 154
E-mail:
ERA-NET website address:
Identification of project participants/beneficiaries (includes PIC
code + official national registration number)
(1) < project coordinator >
(2) < name, organisation, country/region >
(3) < name, organisation, country/region >
(4) < name, organisation, country/region >
(5) < name, organisation, country/region >
2. Final publishable summary report
Final publishable summary report of the key findings and impacts written for
a non-technical audience, which "the ERA-net" may use to communicate
results to user. (The ERA-net can define the maximum number of words)
3. Detailed description of activities and final results
List major objectives of the project. Describe briefly whether the objectives
of the research have been achieved and outline the principal outcomes of
the work and their significance to the field. (The ERA-net should define the
number of pages (3-10?) and font size).
4. Milestones and deliverables
Please report on milestones and deliverables. Explain any changes,
difficulties encountered and solutions adopted (Comments column).
Milestones (The milestones here are examples)
Page 113 of 154
Partner
responsible
Activity
Date
achieved
Comments
Project start-up
Project start-up meeting
Project Kick off meeting
Status report (18 months)
...
...
...
Project finalization meeting
Final report
Project end date
Table of deliverables
Deliverable name
Partner responsible
Page 114 of 154
Date delivered
Comments
5. General description of the cooperation over the duration of the project
Factual description, specifying the input of each participant. Describe the
added value of doing the work in a transnational project.
(The ERA-net should define the number of pages (1-2?) and font size) .
6. Impact statement
Please give a short description of impacts resulting from work.
•
•
•
•
•
•
Ethics
Work force statistics
Gender Aspects
Synergies with Science Education
Interdisciplinarity
Engaging with Civil society and policy makers
7. Use and dissemination
publications (if relevant)
of
foreground
(including
list
of
scientific
Dissemination measures
List of publications derived from this project, including those in
preparation, in review and in press
Dissemination of results and knowledge transfer (Participation in
scientific events; posters and presentations)
Significant external interactions in the project, Technology transfer,
List of achieved degrees / patents / other outcomes in the project
Describe to what degree these results have been achieved as a result of
cooperation between the partners in the project
Follow up activities and plans for further exploitation of the results
What sort of follow-up activities should take place to ensure that the results
of this project are applied to the fullest extent possible?
Media and Communication to the general public
Page 115 of 154
8. Final financial report
Please provide an explanation of personnel costs, subcontracting and any
major direct costs incurred by each beneficiary, such as the purchase of
important equipment, travel costs, large consumable items, etc. linking
them to work packages.
There is no standard definition of "major direct cost items". Beneficiaries
may specify these, according to the relative importance of the item
compared to the total budget of the beneficiary, or as regards the individual
value of the item.
These can be listed in the following tables (one table by participant):
Table 1: Personnel, subcontracting and other major Direct cost items for
Beneficiary 1 for the period
Work
Package
Item description
Amount
Explanations
Salaries of 2 postdoctoral
students and one lab
technician for 18 months
each*
Ex: 2, 5, 8,
11, 17
Personnel costs
235000 €*
5
Subcontracting
11000 €*
Maintenance of the web site
and printing of brochure*
8, 17
Major cost item 'X'
75000 €*
NMR spectrometer*
11
Major cost item 'Y'
27000€*
Expensive chemicals xyz for
experiment abc*
Remaining direct costs
15000€*
TOTAL DIRECT COSTS
363000€*
* The entries in italics are examples and purely for illustration
Page 116 of 154
Table 2: Personnel, subcontracting and other major Direct cost items for
Beneficiary 2 for the period
Work Package
Item description
Amount
Explanations
Personnel costs
Subcontracting
Major cost item 'X'
Major cost item 'Y'
Remaining direct costs
TOTAL DIRECT COSTS
Financial statements – final financial report
Please submit a separate financial statement from each beneficiary together with a
summary financial report which consolidates the statements of all the participants in
an aggregate form, based on the information provided each participant.
6.2 Monitoring of call implementation
The execution of joint calls and the evaluation and selection of the R&D projects will
be followed by joint monitoring of procedures (call mechanics) in order to enable an
acceleration of internal processes and efficient use of resources during future calls.
This can include an internal as well as an external view on the quality of call
management and include all phases of a joint call, including preparation and
submission of proposals, feedback from the agencies and preparation of contracts.
Page 117 of 154
In addition, the joint monitoring of call procedures should include the analysis of
benefits compared to alternative funding instruments.
Opinions within agencies can be collected directly from the ERA-NET consortium,
whereas feedback from the target group can be collected via questionnaires (example
provided online).
In Horizon2020, the ERA-NET Cofund consortium is requested to provide information
about each project selected for funding to the EC after the end of evaluation (of the
co-funded call). This includes data on each participant (principal investigator) and
abstracts of the project proposals. In addition, the EC is provided with a report from
an independent observer, which has been nominated by the consortium (of funders)
to observe the whole call and evaluation process.
The efficiency of call implementation can be tested by a comparison with other ERANETs´calls in the Cost-Benefit Analysis of Calls
(http://netwatch.jrc.ec.europa.eu/web/lp/learning-platform/toolbox/cost-benefitanalysis-of-calls) section of the ERA-LEARN toolbox.
Page 118 of 154
6.3 Call results
At the end of a call it will be a standard task to perform a statistical analysis of
results.
Investigations will include:
•
•
•
•
•
•
•
•
•
•
•
•
Number of submitted proposals (stage 1 & 2)
Number of applicants involved
Number of funded projects and overall success rate
Total costs and requested (or accepted) funding
Number of proposals (projects) per funding programme, per region, per
country
Number of project coordinators per funding programme, per region, per
country
Average number of applicants per project consortium
Average number of regions/countries per project consortium
Response to thematic sub-topics
Composition of consortia in terms of organisation type (e.g. SME, IND, RES,
HE)
Requested (or accepted) funding per participating programme, per region,
per country
Transnational networking of applicants –who cooperates with whom?
After the evaluation of the co-funded call projetcs, the ERA-NET Cofund
consortium must submit to the EC the following documents:
1. The ranking list(s) of the projects
2. The observers' report on the evaluation
3. The joint selection of the projects to be funded
4. From each consortium partner (funding organization) participating in the cofunded call, a formal and duly signed commitment on availability of funds for
the selected projects
5. Information on each project selected for funding, including data on each
participant (principal investigator) as well as abstracts of the project proposals.
Later, at the end of the funding period of the projects, EC is still informed about the
results of the projects (overview of partners with their main results).
Page 119 of 154
6.4 Dissemination of call results
Disseminating the results of a transnational call will highlight the main parameters to
potential future applicants, thus supporting the promotion of future calls. In addition,
results may also be helpful when trying to enhance the commitment from
programme owners. It may be useful to compare the results to effects of other
instruments in the field.
Apart from statistics explaining the interest from the RTD community as well as the
budget invested, the publication of successful projects ("success stories") (to a
certain degree of detail) will increase the transparency of the ERA-NET. Success
stories may also be a valuable illustration of the call scope especially to newcomers
(s2.2.2.4).
Dissemination channels could be the similar to the ones for promoting a call. (2.2.2)
6.5 Analysing impact of joint calls
A final step in assessing the joint call is the analysis of generated impact. This
stretches beyond monitoring of funded projects and is also different from analysisng
the mechanics of call implementation.
The basis for all impact assessments should be the defined goals for the joint call or
the programme. The type of research funded by the ERA-NET will have bearings on
what the expected impact of the funded research should be. ERA-NETs funding
academic research should have different impacts than ERA-NETs funding industrial
R&D for the benefit of SMEs.
For some programmes of industrial research it may be most relevant to ask the
enterprises involved about the revenues generated through their innovative
product/service resulting from funded projects, whereas in other programmes – in
both basic and applied research – it may be relevant to search for scientific gains but
also for societal impact in a broader sense.
Some ERA-NETs are specifically designed for providing a basis for policy making in a
specific field – in such cases the impact must be measured as to which degree
knowledge developed in the supported project is taken up and implemented in policy
making, or ultimately, to which degree this knowledge has led to solving the problem
addressed by the specific policy.
Page 120 of 154
There is agreement among all major agencies that have undergone this task that the
impact of project outcomes even at the national level is difficult to measure and even
less, in quantitative figures. Whatever quantitative measures are applied, i.e. (i)
publications/impact factors in basic science, (ii) patents/market revenue/socioeconomic factors in applied science, and (iii) the only recently emerged issue of
dissemination of results /creation of public awareness (in all sectors), these lack
sound, commonly agreed and undisputed indicators. This puts transnational impact
assessment of outcomes of ERA-NET calls/programmes on a rather poor basis.
Impact assessment of the joint calls should focus on added value when compared to
national funding. The ERA-NET should develop indicators reflecting the goals of the
joint call, decide how the data on these indicators should be gathered and by whom,
and set target figures where appropriate.
The toolbox provides examples of objectives and indicators that can be used as a
point of reference.
• An example of a fully developed system, with objectives and quantitative
targets is found in section 6.5.1. This system could be used as a starting
point for other ERA-NETs, especially ERA-NETs funding applied, industrial
research.
• An according example for ERA-NETs with a focus on basic research is found in
Section 6.5.2, since basic science is (traditionally) well-grounded in
international cooperation, there is less focus on the aspect of
internationalisation.
• For ERA-NETs directly linked to policy making it is important to include
indicators on relevance and use of the knowledge developed in projects
funded by the ERA-NET. An example of such a system is available in section
6.5.3.
In any case, impact assessment of projects goes beyond the usual monitoring process
during the execution of an individual project and could thus extend beyond the life
time of an ERA-NET, too. It involves both a time and a cost component, and
additionally it involves a direct approach channel to the project participants, whereby
the number of questionnaires and reports from the project participants should be held
at a minimum.
1. Time component: Usually, the projects do not turn out direct results that can be
immediately measured in terms of impact. Therefore, the project impact assessment
will only start operatively when first projects resulting from an ERA-NET call are
finished or even later. However, a timely set-up of ex-ante-indicators (i.e. indicators
that can be clearly quantified and that are (indirect) project goals and thus form part
Page 121 of 154
of the funding decision and may be stipulated in the funding contract) is strongly
recommended already for the call planning phase, because these can only be
assessed properly if they have been involved and formed part of the evaluation
process. Contrary, ex-post-indicators (i.e. indicators that are set up after the
evaluation phase and without direct project context) will allow for a later set-up, but
lack qualitative stringency and quantitative robustness.
2. Cost component: As for common project reporting in joint calls, impact
assessment (if jointly done by the call secretariat on behalf of all the agencies in the
ERA-NET) will have to be paid for beyond the ERA-NET Grant Agreement. Financing of
this long-term task has to be considered by the ERA-NET and may be stated in a
Memorandum of Understanding.
3. Channels for collecting data: In planning which channels to use for collecting
data, effort should be made to minimise workload for all parties involved. Possibilities
of integrating questions for project impact assessments and the analysis for
optimising call procedures (see 6.1.2) into joint reporting forms should be explored. If
it is necessary to develop specific questionnaires, these should be easy to answer
(possibly with Y/N answers and/or using pull down answers). There are several tools
available for online surveys. Online tools can make it easier to reply to the survey,
and not least make the processing of the collected data much easier than analysing
data from questionnaires in text or spreadsheet format. As such tools are often used
routinely in funding agencies for collecting feedback, these may be made available
through one of the agencies of the ERA-NET. Suitable tools may also be found on the
web by searching "online survey tool" (http://www.surveymonkey.com/).
6.5.1 Analysing the impact of joint calls: Example 1
This example is taken from ETB-PRO, an ERA-NET in the field of modern
biotechnology (all sectors) targeted towards SMEs and their strategic partners (ROs).
The objectives are taken from the objectives of the DoW and include
(i) to increase trans-national cooperation of SMEs,
(ii) to demonstrate success of trans-national cooperation projects and
(iii) to pinpoint the impact of trans-national cooperation on the national / regional
economies.
Page 122 of 154
MONITORING PHASE: FINAL REPORTING (project impact assessment)
The outcome of the
project was a new
or improved
product (YES/NO),
technology
(YES/NO), service
(YES/NO), process
(YES/NO) or none
(NO)? ("None"
means that the
project goals were
not achieved
Will you
commercialize
the outcome as
a new product,
technology,
service or
process
Approximate
time span
when your
organization
will launch
the new
product,
technology,
service or
process on
the market or
exploit the
results in
another way
Did you
or will
you apply
for IPRs
for any of
the
project
results
The
project will
have/has
an effect
(an
increase)
on the
number of
employees
of the
organizatio
n
The project
will increase
/ has
increased
the annual
turnover of
the
organization
by
The ETB
call was
the first
choice for
submitting
your
project
proposal
International
co-operation
was essential
for this project
(would you
have found a
partner in your
own country)
Other
remarks
Final amount
of received
grants and
loans / partner
grants
drop-down menu
drop-down
menu
drop-down
menu
yes or no
drop-down drop-down
menu
menu
Page 123 of 154
yes or no
yes or no
loans
• The purpose of the first column "The outcome of the project was a new or
improved product (YES/NO), technology (YES/NO), service (YES/NO), process
(YES/NO) or none (NO)? ("None" means that the project goals were not
achieved" is to define the concrete outcome of the project or whether the
project failed (technically). It should be noted that the partners of a single
project could have different objectives in this respect. Eg. one company can
develop special technology whereas another partner can be the final producer
of a new product based on this technology.
• The next column "Will you commercialize the outcome as a new product,
technology, service or process" indicates whether the project has been or is
deemed to be
commercially successful
or not, or if further
studies/development
work
should
be
done
before
making
any
commercialisation decisions.
• In order to get a rough estimate when the project outcome would be on the
market/exploited the 3rd column "Approximate time span when your
organization will launch the new product, technology, service or process on
the market or exploit the results in another way" was installed. The idea is to
get only a rough estimate and therefore only 3 drop-down menu options (< 5
y, 5-10 y and >10 y) were selected. In many cases commercialisation of
biotechnology products is a more challenging and time consuming task than
estimated and more accurate estimations would be less realistic.
• The 4th column "Did you or will you apply for IPRs for any of the project
results" indicates the novelty of the project outcome as well as it´s
commercial potential. The impact of public funding on employment is an
important monitoring parameter at national level in many countries and it is
also relevant monitoring parameter in European context.
• The 5th column" The project will have/has an effect (an increase) on the
number of employees of the organization" will give answers to the socio
economic impact. The focus of ETB funding is on SMEs having a low number
of employees on average and therefore the ready-to-select options in the
drop-down menu are at quite narrow range and low level: 0, 1-3, 3-10 and
>10.
• Another similar important prevalent monitoring parameter is the impact of
public funding on the turnover of the companies. The 6th column "The project
will increase/has increased the annual turnover of the organization" with the
ready-to-select options (0, <1, 1-10, >10 MEUR) gives a rough estimate on
that impact.
• The 7th column "The ETB call was the first choice for submitting your project
proposal" (yes/no) monitors the importance of the ETB funding concept.
There are other European and national funding sources available for
Page 124 of 154
international R&D&I cooperation of SMEs, but they may have different focus
(eg. non-thematic like Eurostars) or quite strict and different set of criteria
(eg. FP7 funding instruments) and therefore may not be that suitable for such
projects which have been funded by ETB.
• The 8th column "International co-operation was essential for this project
(would you have found a partner in your own country)" (yes/no) monitors the
importance of promoting international cooperation in general. The markets for
biotechnology products are almost without exception international as well as
the research in this sector, and therefore international cooperation has been
recognized as a key success factor in biotechnology sector in many instances.
• The 9th column "Other remarks" is reserved for additional comments of the
project partners on these monitoring issues.
• The 10th column "Final amount of received grants and loans / partner"
defines the final amount of the ETB funding granted when the project is really
finished and should match with the figures given by agencies.
6.5.2 Analysing the impact of joint calls: Example 2
Example taken from MNT-ERA.NET. MNT-ERA.NET is focusing on industrial research.
The Objectives are taken from the DoW for this particular ERA-NET. Other ERA- NETs
will have other goals, and can develop a system of indicators, methods of information
gathering and targeted effects related to them.
MNT-ERA.NET call level: Programme impact (call management)
Method
Goals
Indicators
Instrument
Executed by
Frequency
Target / Impact
Continuous MNT ERANET Calls ("MNT-ERA.NET
II will increase cooperation
and synergies between
funding programmes,
research organisations and
industries and secure
durable, long-term
cooperation between key
actors from national and
regional funding systems.")
Number of calls
Central call
announcement
call
secretariat
Annual
3 calls (2009,
2010, 2011)
Size of call, # of
programmes
Budgetary Commitment
of national funding
bodies (" It will improve
Funding volume
(available in
advance or
20-25 programmes
(>90% of MNT
consortium)
Central,
internal exante
Agencies à
call
secretariat
Page 125 of 154
Annual –
before (exante) and
Increase with each
call (target total 50
MEUR)
the coordination and help
pending –
to promote the
afterwards)
convergence of MNT
funding programmes, the
streamlining of programme
procedures, and the more
efficient use of resources,
thereby reducing the
fragmentation of European
MNT funding instruments.")
earmarked +
contracted
funding
Generate transnational
R&D cooperations ("help
create a network of
European R&D players")
Central
statistics
Number of
proposals
after each
call
(contracted,
eg 1 year
after call)
Agencies à
call
secretariat
Annual –
after each
call
Number of
funded/contracte
d projects
Increase with each
call (target total 80
funded projects)
Success rate
>35% of full
proposals
Newcomers to
transnational
RTD projects
Questionnaire
or included in
upload of
proposal
(needs
explanation &
examples)
call
secretariat
Annual –
with each
call
frequent
applicants to
MNT-ERA.NET
calls
Increased and more
rapid uptake of new
micro- and nanotechnologies within
Member states industry
("Innovative joint calls for
collaborative projects,
operated in a variable
geometry approach, will be
widely promoted among
the target groups,
especially high-tech SMEs.
MNT-ERA.NET II will
therefore stimulate the
increased and more rapid
uptake of new micro- and
nano-technologies within
Member states industry.")
Increase with each
call (target total
250 proposals)
Industry
participation:
number of
companies and
% of partners
?
Central
statistics
call
secretariat
Annual after each
call
Industry
participation:
costs EUR and %
of call volume)
SME
participation
>10% of
participants per
call
>50 % of
beneficiaries
% of budget ? à
analyse previous
calls
Central
statistics
call
secretariat
Page 126 of 154
Annual after each
call
>30% of
beneficiaries
% of budget ? à
analyse previous
calls
External view on quality
of call management
Time from
deadline (full
proposal) to
contracting
Agencies à
evaluation tool
Agencies &
call
secretariat
Acceleration of internal
processes (" It will
improve the coordination
and help to promote the
convergence of MNT
funding programmes, the
streamlining of programme
procedures, and the more
efficient use of resources,
thereby reducing the
fragmentation of European
MNT funding instruments."
)
Delivery of
Central,
evaluation
evaluation tool
results according
to plan
Agencies à
call
secretariat
Deadline defined
for each call
Efficient
application
process
Questionnaire
to applicants
Agencies à
national
applicants
Increasingly
satisfied clients
Degree of
innovation
exploit
Questionnaire
monitoring of
projects
Agencies à
call
secretariat
High quality of
transnational R&D
projects ("It will provide
strategic and practical
contributions to enhancing
the competitiveness of the
European micro- and nanomanufacturing industry.")
Annual after each
call
? à project
monitoring
100 days (current
status: 120 days)
? à project
monitoring
Transnational
benefit, new
cooperations
RTD project level: Project impact (outcome)
The questions to the project participants can be integrated with the monitoring of the
funded projects.
Question
Answer
Online
Effect
questionnair
e
1.1 What type of result have you
achieved?
Please select one or more:
1.1.1.
1.1.2.
1.1.3.
1.1.4.
1.1.5.
1.1.6.
multiple
choice
multiple
answers
1.2 How many invention
notifications have been submitted
or patent applications filed as a
result of this project ?
number of inventions
1. Technical Results
Method
Process
Product
Service
Equipment
other
Page 127 of 154
insert number for statistics only -relevance
or 0 or "not
of these criteria is heavily
applicable"
dependent on individual
project
1.3 How many license agreements
have been reached as a result of
this project ?
number of license agreements
insert number for statistics only -relevance
or 0 or "not
of these criteria is heavily
applicable"
dependent on individual
project
1.4 How many publications have
been published as a result of this
project ?
number of publications
insert number for statistics only -relevance
or 0 or "not
of these criteria is heavily
applicable"
dependent on individual
project
1.5 How many degrees (master,
doctoral) have been achieved as a
result of this project ?
number of degrees
insert number for statistics only -relevance
or 0 or "not
of these criteria is heavily
applicable"
dependent on individual
project
1. Scientific results
2. Economic Effects
2.1 Please indicate the increase or volume (EUR)
decrease in R&D expenses (budget)
originating from this project in your
company / research unit:
insert number An increase indicates
or 0 or "not
business and commercial
applicable"
exploitation potential
2.2 Please indicate the increase or
decrease in R&D personnel
originated from this project in your
company / research unit:
volume (persons)
insert number An increase indicates
or 0 or "not
business and commercial
applicable"
exploitation potential
2.3 Please indicate the increase or
decrease in turnover originating
from this project in your company:
volume (EUR)
insert number An increase indicates
or 0 or "not
business and commercial
applicable"
exploitation
3.1.1.
3.1.2.
3.1.3.
3.1.4.
3.1.5.
multiple
choice single
answer
Indicate maturity of results
for utilization and
commercial exploitation
multiple
choice
multiple
answers
Explains possible spill over to
other companies or sectors
multiple
choice
multiple
answers
What is the relevance of
MNT-ERA.NET to realisation
of the co-operation project?
3. Utilisation of results
3.1 What is the time frame for
commercialisation?
Already started
0-2 years
2-5 years
More than 5 years
not applicable
3.2 How will the research results of 3.2.1. for R&D efforts in our
the project be utilised ?
organisation / company.
Please select one or more.
3.2.2. for production and
business operations in our
company.
3.2.3. Other project participants
will utilise the results.
3.2.4. Parties outside the
consortium will utilise the results.
3.2.5. Project results will not be
utilised
3.2.6. The concrete benefits
cannot be assessed yet.
4. Transnational Benefit
4.1 How relevant was MNTERA.NET for the realisation of this
project?
Please select one or more.
4.1.1. The project would have
been carried out on the same
scale and timetable without MNTERA.NET
4.1.2. MNT-ERA.NET has
speeded up the realisation of the
project.
4.1.3. MNT-ERA.NET has
extended the research scope.
4.1.4. MNT-ERA.NET has
supported the definition of more
ambitious objectives
4.1.5. The project would not
have been possible without MNTERA.NET
Page 128 of 154
4.2 Did you experience any
problems in the project in terms of
the following matters ?
(Please use a scale of 1-5: 1=no
problems at all, 5=many problems)
4.2.1. Commitment of partners
to the project
4.2.2. Instability of consortium,
change of partners
4.2.3. Too many participants
4.2.4. Creating concrete cooperation
4.2.5. Partners had different
objectives
4.2.6. Change of objectives
(own, partners) during the
project
4.2.7. Objectives that were too
ambitious and unrealistic
4.2.8. Reliability of partners in
carrying out subtasks
4.2.9. Varying competence levels
among the partners
4.2.10. Cultural differences in
communication and working
methods
4.2.11. Varying technical
solutions and standards
4.2.12. Insufficient financial
resources
4.2.13. Ownership and sharing of
outcome
4.2.14. Rearrangements in our
organisation
4.2.15. A prolonged project and
the problems resulting from it
4.2.16. Delayed funding
multiple
choice
multiple
answers
4.3 Will the project continue ?
Please select one or more.
4.3.1. Yes, we will continue with
a project within this consortium.
4.3.2. Yes, we will continue
separately with an in-house
project.
4.3.3. No, we will not continue
the project.
multiple
choice
multiple
answers
Creation of follow-up
projects?
4.4 Will the cooperation continue ?
Please select one or more.
4.4.1. Yes, we will continue
cooperating on the same topic.
4.4.2. Yes, we will continue
cooperating on a different topic.
4.4.3. No, the cooperation will
not continue.
multiple
choice
multiple
answers
Generatation of transnational
RTD cooperations ?
4.5 Will you apply for funding for a
follow-up project ?
Please select one or more.
4.5.1. Yes, we will apply for a
national project
4.5.2. Yes, we will apply for a
bilateral project (e.g. Nordic
project)
4.5.3. Yes, we will apply for
another ERA-NET project
4.5.4. Yes, we will apply for a EU
Framework Programme project
4.5.5. Yes, we will apply in other
funding schemes.
4.5.6. No, we will not apply for
funding.
4.5.7. Too early to say
multiple
choice
multiple
answers
Prefered funding
instruments?
4.6 What do you consider the main
benefits of the MNT-ERA.NET Call
for this project?
comment
free text
What is the relevance of the
MNT ERA-NET programme /
call to realisation of the cooperation project?
Page 129 of 154
6.5.3 Analysing the impact of joint calls: Example 3
Examples of indicators that can be used for impact assessments and analyses for
optimising call procedures – ERA-nets financing policy-related applied research
For ERA-NETs financing research relevant for policy-making in a specific field, it will
be of particular interest to assess the relevance of the funded research for the
development of the policy. The following indicators are adapted from the ERA-NET
CRUE with a specific field, but can be adapted to other fields:
General:
• Achieved credibility as a valuable advisor to European and/or national policy
& research strategies within this particular field.
• Evidence of uptake & application of jointly funded research outputs in policies
and practices in this particular field on a European and a national level.
• The list of national and regional research agenda's for which the output of the
ERA-NET is explicitly mentioned as a cornerstone.
Dissemination:
• Presence / number of articles in the popular press as well as website hits.
• Attendance at workshops and forums organised by the ERA-NET, and reports
produced.
• The ratio of the research funds available through joint calls in the ERA-NET
compared to the National and Regional research funds (and FP) in this particular
field.
Page 130 of 154
7. Attachments
Printable Check List for Call Implementation
Platform Best Practices and Recommendations
Platform Master Class Report
Access to Commissions Database of Experts for Research Activities
Contents Page
Page 131 of 154
Printable checklist for call implementation:
1
Call planning
Call process & administration:
Call administration
Overall time frame
Submission of proposals
Evaluation of proposals
Funding modes
Financial commitment
Funding decision
Start of funded projects
Monitoring
Administrative conditions and necessary documents:
Timing of the call:
Call schedule
Overlaps with other calls
Fine tuning of call milestones
Plan necessary meetings
NETWATCH call calendar
Scope of the call:
Method for definition of scope
Overlaps of topics with other calls
Identification of target groups
Definition of targeted projects
2
Call preparation
Call documents
Supporting documents
Check coherence of call targets, documents and results
Promotion of the call:
ERA-NET specfic activities
National / regional activities
Promotion via target group
3
Submission
Submission of pre-/full proposals
Submission of national/regional funding application forms
Distribution of submitted proposals to involved funding organisations
4
Evaluation
Result of evaluation process
Procedures
Criteria
Forms
5
Funding decisions
Funding decisions
Challenges of national contract preparation
Distribution and monitoring of funds
Checklist for call implementation. Version Feb 2010.
Page 132 of 154
1
6
After the call: monitoring, dissemination & impact
assessment
Monitoring of funded projects
Monitoring of call implementation
Call results
Dissemination of call results
Analysing impact of joint calls
Checklist for call implementation. Version Feb 2010.
Page 133 of 154
2
Best practices and recommendations for effective and cost‐efficient
call management in bioeconomy related ERA‐NETs
Covering the issues: (A) Electronic systems for submission, evaluation and matchmaking,
(B) Evaluation and ranking, (C) Selection of applications, funding decision and negotiation,
and (D) Project monitoring.
Project deliverable: this is part of D3.1 of the PLATFORM project.
Background. A dedicated event on call management organised by the FP7 project Platform of
Knowledge Based Bio‐Economy relevant ERA‐NETs (PLATFORM) brought together ERA‐NET call
officers, most of them with 2‐8 years of experience in (transnational) call management. Prior to the
event a survey among bioeconomy relevant ERA‐NETs had been conducted to get a picture of the
differences and similarities in the way the ERA‐NETs work, and help to identify areas to discuss. In
this master class, organised by the WP3 of PLATFORM on June 17‐18, 2013 in Brussels, practices for
different aspects of the call cycle were presented and the factors required for proper call
management were discussed. This expert‐driven process resulted in a high level of agreement and
a set of recommendations for call organisation, proposal evaluation and ranking, project selection
and funding, and monitoring. This document is the summary report of the topics selected by the
PLATFORM partners as the most relevant for mutual learning and of the discussions of these topics
in the master class.
Dissemination to partners of the PLATFORM project, wider community of bioeconomy ERA‐NET
actors, ERA‐LEARN and further ERA‐stakeholders.
Contacts of PLATFORM WP3: Christian Listabarth ([email protected]), Ulla Sonne Bertelsen
([email protected]), Christine Bunthof ([email protected]).
Core Statement
The bioeconomy ERA‐NET community stresses the importance of sufficient flexibility in the
evaluation and selection procedures, e.g. in the freedom to use different models for
ERANET calls, such as multiple topics or different participation rules for sub calls, and
different modes for evaluation outcomes (barrels / same score possibility / full ranking
list). This would foster a larger impact of the investment provided through national funds
and EC top‐up funding on building the ERA.
Page 134 of 154
A. Support tools for matchmaking and handling of applications and evaluation
Regarding electronic submission and evaluation systems (ESS) and management tools there is no
one‐size‐fits‐all‐model. Existing models are either tailor‐made or adjusted from existing National
Funding Organisations. There are both ESS service providers and (at least one) open source system
available; in addition, a simple web‐based drop‐box system may be used (protected by user‐key and
password).
Recommendations for the ERA‐NET community




Simple ESS systems and profiles are recommended.
It is recommended that for a single call one of the many ESS providers should be contracted,
while for multiple calls the adaptation and tailor‐made modification of the open source ESS
of ICT‐AGRI would be a very adequate and sustainable solution.
Electronic databases, in particular those built in open source systems might be prone to
security attacks, but one should consider the risk inherent to the technical security issues in
perspective of the low value the data would have to professional cyber criminals and the low
chance of researchers trying to gain competitive advantage by breaking into the database. A
web‐based drop‐box is also considered safe. Submission and proposal handling by E‐mails is
not considered sufficiently secure.
More importantly, the confidentiality of proposal data should be guaranteed during the
assessment phase within the ERA‐NET: confidentiality of data is a relevant issue to be
addressed and adequate confidentiality regulations must be well defined and lived up to.
Page 135 of 154
B. Evaluation and ranking of applications
Recommendations for the ERA‐NET community















1
Transparent communication of National Regulations regarding eligibility is required.
Guidelines for Evaluation and Selection have to be thoroughly thought through and clearly
formulated and structured.
There is absolute need for a clear Code of Conduct on Conflicts of Interest.
There is absolute need for clear instructions to External Reviewers (ER) and Review Panel
(RP) on the selection criteria and about scoring. It is doubtful whether the ranking of
proposals can be sufficiently justified, due to inter‐person differences in the use of scores
and due to low comparability of proposals in many calls.
Ranking lists, in theory, show a linear progression of quality among proposals, however, in
practice, ranking within nearest neighbours do not necessarily represent a (measurable)
sequence of quality. They may rather represent an intellectual artefact when comparing
equally good proposals.
There is practical need for a limited range of scores in order to avoid artefacts in the ranking.
Review Panel evaluation remains desirable in most cases (particularly for the basic sciences).
The ESF provides further guidance for Peer Review1.
Because in most calls the amount of funds applied for is much higher (often three‐fold or
more) than the available funds, a two‐step submission and selection process (pre‐ and full
proposals) is the preferred model. This reduces disappointment on the applicants’ side and
reduces workload for all actors: applicants, agencies, external reviewers, the review panel
and the call office.
Frequently a success rate of at least 25% for full proposals is aimed for. Anything below this
makes a cost‐benefit balance negative. A two‐step procedure enables to reduce the number
of full proposals being submitted. Finally, the number of expected proposals determines
whether a pre‐proposal stage is needed.
The result of the pre‐proposal assessment could be employed as a non‐binding
recommendation in case national regulations don’t allow a rejection without external peer
review or without a full evaluation.
A rebuttal step (after review of full proposals, but before the panel meeting) is
recommended because it adds value to the quality of assessment and transparency. It is
essential that applicants write a rebuttal only on arguments, not on scores (scores might
change during the panel meeting and their disclosure might be prone to abuse).
Access to the expert database of the Commission Services is possible.
To divide the workload in the search for External Referees, some ERA‐NETs agreed to do it
proportional to the proposals: countries that had most applicants involved in proposals were
expected to provide most ER names (and account for financial obligations accordingly).
To pay fees to the External Experts and Review Panel members would value their
indispensable contribution to the quality assurance of the evaluation process. Even only
symbolic fees paid to evaluators, as a token of expressing gratitude, are therefore highly
recommended, in order to maintain the (in principle exploiting) system of scientists to
volunteer as referees.
http://www.vr.se/download/18.2ab49299132224ae10680001647/European+Peer+Review+Guide.pdf
Page 136 of 154
C I. Selection of recommended applications and funding decision
Recommendations for the ERA‐NET community




Funding recommendation should be based on best practice, identified as a two‐step
evaluation with a final panel ranking, in which panel experts have read most or all proposals.
Variable geometry in the provision of funds often leads to the rejection of even highly ranked
projects. Ranking in barrels instead of a prescriptive ranking list is better suited for
transnational calls with distributed pots as it allows optimizing the use of available funding.
Therefore a grouping in barrels is the preferred mode of the evaluation outcome.
Another rationale for preferring barrels over a fully ranked list is that ranking often suffers
from artificial hierarchy among more or less equally ranked proposals. A barrel mode is
therefore more convenient. It divides proposals into different categories (barrels), in which
proposals within a quality category are equally ranked.
In the ERA‐NET Plus scheme the more top‐up funding goes to the gap‐filling, the better a
ranking list can be served. To optimize top‐up funding, one should take this into account, and
balance it against the argument of grant distribution in ratio to the countries funds
contribution.
CII. Project negotiation
Recommendations for the ERA‐NET community




Clear communication by national funding organisations about maximum funding in total as
well as per project is necessary to avoid budget cuts at a late stage. Funders should publish
funding ceilings and the type of project costs they expect to fund. Transparency is important
here.
Avoiding any cut‐back of project budget during funding negotiations is considered best
practice (as long as the funding requests are appropriate and considered necessary), because
doing so would be a form of putting aside the outcomes of an evaluation and as such it
would be disrespectful to “reduce” a positively evaluated proposal. If necessary, any revised
proposal should be re‐evaluated by (part of) the Review Panel.
The partners within a project should start according to the work plan and thus, if needed,
political power must be mobilised to streamline the timeline of national funding decision
procedures.
ERA‐NETs should inform applicants timely that they will have to negotiate their Consortium
Agreement themselves within the project. It is also recommended to point out that
templates for Consortium Agreements are available on the DESCA website and from ERA‐
LEARN / NETWATCH. Specific templates may be added.
Page 137 of 154
D. Joint project monitoring
Recommendations for the ERA‐NET community






In general a trust‐based system for following‐up the projects funded by the ERA‐NETs during
their lifetime should be aimed for, in particular in view of the peer‐based evaluation system.
Research organisations deserve to be trusted and reporting requirements should be done
only at the absolutely required minimum. E.g., project management is an integral part of the
application and has been evaluated at the start of the project. Funding organisations should
trust that the project coordinator implements this as proposed, and only a very light, if any,
monitoring of this aspect is required.
The financial management is usually placed at the national level through national contracts,
and the consequently financial reporting should remain as well at the national level.
Scientific progress monitoring is beneficial at the transnational level as it will depend on the
performance of the consortium as a whole.
The assessment of the project performance should not be overregulated, and in case of
intermediate reviews it should be kept in mind that the aim is rather a help the project to
perform well then to judge and compare. Forms, if any, should by straightforward to fill in
and only contain questions for information that is really required. Funders should aim for
minimal bureaucracy and researchers should only have to spend a minimum of time to
administrative burdens.
The organisation of seminars where project coordinators present results and explain
deviations from the plans in the presence of reviewers and contact points of the national
funding organisations is highly recommended. This is a much more inspiring alternative for
mid‐term and end term progress reporting, and it provides networking opportunities among
the community. The topics of the projects of a thematic ERA‐NET will generally be of interest
to the scientific community of that area, so the seminars could be open to a wider public, e.g.
by organising it within a larger conference.
Financing of monitoring and the ex‐post evaluation is considered to be a bottleneck and
needs to be addressed. This also relates to the discussion about the development of self‐
sustained ERA‐NETs.
Page 138 of 154
Report of the Master Class on call management
17-18 June 2013
Organised by WP3 Mutual Learning
Date:
17 June at 11.00 to
18 June at 15.00
Venue:
KoWi premises
Rue du Trone 98, 8th floor
Brussels, Belgium
Dinner:
17 June at 20:00, Restaurant Hemispéres
Report contents
Monday June 17
Welcome & Introduction
A. Support tools for matchmaking and handling of applications and evaluation
B. Evaluation and ranking of applications
C. Selection of recommended applications, funding decision and project negotiation
Part I selection of projects and funding decision
Tuesday June 18
C. Selection of recommended applications, funding decision and project negotiation
Part II project negotiation
D. Joint project monitoring
Any other topics that were brought up by participants
List of presentations
Participants
Page 139 of 154
Monday June 17
Welcome & Introduction Welcome by Christine Bunthof (P1) with short introduction to PLATFORM followed by tour de table of the participants. A quick introduction was made by Christian Listabarth (P9) to the ERALEARN call toolbox and how to get help for the topics that will not be discussed in this Master Class, and announced that ERA‐LEARN will organize a workshop for ERA‐NET newcomers in September/October this year, in which all topics would be covered. Topics relevant to this master class were selected by the PLATFORM partners and an introduction to the topics of the Master Class presented. ERA‐LEARN started in 2009 and established a toolbox for call implementation and coordination, among others, and organized the annual ERA‐NET and Joint programming Conferences; ERA‐LEARN feeds information into the Netwatch portal (http://netwatch.jrc.ec.europa.eu/web/lp/learning‐
platform). It collects and provides generic information but does not make recommendations, thus PLATFORM could complement the toolbox through recommendations based on the expertise of the participants. The ERA‐LEARN call toolbox has a modular structure with six sections. In the Toolbox all uploaded documents of PLATFORM participants could be made available to the ERA‐NET community as complementary information, and a set of recommendations (outcome of the master class) could be interlinked with the relevant sections of the toolbox. This workshop aims at recommendation of good practises in call management and tries to collect relevant documents and templates from the various ERA‐NETs (to be provided to ERA‐LEARN). The general structure of the meeting was introduced by Ulla Sonne Bertelsen (P11). Each item was introduced by the session chair. After a case presentation, a Round Table was made. The questions and remarks led reflections and further sharing of information and ideas. The session chair guided the discussions to conclusions and recommendations, which were collected and collated for a report on good practices and recommendation. A. Support tools for matchmaking and handling of applications and evaluation Chair Ulla The session focussed on use and performance of electronic submission and evaluation systems (ESS). This also touched upon security standards (the system, the handling of the proposals and storage and access). 1) CASE Iver Thysen from ICT‐AGRI (Information Technology and Robotics in Agriculture) introduced the “Meta Knowledge Base” which is being transformed into an open source system for matchmaking and on‐line submission and evaluation of applications. It can be used by all ERA‐NETs. The ERA‐NET ICT‐AGRI collects metadata of projects and partners and provides it Page 140 of 154
on‐line for about 1100 users at the moment. Through a Member login one can join forces in a secured setting. There are opportunities to create private or public project‐proposals. The tool can also serve for matchmaking; one can share and search for profiles. The coordinator (ICT AGRI) is currently exploring a suitable open‐source platform (DRUPAL) in order to adapt the system and to share data with the community outside ICT‐AGRI. Once ready, it will be available for free use by other ERA‐NETs. Key issues are to fill and maintain the database; at current stage the partner‐profiles are rather advanced; available actual project information is limited. Consortium information of the projects can be posted by coordinator. National regulations such as funding schemes, themes, and eligible funding are provided automatically if the data have been entered into the system. The electronic application forms available are online as well as a tool for on‐line evaluation. The system also provides a tool to automatically calculate budgetary consequences of different proposal selections supporting the discussions and decision taking during selection of proposals and helps maximising the number of funded proposals. 2) DISCUSSION Discussion of working methods and experiences along the questions: do you assist in match making? Do you use an electronic submission system? How do you secure the data? Call management systems in use. A quick survey of other operational call management systems yielded: PTJ uses an ESS (PETERAS); this system is adjustable through a subcontracted ICT company; it is not possible to make it open source. However, carrying the costs of subcontractor and workforce spent for adaptation by Project Management Jülich, Germany, the system is available for other networks. It does not provide a database for selection of referees. Except for Project Management Jülich, there are other providers of ESS, too, mostly offering the ESS including specific modifications on demand: DLR, VDI/VDE, AIT (all GE), NWO (NL), and a Basque private company. ERA‐LEARN toolbox will update the chapter on ESS during summer. ERA‐NET Bioenergy used a drop‐box system (access by user key and password) to handle some 30 proposals successfully. However, the ESS is also a sign of prestige, thus e.g. large companies could be reluctant to submit proposals to a drop‐box. Most ERA‐NETs are already using an ESS, others look for a simple and cost efficient system. Some ERA‐NETs (SUMFOREST/CoFASP) are exploring the market for existing ESSs that would fit for their own purposes. Match‐making methods. WoodWisdom‐Net organised a brokerage event for matchmaking during their last conference: every participant gave a short flash presentation of project ideas or partner competences. Afterwards all flash presentations were made public on the WoodWisdom‐Net website. The over‐all feeling was that it worked quite well to find matching collaboration‐partners. Participants were charged a small fee, thus there were zero costs. ERASynbio and some other ERA‐NETs, offers factsheets for interested partners to profile themselves on the ERA‐NET webpage. Data‐protection. How about data protection when using an open source system? Open source systems can be installed on your own server and you carry own responsibility about the data. Data protection/security is more of a concern once you go into the open‐source system. Though it depends on the competition within the area, it was generally felt that one should not over‐react on security issues. Also a web‐based drop‐box protected by user Page 141 of 154
key and password is considered safe, but submission and proposal handling by E‐mails is not considered sufficiently secure. In order to protect novel ideas access to proposals should be limited to evaluators, who should declare confidentiality. Adequate confidentiality regulations must be defined. National contact points should have exclusive access to all data, however. 3) CONCLUDING AND RECOMMENDING. Concrete conclusions and recommendations from the group:  Electronic Submission and management tools: There is no one‐size‐fits‐all‐model; more advertisement of existing open source models and/or service providers is needed.  The community should get ready for the new ERA‐NET‐Plus top up scheme (funding distribution tool). Simple systems and profiles are recommended.  Existing models are either tailor‐made or adjusted from existing National Funding Organisation.  There is a need for a budgetary tool for financial recommendations after ranking. ERA‐
SysBio Plus uses a system based on Excel sheets (contact Bernhard Gillissen); this is available in the ERA‐LEARN toolbox1.  Confidentiality of proposal data should be guaranteed during the assessment phase; security of data is a relevant issue to be addressed.  List of systems should be provided on ERA‐LEARN (e.g. PTJ system PTERAS; Open Source system, etc.  To aim for a single cost efficient meta‐data system for all ERA‐NETs would be beneficial. B. Evaluation and ranking of applications Chair Christine The session was introduced with a message that quality assurance of your evaluation and ranking procedure is a key‐factor in joint funding of high quality international research‐
projects. The evaluation process is very critical for the selection of high quality projects, and is seen as the most stressful period in a call procedure by some call managers. 1) CASE Part I Paul Beckers from ERA‐CAPs introduced their method of expert‐ and panel‐evaluation, which is based on best practice in the previous ERA‐NET Plant Genomics (ERA‐PG) calls. They used a one‐step full proposal submission procedure using an adjusted ESS of the German Science Foundation (DFG). So far there has been one thematically open call within molecular plant sciences area; Am eligibility check has been performed according to the criteria that are published in Call Notice and National Regulations; external evaluators were used to obtain expert opinions (2‐3 independent experts, non‐numerical rating). Different sources for evaluator recruitment were presented; the procedure included a rebuttal step 1
http://netwatch.jrc.ec.europa.eu/web/lp/learning‐platform/toolbox/call‐implementation/funding‐
decisions/funding‐decisions Page 142 of 154
where applicants are allowed to briefly comment on the anonymous review reports. A Review Panel assessed the proposals, review reports and rebuttals lead to a grouped ranking (=List numerically but grouped). The procedure complies with the basic principles of Peer Review (Code of Conduct on Conflicts of Interest; Application Guidelines; Evaluation and Selection Guidelines. Part II Peer Review Services offered by the European Science Foundation (ESF) were presented as a possibility for outsourcing. ESF has a longstanding experience in international peer review and produced a reference document on best practices2 . The service they offer is modular and can be tailored to specific needs. Costs aspects need to be assessed on a case by case basis. 2) DISCUSSION on methods and experiences Pre‐proposals v. Full Proposals (one‐step v. two‐step procedure).For a combination of reasons from too heavy burden to scientific community and administrative staff to negative cost‐benefit analysis, ERA‐IB skipped the external review step and only used an expert panel with rebuttal step over time. Most other ERA‐NETs use pre‐proposal step but still use external peers for full proposals; pre‐proposals enable you to increase the success rate and to reduce the efforts to be made by applicants, reviewers and administrative staff. In various ERA‐NETs either Funding Agencies or Review Panel decides on pre‐proposals. Applicants tend to object to rejection by administrators; they want an expert opinion as a basis for rejection. External Peers. There is a need for databases on qualified peers; EUREKA may give access to their sources. EC provides access to evaluator databank on request. The process of getting access to the EC database is described in the ERA‐LEARN toolbox3, Evaluation criteria and comparability of scores. It is important to provide strict and clear guidelines on evaluation criteria (quality and weighting), and to provide exact explanations for the use of scores (in almost all ERA‐NETs discrepancies in the use of scores have been observed). It is good practice to reduce the range of scores: evaluators like it and are forced to take decisions. In general, except for very narrow topics and within a panel in which each panellist knows every single proposal, the comparability of proposals to justify a final ranking list was heavily challenged. Rebuttal. Rebuttal‐stage has been welcomed by most of the science community. Some of the ERA‐NETs used it; others are considering to do so. All who employed rebuttals found it a good experience rather than a burden. Outsourcing Peer Review. The interest for outsourcing depends on the actual costs. ERASME and Cornet used Eureka office for their peer review; outsourcing may be an option. European Science Foundation could be contacted to discuss the needs and to obtain quotes for their services. 2
3
http://www.vr.se/download/18.2ab49299132224ae10680001647/European+Peer+Review+Guide.pdf http://netwatch.jrc.ec.europa.eu/web/lp/learning‐platform/toolbox/call‐
implementation/evaluation/procedures/decentralised‐evaluation‐carried‐out‐by‐national‐
programmes. Page 143 of 154
3) CONCLUDING AND RECOMMENDING. Concrete conclusions and recommendations from the group:  There is a need for transparent communication of National Regulations regarding eligibility.  A two‐step process (pre‐ and full proposals) is the preferred model because it decreases disappointment on the applicants’ side and reduces workload for ER and RP (review panel).  Frequently a success rate of at least 25% but preferably 30‐40% for full proposals is aimed for. Anything below 25% makes a cost‐benefit balance negative. A two‐step procedure enables to reduce the number of full proposals being submitted.  Number of expected proposals determines whether a pre‐proposal stage is needed.  The result of the pre‐proposal assessment could be formatted as a non‐binding recommendation in case national regulations don’t allow a rejection without external peer review or without a full evaluation.  Review Panel evaluation remains desirable in most cases.  Need for clear Code of Conduct on Conflicts of Interest and Guidelines for Evaluation and Selection.  Clear instructions to ER and RP about scoring criteria and a limited range of scores in order to avoid artefacts in the ranking.  Access to the EC expert database (and to the EUREKA database – to be explored) is possible; access rules to expert databases should be shared.  To spread the workload in the search for External Referees, some ERA‐NETS agreed to do it proportional to the proposals: countries that had most applicants involved in proposals should provide most ER names.  Rebuttal step is recommended because it is adding value to the quality of assessment and transparency.  Even symbolic fees paid to the Review Panel members would value their indispensable contribution to the quality assurance of the evaluation process. C. Selection of recommended applications, funding decision and project negotiation – part I Chair Christian Part I concerns the selection of projects and funding decision. 1) CASE Nicolas Tinois from FZ‐JUELICH introduced different experiences on selection of applications and funding decisions. Different methods to get to a recommended funding list were addressed. Evaluation results can either be presented as (binding) ranking list or a group model. Possibility to redress a proposal was offered in specific cases. Redressing a proposal is to fulfil a condition in order to become eligible for funding, e.g. to replace a partner who is not eligible. Various models are applied: ETB, CORE Organic: Group model Page 144 of 154
SUSFOOD: binding ranking list per topic (i.e. more than one ranking list) (e.g. ANIHWA tries to fund at least one proposal per topic). FACCE‐JPI; one unique proposal 2) DISCUSSION Differences between Rebuttal (process during evaluation) and Redress (process after first or second evaluation step) were explained. Redressing is meant to be more flexible to replace partners who drop out for unforeseen reasons (e.g. SMEs who went bankrupt). The new ERA‐NET scheme under Horizon 2020 requires that the funding recommendations should be based on a binding ranking list. This would leave no room for using the group model, which most participants considered the best model for funding a maximum of good projects. Often the restricted availability of funding from one (or more) partner(s) prevents other partners to fund projects that are evaluated as good projects, but which are not ranked high enough. An optimization of funds is, therefore, not possible. Simultaneously the reliability of ranking objectively and absolutely within a group of e.g. excellent projects was heavily doubted by the participants. Groups of similar project quality help to avoid jumping along a poorly trusted, absolute ranking list, and are considered the single most appropriate outcome of a thorough evaluation. As for the application of ERA‐NET Plus, successful applicants to FP7 mentioned that the global funding of the consortium must not be overrated in the ERA‐NET Plus application, because of the potential drop‐out of funds considering the distribution of effective funds following the binding ranking list. In the ERA‐NET Plus scheme, for example, a maximum of 8 M€ top‐up funding from EC should be matched by 16M€ actually deployed (not potentially available) for funding from the national partners. The question of how much of the top‐up funding should be spent to fill gaps of the ranking list and how much will be attributed to national funding budgets were discussed. This must be decided before the funding recommendations are made, but it is evident that the more top‐up funding goes to the gap‐filling, the better a ranking list can be served. 3) Preliminary CONCLUDING AND RECOMMENDING  Recommendation for the new ERA‐NET scheme should be in favour of a group scoring without making any concessions to the quality of the evaluation of the research proposals.  Groups must be defined as equally ranked proposals.  Two step evaluation and panel ranking is preferred model/best practice; maintain best practice and reduce unnecessary ranking to assure the optimization of spending tax‐
payers money. *****End of day 1 of the master class***** Page 145 of 154
Tuesday June 18 Conclusions of the discussions of Part C.I Introduction by Christian to summarize, continue and conclude the discussion of the previous day: A two‐step evaluation with a final panel ranking, in which panellists know most or all projects, was identified best practice. It is absolutely important to aim at the highest quality of the evaluation process, but final scores are considered to introduce artificial hierarchy. Final scores of a ranking list with minute differences tend to be arbitrary, due to (i) different individual use of scores and (ii) a poor comparability of projects. Variable geometry in the provision of funds often leads to the rejection of even highly ranked projects. The group model was the preferred model of the evaluation outcome, because it allows optimizing the funding recommendations. A possible solution would require the following characters: (i) to avoid any loss of, and keep the highest quality of evaluation; (ii) to reduce unnecessary (and doubtful) hierarchy in the ranking and allow for equally ranked proposals; (iii) to consolidate the dichotomy of “ideal nuances of scientific quality” v. “the pool of proposals considered cutting edge/excellent“. Reflections from the brainstorming session among participants on how to solve the current proposed EC regulations of the ERA‐NET Plus scheme:  It is virtually impossible for a Review Panel to provide a rational basis for distinguishing between proposals of which the overall quality is very much alike. This leads to an artificial ranking of neighbouring applications which does not reflect the actual scientific appraisal of the proposals. Experience shows that there is no rational basis to rank one proposal over the other among a category that is defined narrowly enough. E.g. among ‘very good’ proposals any further distinction is often arbitrary.  Like the EC, the national procedures for proposal selection aim for excellence, and the intention of national research funding organisations is to provide funds for the best transnational project proposals. Therefore, any loss and/or concessions to the quality of the evaluation of the research proposals are to be avoided.  It is top‐priority that national funds and the top‐up funding must be effectively used in order to benefit the ERA. Options that provide solutions to the problems that arise from binding ranking lists are:  Ranking proposals on equal ranks; as this would reduce the controversy of artificial hierarchy and benefit the optimization of using the funds available.  Creating multiple parallel calls with different scope or participation possibilities instead of one rigid format; as this would allow for desired flexibility in ERA‐NET Plus implementation.  An approach with an over‐all and subtopic ranking lists where the allocation of national money starts at the top‐ranked projects; top‐up funding can be used to cover topics that otherwise would not be funded. These options are constrained by EC documents that already state that there can only be one call with one single and binding ranking list in ERA‐NET Plus in Horizon 2020. Page 146 of 154
 The community is in favour of keeping the flexibility of different evaluation outcomes and models for ERA‐NET calls. This would foster a bigger impact of the investment provided through national funds. C. Selection of recommended applications, funding decision and project negotiation – part II Chair Ulla Part II concerns the project negotiation. The distributed pot mode of funding, which is used most often in ERA‐NET calls combined with unbalanced participation in proposals compared to national budget commitments creates sometimes complex puzzles for funders. 1) CASE Nicolas Tinois presented his experiences on negotiation with the unique proposal for a FACCE‐JPI Knowledge Hub. This proposal had to be redressed based on recommendations of evaluators 2) DISCUSSION Negotiating proposal budgets. In general it is good practise to avoid cutting back parts of projects to save money (in order to guarantee that national budgets are not exceeded). This is perceived as disrespectful towards applicants and evaluators. If necessary any revised proposal should be re‐evaluated by (part of) the Review Panel The National Funding Organisations (NFOs) should check their financial mandates before the Moderating Panel meeting to speed up the process of funding decisions. To increase the transparency and to avoid raising wrong expectations, NFOs should publish funding ceilings and project costs they expect to fund. Expected budget reductions should be communicated to the applicants. National funding decisions. One of the perceived bottlenecks in the negotiations is getting the national funding decisions at the agreed time. This is no longer justifiable in view of the general ambition of smooth and lean public funding procedures. In cases where a signed consortium agreement is mandatory before the start of the project/ a funding decision can be taken, this might cause additional delay. It depends on complexity of the consortium issues how simple the CA can be and how quick this CA can be signed. 3) CONCLUDING AND RECOMMENDING  Various templates for Consortium Agreements are available on ERA‐LEARN/ NetWatch, specific templates might be added, but finally the applicants themselves will have to negotiate their Consortium Agreement individually  Clear communication by NFO’s about maximum fundable amounts is necessary to avoid budget cuts at a late stage.  In general it is good practice to avoid cutting back parts of projects  Political power must be mobilised to streamline the timeline of National funding decision procedures in order to get projects started (simultaneously). Page 147 of 154
D. Joint project monitoring. Methods of following‐up the projects funded by the ERA‐NETs during their lifetime Chair Christian Monitoring of jointly funded projects is mandatory in the ERA‐NET Plus scheme, at least within the life span of the ERA‐NET action. Most national funders have monitoring procedures for projects in place. The question is how to replace or transform national monitoring systems in order to best serve the transnational character of the ERA‐NET calls. There are different levels of monitoring, which are often dealt with all together: (i) during the lifetime of a project, (ii) after the project end (impact assessment), and (iii) impact of the ERA‐NET instrument. Project monitoring can be used as an instrument to guide a consortium to a better project performance, while assessments of impact are rather ex‐
post evaluations. The Master Class item focused on monitoring of projects performance. 1) CASE Christine Bunthof introduced how ERA‐NET Plant Genomics monitored the funded research projects. The objective of the funding programme of ERA‐PG was to foster excellent science, transnational collaboration, and synergy in investments. ERA‐PG launched calls in 2006 and in 2008 that have a combined budget of over 55 million euros. In 2006 there were two sub calls to serve schemes as wished by different funding organisations., Sub Call A aimed at curiosity‐driven/basic science with funders financing academic teams, and Sub Call B aimed at innovation driven research by public‐private partnerships in which private partners were also eligible for receiving funding. In 2008 there was a second call for basic science. In the process of developing a common, effective and efficient set of procedures for the monitoring of funded projects, a working group was established taking into account current practices of the funders involved. The aim was to agree on purpose, content, frequency and form of the monitoring, and to produce a unified system for reporting There were differences between the Sub Calls in the objective of the reporting/monitoring rooted in the national schemes that were brought to collaboration in the ERA‐PG calls. In short, in Sub Call A monitoring was about justification of spent budget and in Sub Call B the follow‐
up process involved a mid‐term evaluation with go/ no‐go decision. Most steps of the process for Sub Call A and Sub Call B were the same, including a common procedure for collecting reports were the Call Secretariat sent mid‐term report forms to the leading PI of the funded projects. The submitted reports were uploaded on a restricted part of the intranet and funding organisations notified. The secretariat and National Contact Points assessed the report. Because projects had started at different times with almost a year difference between the start of the first and the last one, this work was spread out over a long period of time. Extra required steps were incorporated in the procedure for the 14 projects of Sub Call B. An expert panel read the reports and during a closed seminar for Sub Call B, PIs presented the projects and the evaluation panel discussed the progress and gave recommendations. The seminar also served for networking among the participants. The organisation of the monitoring after the contract with the Commission ended was a challenge. There was not a direct continuation and no funding organisation offered to take the whole task. The option chosen was to distribute the task earlier carried out by the Page 148 of 154
project management office (incl. call secretariat) to the funding organisations, so that the organisation funding the lead PI of a research consortium was, as ‘lead funding organisation’ responsible for communication with the consortium and with the other funding organisations to continue the way of working. Having the tasks thus distributed among a few NFO’s works, is not ideal. The recommended solution would be to keep the original secretariat running instead, if needed with the support of fees, and of course with National Contact Points in all organisations still in place. Grant‐holder meetings were held bi‐annually back‐to‐back with large conferences on plant genomics to have the project leaders presenting project plans or progress. This was felt beneficial by the scientific community and the national funding organisations and provided good networking opportunities. The evaluation of the ERA‐PG Programme (programme impact assessment) is taken up into one of the Work Packages of the ERA‐NET Coordination Actions in Plant Sciences, ERA‐CAPS Furthermore, ERA‐CAPS invited at its first programme seminar the consortia funded under the second call of ERA‐PG to present project results. 2) DISCUSSION The discussion on working methods and experiences with reporting and monitoring covered the following topics: Impact assessment. WoodWisdom‐Net has developed some parameters to measure impact. Annual seminars are used to collect and follow scientific progress. It may replace annual content‐reporting; this needs to be decided among NFO’s. The trans‐national aspect is considered the highest added value of impact assessment. Monitoring. The question was raised whether a go/ no‐go decision has an impact on the performance of the consortium. One felt the reporting requirement enables the coordinator to solve potential problems in the consortium. Only PTJ uses go/ no‐go decisions for large scale 5 year programmes. No further experience with this measure among stakeholders was expressed. Most programmes adhere to the national funding regulations which have an annual reporting requirement and are only entitled to withdraw money if a grant‐holder is not performing well or at all. CORE Organic uses web‐based evaluation meetings instead of physical meetings. ETB / ICT‐AGRI has a national monitoring system in place which is shared among the involved NFOs. ICT‐AGRI developed an on‐line monitoring tool to be used for uploading reports, publications and a discussion forum. BBSRC has no annual monitoring; reports are only end‐term. The gap between the end of the ERA‐NET project and the end of the monitoring process needs to be bridged. Frequently voluntary national funding organisations willing to contribute to this part, are employed in this task. The question arose what happens in case of discrepancies among NFO’s in their assessment of the project progress. In ERA‐PG, issues that potentially could result in a funder not accepting the report were generally resolved by the Secretariat asking the lead PI for providing more detailed information, or explaining deviation from the original plans more clearly. Some points are only relevant at national level (e.g. eligibility of expenses or hiring within a certain time after grant decision). In such cases, lack of compliance was dealt with on the national level. Page 149 of 154
Impact of the ERA‐NET instrument. An impact assessment on FP6 ERA‐NETs has been carried out, commissioned by the EC, by Matrix Insight – Rambøll4. They did an extensive
amount of work involving surveys among all ERA-NETs, many interviews and analysis. As far as any of the master class participants knew, there is no such extensive study planned for FP7 ERA‐NETs. Reports are available on Netwatch that map and monitor the ERA‐NETs and the Commission recently updated its report on ERA‐NET, ERA‐NET Plus and JPIs and their joint calls5. The bioeconomy ERA‐NETs welcome these monitoring and summary reports. The question came up if an extensive study on FP7 ERA‐NETS, as done for FP6 ERA‐
NETs would be useful in view of Horizon 2020. The ERA‐NETs present at the PLATFORM Master Class would have a general interest in a potential impact assessment of FP7 ERA‐
NETs and would contribute to surveys. It is, however, difficult to see the added value of such an assessment in the perspective of Horizon2020, since the framework is already set. It only would be useful if highlighting the rational of the FP7 ERA‐NET instrument (along ERA‐NET Plus) can affect the making of policy and regulations. Furthermore, it was recognized that reports demonstrating the impact could contribute to promoting ERA‐NETs as instruments, ultimately to leverage national funding for ERA‐NETs. 3) CONCLUDING AND RECOMMENDING. Concrete conclusions and recommendations from the group:  Financial reporting is better placed and will stay at the national level; monitoring of the scientific progress is felt to be useful at the transnational level.  Project management is integral part of the application and has been evaluated at the start of the project. One should trust the project coordinator that this is implemented as proposed, and no monitoring of project monitoring is required.  Financing of the ex‐post evaluation and monitoring is felt as a bottleneck and needs to be addressed. This relates to the discussion about the development of self‐sustained ERA‐NETs.  There is no joint impact assessment tool for individual ERA‐NETs, they do this individually.  Creating a legacy of what has been achieved in the various ERA‐NETs and emphasise the good aspects for future collaborative funding initiatives would be highly valued.  Use of conferences for mid‐term and end term is recommended.  There is strong support for developing an on‐line common database for monitoring; existing models like the one developed for ICT‐AGRI (Meta Knowledge Base) should be considered 4
Published in: FP6 ERA‐NET Study Summary of the Impact Assessment Study of the ERA‐NET scheme under the Sixth Framework Programme (EC report EUR 23909 EN, June 2009) and four volumes composing the full report. All can be retrieved at http://ec.europa.eu/research/evaluations (search in Documents on topic ‘ ERA‐NET). 5
Report on ERA‐NET, ERA‐NET Plus and JPIs and their joint calls, Directorate B – European Research Area B.4 – Joint Programming; May 2013 Page 150 of 154
End of the official agenda: A general request to the participants was presented: To upload relevant documents (in the context of the workshop) to the PLATFORM Intranet. Any problems with failing logins for PLATFORM should be forwarded to Linda Oud (PLATFORM project administrator). Any other topics that were brought up by participants: Selection of call topics 1. Iver Thysen presented a case on how to select topics for a Call for Proposals. As an example to demonstrate the need, he mentioned ICT‐ AGRI which did not receive project suggestions in the topics they thought were the most needed when launching broad calls. They decided to do topical calls instead. From a stakeholder consultation they deduced a Strategic Research Agenda and Action Plan. In order to implement the plan they would like to identify and involve new funders and partners. They used the ICT‐AGRI on line tool to consult the scientific community for identifying bottlenecks in their specific scientific area. This led to a harmonisation of topics that were useful for dedicated calls. 2. Alois Egartner presented the EUPHRESCO On‐line tool for the topic suggestion and selection process. To identify and select topics and build consortia EUPHRESCO developed an individual procedure with eight different phases within each round of research initiation (= call). Theses phases are independent from the funding mechanisms that finally are in use and require a one year period from the ‘preparation of initiation’ until the ‘funding decision phase’ in which partners decide on their support for the agreed topics. An online timetable (with e‐alerts) informs partners about the phases and the required actions. In the phase ‘initial identification of topic suggestions’ partners can propose topics via the online tool on the EUPHRESCO website. After a merge of topic suggestions with overlaps by call‐coordinators (‘reviewers’), partners can join the suggested topics and therefore inform the community about their potential funding interest in the phase ‘joining listed topic suggestions’. This is followed by three automated topic selection steps (separate phases) after which only topics remain in the process that fulfil the minimum criteria of having sufficient interest of partners, having an assigned Topic‐Coordinator and having an agreed short topic description. The ‘funding decision phase’ in which partners finally decide on their funding participation and therefore form the funding consortia closes EUPHRESCO’s topic suggestion and selection process. http://www.euphresco.org/public/calls/index.cfm Industrial participations registered in several ERA‐NETs (ERA TransBio, ERA‐PG, WoodWisdom‐Net) could be collected by consultation of the databases of those ERA‐NETS, which might be used to collect information about potential industrial stakeholders for topic selection and partnering in project proposals Page 151 of 154
List of presentations
The presentations have been distributed to the participants of the Master Class and are available for the PLATFORM partners on the project intranet. 1.
ERA‐LEARN Toolbox. Christian Listabarth 2.
ICT‐AGRI Call Submission and Administration. [CASE for item A] Iver Thysen 3.
ERA‐CAPS Joint Call for proposals 2012. Evaluation and ranking [CASE for item B] Paul Beckers 4.
ESF Peer Review Services [Contribution for item A] Paul Beckers 5.
Selection of recommended applications, funding decision and project negotiation [CASES for Item C] Nicolas Tinois 6.
ERA‐PG joint project monitoring [CASE for item D] Christine Bunthof 7.
EUPHRESCO Online tool for topic selection [Extra topic] Alois Egartner Page 152 of 154
Name Email Network 1 Alex Percy‐Smith [email protected]
ERA‐ARD 2 Alois Egartner [email protected] EUPHRESCO 3 Anabel de la Peña [email protected]
IPM 4 Anna Macey [email protected] ERA‐CAPS 5 Annette Kremser a.kremser@fz‐juelich.de ERASynBio 6 Dominique Vandekerchove1 7 Elfriede Fuhrmann2 [email protected]
elgie.be Elfriede.FUHRMANN@lebensministerium.
at 8 Ignacio Baanante Balastegui [email protected]
9 Iver Thysen [email protected]
10 Johannes Bender [email protected]
SUMFOREST 11 Katerina Kotzia [email protected]
CORE Organic and COFASP 12 Marie Ollagnon [email protected]
13 Marion Karrasch‐Bott m.karrasch@fz‐juelich.de ERA‐IB 14 Marta Norton [email protected] ERA‐MBT 15 Matte Brijder [email protected]
ERA‐NET BIOENERGY 16 Mika Kallio [email protected]
WoodWisdom‐Net 17 Nicolas Tinois n.tinois@fz‐juelich.de FACCE‐JPI 18 Paul Beckers [email protected]
ERA‐CAPs 19 Petra Schulte petra.schulte@fz‐juelich.de 20 Veronika Deppe v.deppe@fz‐juelich.de
ETB Organisers 21 Christine Bunthof [email protected] PLATFORM 22 Christian Listabarth [email protected]
23 Ulla Sonne Bertelsen [email protected]
RURAGRI ERASysAPP ICT‐AGRI ANIHWA 1) day 1 only. 2) day 1 only ERA‐NETs not attending: EMIDA, SAFEFOODERA, FORESTERRA, BiodivERsA.
Page 153 of 154
ARIMNET ERA‐MBT PLATFORM PLATFORM Page 154 of 154