Earth Observation DT-SPACE-01-EO-2018-2020: Copernicus ... · The proposal broadly addresses the criterion, but there are significant weaknesses. 3 — Good. The proposal addresses

Iulia SIMION, B1 - Space Research Unit, REA

Earth Observation

DT-SPACE-01-EO-2018-2020: Copernicus market uptake

Purpose of this briefing

• To ensure a common understanding of the work programme's key elements in order to have a single and harmonised approach to the evaluation

• To review the evaluation criteria and provide guidelines, including what should be addressed under which sub-criterion

General info

• During the briefing and remote evaluation, all evaluators must remain anonymous in order to avoid any bias

• Evaluators are invited to ask questions by email ([email protected]) Please specify the topic in the email subject: EO1. We will reply to these questions either during the briefing or afterwards by email.

• An independent observer is attending the topic briefing to monitor the process

• The slides and the FAQs will also be available on the Space 2018 Evaluation website at https://ec.europa.eu/info/h2020-space-2018_en

• You have already received by email the IER Guidelines document that summarises the information on the evaluation criteria presented here

Copernicus Value Chain - Process View

Important role for H2020 to enable

EO1 topic

overview

DT-SPACE-01-EO-2018-2020: Copernicus marketuptake

What? - Innovation Action (IA): to foster market development, exploiting the added

value of integration of Earth observation (EO) technologies (both satellite, airborneand ground based) other data from different sources and across different marketsegments through the development of applications, and encourage theirinsertion into the market.

How much? - Total budget: 9 million EUR

- Estimated amount per proposal: 1-2 million EUR

- Funding rate: 70%

• Focus Area 'Building a low-carbon, climate resilient future' (LC)


Funding instrument: Innovation Action

Description: Action primarily consisting of activities directly aiming at producing plansand arrangements or designs for new, altered or improved products, processes orservices. For this purpose they may include prototyping, testing, demonstrating,piloting, large-scale product validation and market replication.

A ‘demonstration or pilot’ aims to validate the technical and economic viabilityof a new or improved technology, product, process, service or solution in anoperational (or near to operational) environment, whether industrial or otherwise,involving where appropriate a larger scale prototype or demonstrator.

A ‘market replication’ aims to support the first application/deployment in themarket of an innovation that has already been demonstrated but not yetapplied/deployed in the market due to market failures/barriers to uptake. 'Marketreplication' does not cover multiple applications in the market of an innovation that hasalready been applied successfully once in the market. ‘First’ means new at least toEurope or new at least to the application sector in question. Often such projectsinvolve a validation of technical and economic performance at system level in real lifeoperating conditions provided by the market.

Projects may include limited research and development activities.

Funding rate: 70% (except for non-profit legal entities, where a rate of 100%applies)


EXCELLENCE:Foster market development [...] through the development ofapplications + their insertion into the market

Proposals• address the scalability and cost efficiency of the solution,demonstrating how it will work on a large region or even global scale.• innovative in at least one of these dimensions: market, product,process or business model

Earth Observation•exploiting the added value of integration of EO [...] other data fromdifferent sources• smart integration with other related technologies• the Copernicus […] as part of the solution + DIAS [...] leveraging onthe big data collection and processing

User• products […] shaped according to user needs• value to users must be openly demonstrated to the wider usercommunity• achieved in an environment integrated at the level of the user• evidence of user engagement shall be compulsory

IMPACT• Establish sustainable […] value added products and services

with demonstrated commercial value and targeted clientcommunities

• Demonstrate complete integration, based on internationalstandards, into the customer’s existing business processesand processing chains + the economic viability of theapplication

• Enhance European industry’s potential to take advantage ofmarket opportunities and establish leadership in the field, aswell as boost business activity

• Lead to new or improved products, processes or services on themarket, which are capable of generating a significantturnover and creating new jobs

Topic code: DT-SPACE-01-EO-2018-2020Topic name: Copernicus market uptake

Individual Evaluation

Report ( IER)

A w a r d c r i t e r i a , s c o r e s a n d w e i g h t i n g

AW A R D C R I T E R I A

Excellence Impact Quality and efficiency of the implementation

Scoring: from 0 to 5*, weakness / shortcoming

0 — The proposal fails to address the criterion or cannot be assessed due to missing or incompleteinformation.

1 — Poor. The criterion is inadequately addressed, or there are serious inherent weaknesses.

2 — Fair. The proposal broadly addresses the criterion, but there are significant weaknesses.

3 — Good. The proposal addresses the criterion well, but a number of shortcomings are present.

4 — Very Good. The proposal addresses the criterion very well, but a small number of shortcomings arepresent.

5 — Excellent. The proposal successfully addresses all relevant aspects of the criterion. Any shortcomingsare minor.

*You can award 0.5 points

Thresholds & weighting

Threshold for individual criteria = 3 + Overall threshold, applying to the sum of the three individual scores,=10.

Specific cases for Innovation actions (IA)

For the ranking, the score for the criterion ‘impact’ will be given a weight of 1.5.

Criter ion 1

- Excel lence

E X C E L L E N C E

• HOW DOES THE PROPOSAL ADDRESS THE SPECIFICCHALLENGE AND SCOPE OF THE WORK PROGRAMMETOPIC?

In particular, does the proposal “foster marketdevelopment, exploiting the added value ofintegration of Earth observation (EO) technologies(both satellite, airborne and ground based) other datafrom different sources and across different marketsegments through the development ofapplications, and encourage their insertion intothe market”?

• ARE THE OBJECTIVES CLEAR, MEASURABLE, REALISTICAND ACHIEVABLE WITHIN PROJECT DURATION?

C l a r i t y a n d p e r t i n e n c e o f t h e o b j e c t i ve s

E X C E L L E N C E

C l a r i t y a n d p e r t i n e n c e o f t h e o b j e c t i ve s

STRENGTHS

+ The objectives are clearly stated and are (fully) within the scope of the DT-SPACE-01-EO-2018-2020: Copernicus market uptake topic

+ The objectives are described clearly and appropriate quantifiable.

+ Key Performance Indicators have been identified to monitor xxxx.

WEAKNESSES/SHORTCOMINGS

- The specific objectives have not been presented with sufficient precision and have not been convincingly demonstrated as fully pertinent to the topic.

- The objectives are formulated in too generic terms and as general problems to be solved rather than as solutions to be provided. They are also not focused adequately on xxxx.

S o u n d n e s s o f t h e C O N C E P T , a n d c r e d i b i l i t y o f t h e p r o p o s e d m e t h o d o l o g y

• IS THE CONCEPT OF THE PROPOSAL SOUND (I.E. BASEDON MODELS OR ASSUMPTIONS WHICH ARE VALID ANDCONVINCING)?

E X C E L L E N C E

Does the concept include the following elements as highlighted inthe work programme?

" use of Earth observation and its smart integration with otherrelated technologies."“Copernicus should be considered as part of the solution whichmay include other space or non-space inputs."'Data and Information Access Services (DIAS) [or Alternativecloud computing providers] leveraging on the big data collectionand processing should be part of the proposed application'."Proposals need to address the scalability and cost efficiency ofthe solution, demonstrating how it will work on a large region oreven global scale."

S o u n d n e s s o f t h e C O N C E P T , a n d c r e d i b i l i t y o f t h e p r o p o s e d m e t h o d o l o g ySTRENGTHS

+ The concept is clear, achievable and sound, using xxxx models to determine xxxx.


- The concept is sound in general terms as it builds on xx and aims to xxx. However, therequirements of xxx in terms of the xxx information have not been considered insufficient detail and therefore the added value of the outcomes has not beendemonstrated convincingly.

E X C E L L E N C E

S o u n d n e s s o f t h e c o n c e p t , a n d c r e d i b i l i t y o f t h e p r o p o s e d M E T H O D O L O G Y

• DOES THE PROPOSAL SHOW THAT THE PROPOSEDMETHODOLOGY IS APPROPRIATE AND EFFECTIVE FORACHIEVING THE STATED OBJECTIVES?

E X C E L L E N C E

S o u n d n e s s o f t h e c o n c e p t , a n d c r e d i b i l i t y o f t h e p r o p o s e d M E T H O D O L O G YSTRENGTHS

+ The proposed methodology is credible, appropriate and coherent. How the specific challenges will be addressed is demonstrated very well, for example by xxxx.


- The methodology is presented too briefly, which affects the overall credibility. For example, the xxxx is not described in sufficient detail.

E X C E L L E N C E

E x t e n t t h a t p r o p o s e d w o r k i s b e y o n d t h e s t a t e o f t h e a r t , a n d d e m o n s t r a t e s i n n o v a t i o n p o t e n t i a l

• IS THE STATE OF THE ART WELL RESEARCHED AND

PRESENTED AND TO WHAT EXTENT DOES THE PROPOSED WORK GO BEYOND THESTATE OF THE ART IN THE PROPOSED PRODUCTS AND/OR SERVICES?

• IS THE PROPOSED WORK AMBITIOUS ENOUGH TO BE INNOVATIVE,WITHOUT BEING SO AMBITIOUS THAT IT IS NOT FEASIBLE?

DEMONSTRATES INNOVATION POTENTIAL

- WHICH ARE THERE NEW PRODUCTS, SERVICES OR BUSINESS ANDORGANISATIONAL MODELS PROPOSED AND TO WHAT EXTENT ARE THEYINNOVATIVE?

• As requested in the work programme, is the proposal innovative in atleast one of these dimensions: market, product, process or businessmodel and to what extent are they innovative?

• Is the proposal convincing on the strategy to encourage the insertion onthe market, as requested in the work programme?

E X C E L L E N C E

E x t e n t t h a t p r o p o s e d w o r k i s b e yo n d t h e s t a t e o f t h e a r t , a n d d e m o n s t r a t e s i n n o va t i o n p o t e n t i a l

STRENGTHS

+ A good analysis of the limitations of existing solutions has been provided and theoutcomes will go beyond the state of the art.

+ The end results have substantial innovation potential in terms of providing new toolsfor a very large world community. In particular, the innovation lies in making newinformation available to end users about the temporal and spatial resolution of xxx bydeveloping new models using Copernicus data.


- The proposed outcomes are designed as an improvement of an existing service, but theproposal is unconvincing about providing potential innovation or new approaches that gobeyond services that are already available, except for the possibility of xxx tools.

E X C E L L E N C E

Ap p r o p r i a t e c o n s i d e r a t i o n o f i n t e r d i s c i p l i n a r y a p p r o a c h e s a n d , w h e r e r e l e v a n t , u s e o f s t a k e h o l d e r k n o w l e d g e a n d g e n d e r d i m e n s i o n i n r e s e a r c h a n d i n n o v a t i o n c o n t e n t

• ARE RELEVANT INTER-DISCIPLINARY APPROACHES (I.E. THE INTEGRATION OF DATA, TECHNIQUES, TOOLS OR CONCEPTS FROM TWO OR MORE DISCIPLINES E.G. NATURALSCIENCES, TECHNOLOGY, ENGINEERING, ECONOMICS, SOCIAL SCIENCES) CONSIDERED INTHE PROPOSED CONCEPT?

• USE OF STAKEHOLDER KNOWLEDGE

WHO ARE THE STAKEHOLDERS (E.G. POTENTIAL USERS OR CUSTOMERS OR OTHERGROUPS OR ORGANISATIONS WITH AN INTEREST IN THE RESULTS OF THE PROPOSEDPROJECT)?

HOW DOES THE PROPOSAL DEMONSTRATE THAT THE CONSORTIUM WILL GET RELEVANTKNOWLEDGE FROM THEM (E.G. ORGANIZING WORKSHOPS WITH USERS, DISCUSSINGWITH CUSTOMERS, PARTICIPATING IN RELEVANT WORKING GROUPS ETC.)?

HOW WILL THIS KNOWLEDGE BE USED?

• Dose the proposal comply with the following requirements of the workprogramme

evidence of user engagement shall be compulsory and shall be provided aspart of the proposal, to demonstrate the user need and sustainability of theprojectthe product needs to be shaped according to user needs and their value tousers must be openly demonstrated to the wider user community; to beachieved in an environment integrated at the level of the users

E X C E L L E N C E

Ap p r o p r i a t e c o n s i d e r a t i o n o f i n t e r d i s c i p l i n a r y a p p r o a c h e s a n d , w h e r e r e l e v a n t , u s e o f s t a k e h o l d e r k n o w l e d g e a n d g e n d e r d i m e n s i o n i n r e s e a r c h a n d i n n o v a t i o n c o n t e n t

STRENGTHS

+ The proposed approach demonstrates very good interdisciplinarity, including xxx, yyyy, zzz and www.

+ The use of stakeholder knowledge has been addressed well. In particular, the SME partners have good access to international customers, and the proactive approach to market considerations will enhance the use of stakeholder knowledge.


- However, potential interdisciplinary and multidisciplinary research paths through, for example, the establishment of a dialogue with other areas of research, have not been sufficiently considered.

- The consortium includes stakeholders representing the demand side, but insufficient information has been provided about where their knowledge will be used in the development of the services.

E X C E L L E N C E

Criter ion 2

- Impact

• EXPECTED IMPACT AS DESCRIBED IN THE WORK PROGRAMME TOPIC:

“ESTABLISH SUSTAINABLE SUPPLY CHAINS FOR INNOVATIVE EARTH OBSERVATION VALUEADDED PRODUCTS AND SERVICES WITH DEMONSTRATED COMMERCIAL VALUE AND TARGETEDCLIENT COMMUNITIES;ESTABLISH SUSTAINABLE SUPPLY CHAINS FOR INNOVATIVE EARTH OBSERVATION VALUE

ADDED PRODUCTS AND SERVICES WITH DEMONSTRATED INNOVATIVE TECHNOLOGY;DEMONSTRATE COMPLETE INTEGRATION, BASED ON INTERNATIONAL STANDARDS, INTO THE

CUSTOMER’S EXISTING BUSINESS PROCESSES AND PROCESSING CHAINS, AS WELL AS THEECONOMIC VIABILITY OF THE APPLICATION;ENHANCE EUROPEAN INDUSTRY’S POTENTIAL TO TAKE ADVANTAGE OF MARKET

OPPORTUNITIES AND ESTABLISH LEADERSHIP IN THE FIELD, AS WELL AS BOOST BUSINESSACTIVITY;LEAD TO NEW OR IMPROVED PRODUCTS, PROCESSES OR SERVICES ON THE MARKET, WHICH

ARE CAPABLE OF GENERATING A SIGNIFICANT TURNOVER AND CREATING NEW JOBS.”• HOW MUCH WOULD THE PROPOSAL CONTRIBUTE TO EACH OF THE EXPECTED IMPACTS MENTIONED IN THE WORK

PROGRAMME? (Here you should base your evaluation on the assumption that the proposed concept andmethodology would work in order to avoid penalising the proposal twice for the same shortcomings.)

• Does the proposal identify any relevant barriers/obstacles and any framework conditions (such asregulations, standards, public acceptance, financing of follow-up steps) that may affect the extent towhich the expected impacts will be achieved?

I M P A C T

T h e e x t e n t t o w h i c h t h e o u t p u t s o f t h e p r o j e c t w o u l dc o n t r i b u t e t o e a c h o f t h e e x p e c t e d i m p a c t s m e n t i o n e d i nt h e w o r k p r o g r a m m e u n d e r t h e r e l e v a n t t o p i c

T h e e x t e n t t o w h i c h t h e o u t p u t s o f t h e p r o j e c t w o u l d c o n t r i b u t e t o e a c h o f t h e e x p e c t e d i m p a c t s m e n t i o n e d i n t h e w o r k p r o g r a m m e u n d e r t h e r e l e va n t t o p i c

STRENGTHS

+ The proposal demonstrates clearly that the outcomes will contribute significantly to theexpected impacts mentioned in the topic description, for example…


- The proposal only partly addresses the enabling of value-added services without the needto deploy storage and processing facilities. Moreover, it has not been demonstratedconvincingly that the outcomes will foster the establishment of interoperable access facilitiesto all EU Member States.

I M P A C T

An y s u b s t a n t i a l i m p a c t s n o t m e n t i o n e d i n t h e w o r k p r o g r a m m e , t h a t w o u l d e n h a n c e i n n o v a t i o n c a p a c i t y , c r e a t e n e w m a r k e t o p p o r t u n i t i e s , s t r e n g t h e n c o m p e t i t i v e n e s s a n d g r o w t h o f c o m p a n i e s , a d d r e s s i s s u e s r e l a t e d t o c l i m a t e c h a n g e o r t h e e n v i r o n m e n t , o r b r i n g o t h e r i m p o r t a n t b e n e f i t s f o r s o c i e t y

• ARE THERE ANY OTHER SUBSTANTIAL IMPACTS NOT MENTIONED IN THE WORKPROGRAMME WHICH ARE DESCRIBED IN THE PROPOSAL AND ARE RELEVANT TOTHE ISSUES LISTED ABOVE?

(THE PROPOSAL SHOULD NOT BE PENALISED IF IT DOES NOT HAVE ADDITIONALIMPACT.)

PLEASE DO NOT COMMENT ON 'CREATE NEW MARKET OPPORTUNITIES,STRENGTHEN COMPETITIVENESS AND GROWTH OF COMPANIES,' AS THESEELEMENTS ARE INCLUDED IN THE WORK PROGRAMME IMPACTS AND SHOULD BEASSESSED IN THE CRITERIA ABOVE

I M P A C T

An y s u b s t a n t i a l i m p a c t s n o t m e n t i o n e d i n t h e w o r k p r o g r a m m e , t h a t w o u l d e n h a n c e i n n o v a t i o n c a p a c i t y , c r e a t e n e w m a r k e t o p p o r t u n i t i e s , s t r e n g t h e n c o m p e t i t i v e n e s s a n d g r o w t h o f c o m p a n i e s , a d d r e s s i s s u e s r e l a t e d t o c l i m a t e c h a n g e o r t h e e n v i r o n m e n t , o r b r i n g o t h e r i m p o r t a n t b e n e f i t s f o r s o c i e t y

STRENGTHS

+ There is potential for the imaging platform to improve collective management and thus decrease impact of xxxx practices on the environment.

+ The proposal is clear about the potential societal and environmental impacts of the outcomes in areas such as health, climate change……, including secondary impacts such as improving productivity by lowering sickness rates.

+ The development of expertise in xxx observations has the potential to benefit long term environmental initiatives.

I M P A C T

• PLEASE ASSESS THE QUALITY OF THE "DRAFT PLAN FOR THE EXPLOITATION AND DISSEMINATION OF THE PROJECT'S RESULTS(PEDR)" WITH RESPECT TO THE SPECIFIC MEASURES TO BE IMPLEMENTED BOTH DURING AND AFTER THE END OF THEPROJECT.

“A BUSINESS PLAN […] SHALL BE COMPULSORY AND SHALL BE PROVIDED AS

PART OF THE PROPOSAL, TO DEMONSTRATE THE […] SUSTAINABILITY OF THE

PROJECT.”IS A BUSINESS PLAN INCLUDED?

IS THAT PLAN PROPERLY CONSIDERING THE COMPETITION, PRICING, REVENUE STREAMS AND FINANCIAL FORECASTS, THESATELLITE DATA COST IF NOT SENTINEL, (INCLUDING THE PHASE OF THE PROJECT AFTER THE PUBLIC FUNDING)?

IS THIS PLAN REALISTIC?

• IS THE STRATEGY FOR THE MANAGEMENT OF IPR APPROPRIATE? DOES IT DESCRIBE THE MAIN BACKGROUND IPR NEEDED TOCARRY OUT THE PROJECT, ITS OWNERSHIP AND ACCESSIBILITY, AUTHORISATION TO USE THIRD PARTY RIGHTS, AND THEOWNERSHIP PRINCIPLES THAT WILL GOVERN THE PROJECT RESULTS?

• IF RESEARCH DATA IS GENERATED, DOES THE PROPOSAL IDENTIFY THE TYPES OF DATA THAT WILL BE GENERATED, THESTANDARDS THAT WILL BE USED, HOW THE DATA WILL BE EXPLOITED AND/OR SHARED/MADE ACCESSIBLE FOR VERIFICATIONAND RE-USE?

• IS THE PLAN TO DISSEMINATE PROJECT RESULTS TO AUDIENCES WHO MIGHT USE THE RESULTS IN THEIR OWN WORK (E.G.PRESENTATIONS IN CONFERENCES, PUBLICATIONS IN PEER REVIEW JOURNALS ETC.) APPROPRIATE?

• ARE ACTIVITIES TO COMMUNICATE PROJECT ACTIVITIES TO MULTIPLE AUDIENCES BEYOND THE PROJECT'S OWN COMMUNITY(MEDIA, PUBLIC ETC.) INCLUDED? HOW ARE THESE TAILORED TO THE NEEDS OF VARIOUS AUDIENCES?

I M P A C T

Q u a l i t y o f t h e p r o p o s e d m e a s u r e s t o :- e x p l o i t a n d d i s s e m i n a t e t h e p r o j e c t r e s u l t s( i n c l u d i n g m a n a g e m e n t o f I P R ) , a n d t o m a n a g er e s e a r c h d a t a w h e r e r e l e v a n t- c o m m u n i c a t e t h e p r o j e c t a c t i v i t i e s t o d i f f e r e n t

t a r g e t a u d i e n c e s

STRENGTHS

+ The envisaged dissemination and communication measures are very good and are tailored to maximisecustomer participation, for example by visiting customers and using targeted webinars.

+ Proposal also presents an effective communication plan that clearly defines the target audiences andactivities foreseen, with specific strategies devised for each of them.

+ Key aspects of IPR, in particular licensing and the potential for patenting, have been considered withresponsibilities for activities in the area being clearly defined. There is clear consideration of both backgroundand foreground IP in the proposal.


- While it is stated that IPR will be addressed in the consortium agreement, the management of IPR has notbeen sufficiently considered in the proposal.

- Plans for exploitation are not convincing and are not clearly presented. In particular, the path toexploitation at the partner level is not adequately outlined.

- Dissemination and communication activities are treated together with an insufficiently clear distinctionbetween them. The proposed activities are in general appropriate but explained too superficially.

- The communication plan has not been presented in sufficient detail as it is described in terms of genericgoals rather than specific activities.

I M P A C T

Q u a l i t y o f t h e p r o p o s e d m e a s u r e s t o :- e x p l o i t a n d d i s s e m i n a t e t h e p r o j e c t r e s u l t s ( i n c l u d i n g

m a n a g e m e n t o f I P R ) , a n d t o m a n a g e r e s e a r c h d a t a w h e r er e l e v a n t

- c o m m u n i c a t e t h e p r o j e c t a c t i v i t i e s t o d i f f e r e n t t a r g e t a u d i e n c e s

• BUSINESS PLAN

IS A BUSINESS PLAN INCLUDED?

IS THAT PLAN PROPERLY CONSIDERING THE COMPETITION, PRICING, REVENUE STREAMSAND FINANCIAL FORECASTS?

IS THIS PLAN REALISTIC?

I M P A C T


m a n a g e m e n t o f I P R ) , a n d t o m a n a g e r e s e a r c h d a t a w h e r e r e l e v a n t


STRENGTHS

+ The exploitation strategy and business plan are very good, with valuable preliminarymarket and competitive analysis and credible financial considerations. For example, pricingmodels will be based on …

+ … Several different business cases have been described, including descriptions of potentialcustomers and their requirements. Each business opportunity will be analysed to predictexpected revenues in each case. Moreover, new business sectors have been identified,including ……


- Preliminary market analysis and details of annual operating costs have been provided.However, the commercial viability of the outcomes is not presented clearly in that the valueproposition, the technical service delivery, user needs and market prospects are notaddressed adequately.

I M P A C T




• IS THE STRATEGY FOR THE MANAGEMENT OF IPR APPROPRIATE? DOES ITDESCRIBE THE MAIN BACKGROUND IPR NEEDED TO CARRY OUT THE PROJECT, ITSOWNERSHIP AND ACCESSIBILITY, AUTHORISATION TO USE THIRD PARTY RIGHTS,AND THE OWNERSHIP PRINCIPLES THAT WILL GOVERN THE PROJECT RESULTS?

• IF RESEARCH DATA IS GENERATED, DOES THE PROPOSAL IDENTIFY THE TYPES OFDATA THAT WILL BE GENERATED, THE STANDARDS THAT WILL BE USED, HOW THEDATA WILL BE EXPLOITED AND/OR SHARED/MADE ACCESSIBLE FOR VERIFICATIONAND RE-USE?

I M P A C T




STRENGTHS

+ The tools for research data management are appropriate. Data produced will be preservedin the partners' information systems and made available (free and open).

+ Key aspects of IPR, in particular licensing and the potential for patenting, have beenconsidered with responsibilities for activities in the area being clearly defined. There is clearconsideration of both background and foreground IP in the proposal.


- While it is stated that IPR will be addressed in the consortium agreement, the managementof IPR has not been sufficiently considered in the proposal.

I M P A C T




Criter ion 3 - Qual i ty

and eff ic iency of the implementat ion

Q u a l i t y a n d e f f e c t i v e n e s s o f t h e w o r k p l a n , i n c l u d i n ge x t e n t t o w h i c h t h e r e s o u r c e s a s s i g n e d t o w o r k p a c k a g e sa r e i n l i n e w i t h t h e i r o b j e c t i v e s a n d d e l i v e r a b l e s

• HOW GOOD IS THE OVERALL STRUCTURE OF THE WORK PLAN?

• HAVE THE DIFFERENT WORK PACKAGES BEEN SCHEDULED AND LINKEDAPPROPRIATELY? ARE THEIR INTER-DEPENDENCIES CLEAR?

• ARE MILESTONES AND DELIVERABLES WELL DEFINED AND APPROPRIATELYTIMED?

• ARE THE WORK PACKAGES AND ASSOCIATED TASKS SUFFICIENTLY DESCRIBEDTO PROVIDE A GOOD UNDERSTANDING OF THE WORK INVOLVED AND JUSTIFYTHE PROPOSED RESOURCES TO BE ALLOCATED?

Q u a l i t y a n d e f f i c i e n c y o f t h e i m p l e m e n t a t i o n

Q u a l i t y a n d e f f e c t i v e n e s s o f t h e w o r k p l a n , i n c l u d i n ge x t e n t t o w h i c h t h e r e s o u r c e s a s s i g n e d t o w o r k p a c k a g e sa r e i n l i n e w i t h t h e i r o b j e c t i v e s a n d d e l i v e r a b l e s

STRENGTHS

+ The overall structure of the work plan is described well and is appropriate for theachievement of the objectives. The work package descriptions are thorough and appropriateresources have been allocated. The timing of work packages, tasks and activities is alsoappropriate and the deliverables match the objectives well.


- Some of the work package descriptions are defined in terms that are too generic. Forexample, a survey will be used for collecting user needs, but the nature of the survey or itstarget subjects have not been defined with enough precision.

- The allocation of resources across tasks is not well-balanced; for example, the allocationfor WP5 is heavily over-estimated.

- How the tasks and work packages relate to the presented work breakdown structure is notsufficiently described i.e. it is not sufficiently clear how nodes in figure xx relate to the tasksin the work package descriptions.


Ap p r o p r i a t e n e s s o f t h e m a n a g e m e n t s t r u c t u r e s a n d p r o c e d u r e s , i n c l u d i n g r i s k a n d i n n o v a t i o n m a n a g e m e n t

• ARE THE DESCRIBED ORGANISATIONAL STRUCTURE AND DECISION MAKINGMECHANISMS APPROPRIATE TO THE SCALE AND COMPLEXITY OF THE PROJECT?

• ARE THE MANAGEMENT PROCEDURES, INCLUDING QUALITY MANAGEMENT ANDCONFLICT RESOLUTION, APPROPRIATE TO THE SCALE AND COMPLEXITY OF THEPROJECT?

• ARE CRITICAL INTERNAL AND EXTERNAL RISKS RELATED TO PROJECT IMPLEMENTATIONDESCRIBED WITH APPROPRIATE MITIGATION MEASURES AND SEVERITY/PROBABILITYESTIMATES?

• HAS INNOVATION MANAGEMENT BEEN APPROPRIATELY ADDRESSED IN THEMANAGEMENT STRUCTURE AND WORK PLAN (I.E. DOES THE PROPOSAL SHOW HOW ANYSCIENTIFIC OR TECHNOLOGICAL INVENTIONS OR ADVANCES WILL ACTUALLY BE TURNEDINTO RELEVANT CONCRETE INNOVATIONS WITH SPECIFIC PRACTICAL OR COMMERCIALADVANTAGES OVER EXISTING SOLUTIONS)?


Ap p r o p r i a t e n e s s o f t h e m a n a g e m e n t s t r u c t u r e s a n d p r o c e d u r e s , i n c l u d i n g r i s k a n d i n n o v a t i o n m a n a g e m e n t

STRENGTHS + Management structures are appropriately addressed with roles and conflict resolution procedures being clearly defined.+ The main risks have been identified with appropriate mitigation measures. + Innovation management has been addressed very well using an incremental and iterative development approach.


- Project management principles are described; however, the detailed management structures and procedures (for example, the overall management board structure and the decision making and conflict resolution) are not adequately described.

- While risk management is, in general, adequate, risks pertaining to xxxx and yyyy are not sufficiently considered.

- Innovation management is inadequately addressed with no substantive detail provided in the proposal.


C o m p l e m e n t a r i t y o f t h e p a r t i c i p a n t s a n d e x t e n t t o w h i c h t h e c o n s o r t i u m a s a w h o l e b r i n g s t o g e t h e r t h e n e c e s s a r y e x p e r t i s e

• HOW IS THE COMPLEMENTARITY OF THE PARTICIPANTS DESCRIBED?

• DOES THE CONSORTIUM BRING TOGETHER THE NECESSARY EXPERTISE FOR THEPROJECT?


C o m p l e m e n t a r i t y o f t h e p a r t i c i p a n t s a n d e x t e n t t o w h i c h t h e c o n s o r t i u m a s a w h o l e b r i n g s t o g e t h e r t h e n e c e s s a r y e x p e r t i s e

STRENGTHS

+ The consortium is well-balanced and shows good complementarity in terms of the required technical and market expertise, and the presence of experts in xxx is very valuable.

+ The participants have the complementary skills required for the coordination and support activities and demonstrate extensive experience in xxxx. The consortium covers all the required expertise.


- The consortium consists of x participants with complementary skills; however, the high level of subcontracting required indicates that not all the required expertise is present in the consortium.


Ap p r o p r i a t e n e s s o f t h e a l l o c a t i o n o f t a s k s , e n s u r i n g t h a t a l l p a r t i c i p a n t s h a v e a v a l i d r o l e a n d a d e q u a t e r e s o u r c e s i n t h e p r o j e c t t o f u l f i l t h a t r o l e

• DOES EACH PARTNER HAVE A VALID ROLE AND ADEQUATE RESOURCES (BOTHEFFORT AND BUDGET) TO FULFIL THAT ROLE?

• IF "OTHER DIRECT COSTS" EXCEED 15% OF PERSONNEL COSTS:

HAVE THEY BEEN EXPLAINED ADEQUATELY?

ARE THESE COSTS "REASONABLE, JUSTIFIED AND COMPLY WITH THE PRINCIPLE OFSOUND FINANCIAL MANAGEMENT, IN PARTICULAR REGARDING ECONOMY ANDEFFICIENCY"(ART. 6.1.)?

PLEASE PARTICULAR ATTENTION SHOULD BE GIVEN FOR PROPOSAL WITH VERY HIGHOTHER DIRECT COSTS.

• IF ANY OF THE WORK WILL BE SUBCONTRACTED TO THIRD PARTIES, HASTHIS BEEN JUSTIFIED CONVINCINGLY?


Ap p r o p r i a t e n e s s o f t h e a l l o c a t i o n o f t a s k s , e n s u r i n g t h a t a l l p a r t i c i p a n t s h a v e a v a l i d r o l e a n d a d e q u a t e r e s o u r c e s i n t h e p r o j e c t t o f u l f i l t h a t r o l e

STRENGTHS

+ Tasks have been allocated appropriately and in accordance with the skills and expertise of each partner.

+ Each participant has a valid role with adequate resources.

WEAKNESSES/SHORTCOMINGS- The level of other direct costs in the proposal is not sufficiently justified; for example, the travel budget proposed are not substantiated by the number of trips foreseen, and the budgets for partners 2 & 3 do not have sufficient explanation.- The large allocation of effort to partner 2 is inadequately justified given the fact that its experience and expertise in the domain is insufficiently demonstrated, in particular in the…- The rationale for the subcontracting costs required is not adequately explained.


Quali ty of

the reports

‘If this was my proposal,would I find this report fair, accurate, clear and

complete?’

… even if it brings bad news …

Quality of the reports

Avoid:Comments not related to the criterion in question.

Double counting of same weakness under different criteria.

Contradicting statements relative to strengths and weaknesses.

Comments too short / too long / inappropriate.

Categorical statements not properly verified and quoting texte.g. “The proposal doesn’t mention user requirements” – when there is a short reference…

Scores that don’t match the comments.

Making recommendations.

Evaluating on potential, proposals need to be evaluated as they are.

Discrimination/political incorrectness.


As Evaluator you have to ensure that…

Comments in IERs and CRs are:• Specific to the relevant criterion• Clear and substantial• Facts, not opinions: "We think that…" "This proposal is… "• Consistent with the score, balancing strengths and

weaknesses• Of adequate length: not just one sentence, not a booklet!

Comments judge the proposal, they do not summarise it.

Do not provide advice on improving the proposal.

Good vs. poor comments


Poor comments merely echo the score

“The innovative aspects of the research programme are poor”.

Good comments explain it

“This proposal is not innovative in X or Y, and it does not take Z into account”.

Good comments are clear

“The resources in WP 4 and 6 are seriously underestimated given the complexity of the activity proposed”.

Poor comments are ambiguous

“The resources for the project are unrealistic”


Poor comments are vague, subject to interpretation

“We think the management plan is probably inadequate given the duration of the project and the number of partners”.

Good comments are precise and final

“The management plan is inadequate. It does not include clear risk assessment; it doesn't foresee a problem-solving mechanism in the event of disputes between partners”.


Poor comments are inaccurate and provide an opening for a complaint

“There is no discussion of a dissemination strategy.”

“There is only one industrial partner in the consortium.”

“The coordinator is not adequately experienced."

Good comments close the question

“Dissemination activities are listed but the proposal doesn't have a clear dissemination strategy”.

“The consortium doesn't include a sufficient industrial participation.”

“The coordinator does not demonstrate in the proposal an adequate level of experience in this field.”


Poor comments include words like…

PerhapsThinkSeems

AssumeProbably

Good comments include words like…

BecausePercent

SpecificallyFor example


Useful vocabulary

Insufficient, minimal, fails to describe, unacceptable, inadequate, very generic, not evident, unfocused, very weak, bad, does not meet the requirements, inappropriate, limited, unclear, not sound enough, not specified, no significant impact, unjustified, overestimated …

Extremely relevant, credible, very clear, precisely specified, realistic, very innovative, extremely well suited, timely, convincing, comprehensive, high quality, justified, very well identified, strong, highly effective, thoughtful, very promising, evidence, well-formulated, carefully prepared, very professionally prepared, fully in line, very profound, sound, very convincingly integrated, clearly articulated, coherent, well balanced, very plausible, ambitious, clear advances, well above average …


FAQs

EO1: FAQs:

Final

reminders

Process reminders

• Evaluators: accept the proposals assigned in SEP (by EOB Monday 23.4.)

- In case of non-acceptance provide a justification (e.g. CoI)

• Finalise and submit one individual evaluation report by Thursday 26.4.

- So that quality controllers can check the completeness of your report and give you feedback on whether issues were commented under appropriate subcriteria

- You will receive comments in your IER reports in SEP

- In some cases your report may be re-opened by REA staff so that you may complete it

• Complete the Remote Evaluation by Friday May 11th

- Necessary to allow the rapporteur to prepare a draft consensus report (CR) prior to the consensus meeting

• The draft CR will be available in the system a few days before the consensus meeting

- Read the draft CR prior to the meeting to prepare for the consensus discussion

Documents to read

Go to the Portal:

Work programme topic description:https://ec.europa.eu/research/participants/portal/desktop/en/opportunities/h2020/topics/dt-space-01-eo-2018-2020.html#budgetTable

H2020 Rules: http://ec.europa.eu/research/participants/data/ref/h2020/grants_manual/pse/h2020-guide-pse_en.pdf

Frequently Asked Questions

Thank you! In case of questions,

do not hesitate to contact the EO1 topic coordinator:

[email protected]

(for technical issues: IT Helpdesk)

Disclaimer:This presentation is intended for information purposes only within the framework of the evaluation of the call H2020-SPACE-2018.Please refrain from sharing it with other parties not involved in this process or from making it publicly available.