Activities of the ESF Member Organisation Forum on Research Evaluation

euroCRIS Strategic Seminar, Katharina Fuß

Brussels, 13.09.2010

Activities of the ESF Member Organisation Forum on Research Evaluation



1. Objective of ESF Member Fora

2. ESF Member Forum on Ex-Post Evaluation of Funding Schemes and Research Programmes

3. The new Forum on Evaluation of Publicly Funded Research

Beispielfolie

Inhaltsverzeichnis (mit Bild)

ESF Member Fora

Structure of the presentation






Beispielfolie


ESF Member Fora




The instrument “ESF Member Organisation FORUM” was developed in response to clearly expressed needs from MOs in consultations on the Strategic Plan 2006-2010

Quoting the Strategic Plan:

“MO Fora is an output-oriented, issue-related venue for MOs, involving others as appropriate, to exchange information and experiences and develop joint actions in science policy”.

“ESF aims to be the actor of choice and preferred partner when national research organisations develop joint activities.”

ESF Member Fora

Objective of the instrument



► Completed Member Fora

● Promoting Internationalisation of Social Sciences in Central and Eastern Europe (6/2006-12/2008)

● Evaluation of Funding Schemes and Research Programmes (10/2007-11/2009)

● Research Careers (11/2007-4/2010)

► Current Member Fora

● Peer Review (10/2007-12/2011, extended)

● Research Integrity (9/2008-12/2010)

► New Member Fora

● Medium Sized Research Infrastructures (1/2010-2011)

● Science in Society Relationships (1/2010-9/2012)

● Evaluation of publicly funded Research (4/2010-1/2012)

● Indicators of Internationalisation (11/2009-11/2010)

● Research Careers (...)

Completed, current and new Member Fora






Beispielfolie


ESF Member Fora




► Focus:

“Post-Grant” Evaluation i.e. if the funding schemes or the research programmes achieve their stated aims.

► Objectives

● exchange and document experiences with current practices

● facilitate networking of science officers engaged in evaluation

● explore needs and possibilities for collaboration in future evaluation exercises.

► Participation: 38 Organisations

ESF MOF on Evaluation of Funding Schemes and Research

Programmes – Aims



1st Workshop

► Evaluation of Funding Schemes and Research Programmes: Expectations, Practices and Experiences. Berlin, 22.-23. October 2007 (Host: DFG)

2nd Workshop

► Quantitative Indicators in ex-post Evaluation of Funding Schemes and Research Programmes: Case studies and Rationales. Rome, 31. March -1. April 2008 (Host: INFN)

3rd Workshop

► Quality of Quality Assurance. Striving for Highest Standards in ex-post Evaluation. Vienna, 6.-7. October 2008 (Host: FWF)

4th Workshop

► Evaluating socio‐economic impact of research funding schemes. Budapest, 27.-28. April 2009 (Host: OTKA)

5th Workshop

► Evaluation Strategies and Comparative Evaluation Techniques. Dublin, 16.-17. November 2009 (Hosts: HRB and SFI)


Programmes – Workshops



► Action Items

1. Mapping study on evaluation of funding schemes –finalised

2. A pilot study on international benchmarks – will continue in a new forum

► Final Outcome

1. Report: Evaluation in National Research Funding Agencies: approaches, experiences and case studies

can be downloaded: http://www.esf.org/activities/mo-fora/publications.html


Programmes – Outcome






Beispielfolie


ESF Member Fora




ESF MO Forum Future

► New Forum on Evaluation of Publicly funded Research, approved by ESF Governing Council

● Second Forum „Indicators of Internationalisation“

► Proposing Member Organisations: ● DFG, Germany; HRB, Ireland; NOW, Netherlands; FWF, Austria; FNR, Luxembourg; The

Research Council of Norway; Swedish Research Council; Hungarian Scientific Research Fund.

► Towards good practices and joint evaluation exercises

► Alignment with EUROHORCs/ESF Road Map:

● Chapter 6. Develop common approaches to ex-post evaluation of funding schemes and research programmes

► Strengthen Workshop format



ESF MO Forum on Publicly Funded Research

► First meeting 10th to 11th May 2010 in Stockholm

► three more meetings within the next two years (next meeting Nov. 2010)

► set up three working groups that drafted their work plan:● Assessing the Impact of Funding Schemes on Science and Society

● Ensuring and Improving quality in ex-post evaluation of Funding Schemes and Research Programmes

● Classification systems and Categorization of output data

► Envisaged outputs:● a web-based repository of recent evaluation studies

● guidelines for evaluation – commenting on state of the art

● mapping the evaluation practices/the evaluation function in funding organisations

● overview of impact studies and identification of gaps

● recommendation on possible design of impact studies

● a pilot study on the classification and collection of output data

● a document on steps and pitfalls on the way towards a common classification



Working Group on Classification systems and Categorization of

output data

►Examine feasibility pilot for Health Research Classification System (HRCS) coding of portfolios in FNRS and DFG

►Final reports and data for statistical and evaluative purposes: Structuring final reports helps to gather data that allow to answer statistical questions on topics like

• male/female scientific staff in funded projects

• nationality of scientific staff-members (indicates internationality)

• age of staff-members (indicates „age of a field“)

• collaboration with national and international partners (by country) (indicates

cluster and networking)

• Output (publications, patents, articles in proceedings etc.)

• Qualifications (dissertations of staff members)



Final Reports: Method for achiving structured informations in the

DFG

► DFG-Guidelines name the topics to be mentioned in a final report

► DFG plans to collect these data via a PDF-based standard-form

► Data within these forms will be extracted automatically and transferred into a database

► Some of these data are published via DFG-Internet (e.g. the publication list http://gepris.dfg.de )

► Others will be used only for statistical purposes (e.g. age and nationality of scientific staff members)

Project Status DFG:

►Conceptual considerations on how to and which data to collect are finished

►Considerations on technical solutions (e.g. transfer to database) and legal framework in final stage

►In the beginning of 2011 the standardized collection of final reports of individual grants will be implemented



Method for achiving structured informations in the FWF (Austrian

Science Fund)

► Collection of project output data in the FWF

Source: Collecting and using project output data at the FWF, Rudolf Novak, FWF, Austria



Method for achiving structured informations in the FWF (Austrian

Science Fund) - Publications

ca. 10.000 Scientific Journals

Source: Collecting and using project output data at the FWF, Rudolf Novak, FWF, Austria



Working Group on Classification systems and Categorization of

output data

► Next Steps:

● Produce short paper summarising approaches for the analysis of output data

● Circulate/share details of work ongoing in each research organisation which involves the gathering of information about research outputs/outcomes and impacts.

● Generate recommended “core” set of outputs that could/should be collected by research organisations, plus guidance on definitions etc.

● Plan pilot exercise in which data for “core” outputs is brought together across a number of research organisations.

► Discuss the results of each step in the whole MO Forum

► Contribute to the comparability of Research Funding!



Thank you for your attention!

for further questions please contact:

Anke ReinhardtDeutsche Forschungsgemeinschaft (DFG)-Informationsmanagement-D-53170 Bonn Tel. +49 (228) 885-2588Fax +49 (228) [email protected]://www.dfg.de

Laura MarinScience Officer MOs Relations and PartnershipsCorporate Science Strategy DevelopmentEuropean Science Foundation

Tel.: +33 388 76 [email protected] http://www.esf.org

Activities of the ESF Member Organisation Forum on Research Evaluation

Documents