The Netherlands eScience Center Mid-Term Evaluation · The Netherlands eScience Center Mid-Term Evaluation December 2013 Including: Evaluation Netherlands eScience Center (NLeSC):

www.esciencecenter.nl

The Netherlands eScience Center Mid-Term Evaluation

December 2013

Including:

Evaluation Netherlands eScience Center (NLeSC): Report of

the Midterm Evaluation Committee

Reflection on the Midterm Evaluation of the Netherlands

eScience Center by its Board of Directors

Content

Part 1:

Evaluation Netherlands eScience Center (NLeSC): Report of

the Midterm Evaluation Committee

Part 2:

Reflection on the Midterm Evaluation of the Netherlands

eScience Center by its Board of Directors

Part 1

Evaluation Netherlands eScience Center (NLeSC)

Report of the Midterm Evaluation Committee

- Jan de Jeu (chair) - Wolfgang Gentzsch - Eduard Hovy - Glenn Ricart - Milan Petkovic - Anne Trefethen - Manfred Thaller

December 2013

2

Executive summary

The evaluation committee has been evaluating the Netherlands eScience Center (NLeSC) for

the period of 2011-2013. In this executive summary the main conclusions and

recommendations of its mid term review are formulated.

Is the NLeSC focussing on the right things?

The committee is of the opinion that the quality of the participating groups and scientists in the different

NLeSC projects is excellent. After two years considerable progress has been made with the current

projects portfolio; it is an excellent starting point to build upon the coming years. Now, the following step

is to develop a clear agenda and to follow it. This agenda has to take into account the balance of

different disciplines, the degree to which they work with large data and HPC, and which investments into

other types of ICT enhancement are needed. The committee would also like to encourage the center to

give a stronger support to multidisciplinary and cross-cutting research. In the medium to longer term,

the committee recommends that the strategic effects of funding decisions be more clear, and that future

funding calls be structured to align with the overall vision, and coordinated with the centers

management and Board.

The visibility and network of NLeSC can be improved: there is currently a lack of transparency, of a

clearly articulated long term vision and structural approach towards other organizations. The committee

suggests that a clearer image of the center, its function and the accessibility of the services, and cross

cutting and operational opportunities are needed. Although the outreach to the outside network is

successful (the network surrounding the NLeSC, although still strongly Amsterdam-based, is gradually

enlarging nationally and internationally) the center could be more pro-active in engaging and networking

to parties that are not part of the network yet. This needs more thought and more structure. The NLeSC

should focus the coming period on national visibility. Once this is established international visibility will

follow automatically. The NLeSC should aim to develop a strategy how to propagate possibilities to

potential partners, for example, by using professional marketing and a pro-active attitude toward other

organizations.

Do the center's activities have added value; is this the right model to achieve

the goals in the Netherlands?

The committee is positive about the added value, but there is also room for improvement and there is a

lot of potential to exploit better. When it comes to reusability of software, a stronger vision of the NLeSC

is missing; the committee, for example, is of the opinion that there are opportunities across projects

nationwide for the NLeSC to take a stronger lead to develop relevant techniques, algorithms, models,

and concepts that can be generalized and shared.

The committee is very positive about the work of the engineers. The model is innovative and interesting,

and appears to be producing results. However, there has to be more clarity about their role, how they fit

into the different projects, and the possibilities for their own career inside and outside the center.

Ensuring personal development for engineers who want to go to academia, for example through

mentoring by integrators, is recommended. The committee is not totally convinced about the current

model and work of the integrators: the activities and network of each integrator vary considerably per

person. A clear set of goals and ambitions for the integrators is missing and should be put into place.

Furthermore, the committee urges for a more clear-cut differentiation between the integrator role and

the role of project leader as there might be chances of conflict of interest and non-transparency in roles.

The current governance structure is not transparent; people outside the center do not always understand

the different roles, and the Board is too distant from the center. The current Executive Committee does

not appear to follow a clear plan or exercise clear leadership.

3

How can eScience research and coordination best be implemented and governed

in the short to medium term?

This evaluation recognizes the need for the NLeSC, but it still has to find its place in the Dutch and

international eScience ecosystem. The committee recommends that NLeSC writes down a long-term

vision, strategy, and mission, and then acts on it, for example with regard to balancing the breadth and

depth of the research portfolio. In developing programmes and networks the center should take note of

those areas already being invested in by others. There are opportunities to take a leading role in the

areas of data stewardship and software sustainability, and the committee recommends that NLeSC

focuses the coming years on developing relevant techniques, algorithms, models, and concepts that can

be generalized over disciplines and shared.

It is important to show industry what is being done, and invite industry to participate (in various ways:

providing people or resources or money) in return for benefits (first right to results, training, data, etc.).

It would be good to develop a clear strategy together with industry, placing industry where relevant in

the drivers seat to specify goals and objectives. The idea is that industry then buys into the whole

process, and becomes willing to provide matching funds. However, for certain areas, such as

eHumanities, this is not the model to follow.

The committee recommends NLeSC to focus primarily on its role of funder (in kind and in cash, as is the

case now), and secondarily on being a research center and a marketplace of ideas. The review

committee believes it is possible to achieve all of these seemingly conflicting goals by carefully

considering in which research areas the NLeSC would use a formal research institute status to seek

additional outside funding. It should be able to seek funding when there is a clear and convincing case

that this is in the best interest of the Netherlands and its institutions. The committee recommends a

formal institute status for the NLeSC after these suggested roles are approved by the Executive

Committee and the Board.

Ideally, the committee recommends that the Board be made up of five persons: one representative of

the Dutch scientific community applying eScience methods, one person representing ICT science, one

industry representative, one NWO representative, and one SURF representative. The Advisory Board

should also include representatives from parallel activities/organizations (COMMIT / DANS). The new

executive director must be experienced enough not only in ICT but also in collaborating with at least one

or two other disciplines so that he or she can make outreach and cross-disciplinary work easier.

4

Table of contents

1. Mid-term review

1.1. Background

1.2. Organisation of the review

1.3. Methods and data

1.4. Structure of the report

2. The Netherlands eScience Center 2.1. Background

2.2. Definition of eScience

2.3. Mission of the Netherlands eScience Center

2.4. Organization / governance of the center

2.5. Project funding mechanisms

2.6. Prioritized scientific domains

3. Is the NLeSC focussing on the right things?

3.1. Quality of eScience Portfolio

3.2. Visibility and Network

4. Do the center's activities have added value; is this the right model to achieve

the goals in the Netherlands?

4.1. Value-added premise

4.2. Sustainability and Governance

5. Conclusions and recommendations: how can eScience research and

coordination best be implemented and governed in the short to medium term?

5.1. About coordination of research

5.2. About visibility and network

5.3. About public private collaboration

5.4. About the engineers

5.5. About the governance

5.6. About status of the NLeSC

5.7. Conclusions and recommendations

Annex 1: NLESC-evaluation committee 2013

Annex 2: Terms of Reference for the evaluation committee

Annex 3: Responses to the web survey

Annex 4: Programme of the review

5

1. Mid-term review

1.1. Background

The review of the Netherlands eScience Center (NLeSC ) took place about two years after the setting up

of the physical center in Amsterdam. The review has been requested by the eScience Center Board

during meetings in 2012. The review report of the Netherlands eScience Center was prepared in

November 2013.

The goal of the review is to obtain a comprehensive picture of NLeSC's activities. The review also delivers

an opinion on NLeSCs effectualness, based on the results obtained during the first two years.

Furthermore, the evaluation identifies areas requiring improvement during the remainder of NLeSCs

lifespan and beyond.

The main basis for the assessment is an examination of what NLeSC has achieved over the past two

years. The questions stated in the Terms of Reference provide guidance. The assessment made use of

factual information on the one hand, and opinions and suggestions from national and international

stakeholders on the other. The committee made its recommendations based on all information and

interviews as well as its own expertise.

1.2. Organisation of the review

The review was performed by an independent international committee, assisted by a secretariat. The

international committee consisted of experts in the field of computer science, management of

organisations, eScience policy, and disciplines that make use of eScience. The committee met in

November 2013 in Amsterdam for two days to discuss and prepare the review. The committee members

first attended a symposium on 7 November organised by NLeSC, and on the following day they held

interviews with the various stakeholders. Before and after the physical meeting in Amsterdam telephone

conferences were organised for the committee to discuss the review and the report.

1.3. Methods and data

The review made use of five methods:

1. The gathering of information and data by sending a questionnaire. Each of the various stakeholder

groups received a list of questions (with open and closed questions), based on the questions in the terms

of reference. In total 32 people out of 50 filled out the survey. The main conclusions can be found in

appendix 3.

2. Holding interviews with stakeholders. The stakeholders were briefed beforehand on the set of

questions. The committee interviewed and exchanged thoughts with the following stakeholders: Board of

the NLeSC, the Management Team of the NLeSC, leaders of eScience projects (project leaders), eScience

integrators and leaders of eScience centers and of other related organisations in the Netherlands.

3. The review made use of information on key achievements, publications, personnel, projects and

governance provided by the self-review of the NLeSC.

4. The committee read and analysed the NLeSC's annual reports.

5. The committee attended the first NLeSC symposium.

1.4. Structure of the report

After describing briefly the background and setup of the NL eScience Center (chapter 2), this report

follows the three main evaluation questions that are described in the Terms of Reference:

1. Is the NLeSC focussing on the right things? (chapter 3)

2. Do the center's activities have added value; is this the right model to achieve the goals in the

Netherlands? (chapter 4)

3. Conclusions and recommendations: how can eScience research and coordination best be implemented

and governed in the short to medium term? (chapter 5) Each of these three questions is addressed by answering a subset of questions. At the end of chapter 5

a summary of conclusions and recommendations is given.

6

2. The Netherlands eScience Center

2.1. Background

The Netherlands eScience Center (NLeSC) was launched in 2011 as a collaboration between NWO, the

principle Dutch scientific funding body, and SURF, the Dutch higher education and research partnership

for ICT. The center was initiated as a direct response to the Dutch governments request to develop a

sustainable, coherent and costeffective eScience environment and einfrastructure across all scientific

disciplines.

The Government phrased the assignment as follows:

Found an eScience Center at which the existing universities and institutions cooperate on research

aimed at innovation of ICT infrastructures and scientific applications. This will make it possible to

continue the valuable multidisciplinary collaboration between universities, institutions, and businesses.

The Netherlands will continue to be an attractive location for knowledgeintensive industry and top

researchers. The new center will also raise the international profile of the Netherlands as a country of

open, international science and scholarship, with highly developed connectivity.

The center is located at the Amsterdam Science Park and employs around 30 people (25.3 fte). The

annual budget is 5.4 million euro, provided by NWO and SURF.

2.2. Definition of eScience

eScience is an inherently multidisciplinary pursuit concerned with the need to bridge the gap between

advanced computing and networking on one side and data driven science on the other. The challenge of

eScience is to ensure that the most value is gained from all new scientific endeavors by using

information and communication technology (ICT) to improve experimental design, data management,

data analysis and communication. The ultimate goal of eScience is to enable datadriven science and

multidisciplinary scientific discovery and to challenge traditional research by the creative use of ICT in all

its manifestations. In this respect eScience is defined as enhancedScience.

2.3. Mission of the Netherlands eScience Center

The NLeSC formulated the following mission:

The Netherlands eScience Center reinforces and accelerates multi-disciplinary and data-intensive

research in the Netherlands by developing and applying eScience and by combining forces. The eScience

Center breaks down the barriers between traditional disciplines and ICT technologies. In doing so,

eScience is both a catalyst and enabler.

The eScience Center will ensure that scientific ICT in all its forms is deployed sustainably in support

of the modern research process. The eScience Center develops relevant techniques, algorithms, models,

and concepts that can be generalised and shared. The basic principles are to make research

breakthroughs possible and to generate value for society.

In pursuing its mission, the eScience Center combines a scientific eScience research programme with an

actual physical centre at the Science Park in Amsterdam. This combination is crucial in order to achieve

cross-fertilisation between scientific disciplines and between science and ICT. It also makes it possible to

construct the desired eScience environment for the natural sciences, life sciences, humanities, and social

sciences. The eScience Center functions as a kind of scientific hotel.

The research projects that the eScience Center undertakes in collaboration with institutions serve to

develop relevant techniques, algorithms, models, and concepts that can be generalized and shared. The

starting point for the projects is in all cases the relevant research question.

7

2.4. Organization / governance of the center

The governance and organization is made up of the following groups:

NLeSC Board: The NLeSC Board oversees the operation and direction of the center and consists of members of the two founding organizations, SURF (2 members) and NWO (2 members).

Executive Committee: The eScience Center's CEO, together with his management team of five, is

responsible for its operation, scientific direction and delivery, technology portfolio as well as business

development, communication and community building. The Executive Committee meets weekly and is

joined by external support from NWO for HR issues and operational details related to calls. Notes and

action points are taken from each MT meeting and a summary is verbally communicated to the team of

employees shortly after each MT meeting.

NLeSC integrators: NLeSC has built a team of eScience Integrators. Each of the eScience Integrators is a

leading researcher from a different discipline with a broad experience and understanding of the

possibilities of eScience within his of her domain. eScience Integrators provide a bridge between NLeSC

and their own scientific domains as well as their own academic institutes. The Integrators play a key role,

for example by opening up access to and combining data and creating new types of collaboration

between different disciplines.

eScience Engineers: At the core of NLeSC are the eScience Engineers, a team of broadlyoriented digital

scientists, mostly PhDs, able to work at the interface of their own scientific disciplines and advanced ICT.

The engineers collaborate in multidisciplinary research teams to implement and develop eScience

technologies and datadriven scientific research. All of the current projects are collaborations between

eScience Engineers and project teams working in the traditional academic environments. Most of the

eScience Engineers split their time between working at the eScience Center and sharing ideas and

knowledge with their eScience engineering colleagues, and spending time at the various Universities

working with the domain experts in specific projects.

eScience Advisory Committee: The eScience Center has an independent and international scientific

Advisory Committee (eSAC) that plays an important role in reviewing project proposals and helping

define the scientific strategy of the center.This task is undertaken independently of the NLeSC MT, but

the eSAC can ask for technical input when required. Furthermore, the eSAC encourages new and open

forms of scientific endeavor and building bridges between NLeSC and its applications in business and

industry.

8

eScience Project Leaders: Each project is led by academic researchers who are in frequent contact with

NLeSC management.

2.5 Project funding mechanisms

Different funding instruments, from selected calls to open calls, were developed in the first two years of

NLeSC with the support and using the transparent procedure of NWO. NLeSC also partners with other

organizations, e.g. NWO and SURF, to launch joint project calls. In 2013, for example, NLeSC and

NWOPhysical Sciences conducted a joint call on Data Science which included several mechanisms to

fund collaboration between industry and academia. Approved projects are partly funded in cash and

partly in kind (eScience engineer support).

The following six domains are recommended by the eSAC for support during the next round of funding in

2013/2014: Water Management/Climate Research, Foods, Green Genetics,

Chemistry, eScience Technology and eHumanities.

2.6 Prioritized scientific domains

At the inception of NLeSC, an initial series of eleven scientific priority areas were defined. In March

2012 the NLeSC executive committee, eScience Integrators and scientific advisory board discussed

the previously defined priority areas and agreed to extend the list to include Social Sciences and

Energy. Twelve scientific areas are currently prioritized by NLeSC:

Within Chemistry:

1. Chemistry and materials

Within the Life Sciences:

2. Green genetics

3. Life Sciences & translational research

4. Foods

5. Cognition/neuroscience

Within eScience Methodology & Big Data:

6. eScience technology,

7. astronomy

Within Humanities & Social Sciences:

8. Humanities & Social Sciences

Within Sustainability & Environment:

9. Climate research

10. Water management

11. Ecology

12. Energy

The selection is stimulated by the current Top Sectors defined by the Dutch government, but is dynamic

and will develop over time.

9


3.1. Quality of eScience Portfolio

The NLeSC currently has two different sets of projects:

a. Full Projects

The full projects are supported to the value of K500 and include the deployment of an eScience

engineer. Project proposals are prioritized by NLeSCs scientific advisory committee and finally

sanctioned by NLeSCs board of directors. In the period of 2011 - 2013 a total of 21 projects have been

initiated.

b. Pathfinding Projects

In addition to the major projects, NLeSC sponsors smaller eScience initiatives, referred to as "Path

Finding" projects, which are intended to provide NLeSC with the opportunity to rapidly meet short-term

scientific challenges, address immediate technological goals or investigate the potential to initiate full

projects. Each project receives K50 funding or FTE 0.5 eScience engineer support.

ToR question: Does the projects portfolio represent world-class applications of eScience?

(For example by looking at scientific originality, breadth of subject, likelihood of delivering scientific

breakthroughs, depth of eScience involved and the long-term value of collaborations.)

Next to the self-evaluation, the answers to the web survey and presentations in the symposium, the

committee has based its opinion on this matter through interviews with the project leaders and the

eScience integrators. The respondents to the web survey are in majority positive about the portfolio of

projects. Nonetheless, a few questions were raised, such as How could synergy between the different

projects be improved? and Is the balance between the different scientific disciplines right? The center

currently serves mainly the exact sciences.

The committee attended the first symposium organized by the NLeSC and is of the opinion that the

scientific achievements of the participating groups and scientists in the projects of NLeSC is excellent.

The committee is impressed by the quality of participating groups and scientists. After two years

excellent results have already been achieved.

Examples of answers to the web survey

Project leader: A significant number of scientific areas that will depend on eScience technology for scientific

breakthroughs in the future are covered by the NLeSC portfolio. It includes a strong variety of disciplines.

eScience engineer: I think the portfolio can be further improved by taking synergy between projects into account

when accepting new projects. Currently there are several projects that do not 'fit in'..

Project leader: The NLeSC currently services mainly the exact sciences, even though most physicists, chemists and

biologists know very well how to operate a computer. Possibly their service would be better aimed at the social

sciences, humanities, where computing is not as well established.

There is, however, according to the committee a significant imbalance in the sizes of the disciplines

supported. The social sciences (law, economy, psychology, etc.), for example, are underrepresented.

Furthermore, the humanities is being considered as a single area, which is like regarding all the hard

sciences as a single discipline. While it is appropriate to target areas in which near-term successes are

more likely, it is also important to diversify within representationally complex areas such as are found in

the humanities. In addition to language technology, areas with large data availability are film studies

and communications, history, economics, and sociology.

The committee has discussed the open calls and the more focused calls with the integrators and project

leaders, and has concluded that the pattern of open calls and focused calls is an interesting mix. The

process of writing and selecting topics for the calls, however, is not clear. The integrators, for example,

10

are not involved in the strategy and writing of the calls.

The integrators, during the interview, stressed that a better match between new projects and existing

software solutions should be looked for; the wheel should not be reinvented. The committee also

discussed multidisciplinary or cross cutting research with the integrators and engineers. Currently there

is no pro-active support or mechanism for multidisciplinary research. Multidisciplinary research often is

not funded because of the difficulty to find reviewers that have the right knowledge about the bridge

between two or more disciplines. The bridge to other disciplines and the reusability in different domains

remains difficult.

Emerging domains in Big Data, in which the Netherlands can lead the way for Big Data, should be

explored and used according to the committee. Big Data continues to be a significant area of opportunity

for eScience Center leadership nationally and internationally. In addition to Big Data projects in the style

of the existing portfolio, the Center should encourage work in emerging domains of Big Data,

opportunities for cross-disciplinary Big Data work, and projects which leverage interrelationships between

multiple synergistic datasets. The committee also stresses the need for a clearly defined vision: a

development and articulation of a longer term vision and strategy, specifically with regard to the balance

of breadth and depth of the research portfolio.

3.2. Visibility and Network

a) National

ToR question: Is NLeSC recognised as the driving force and primary center of excellence in the

application of eScience at a national level? Is NLeSC able to speak with authority on behalf of the Dutch

eScience community? Does the Dutch eScience community have a voice in NLeSC, and feel able to

influence its direction? Is NLeSC recognised as the central platform within the national eScience

community?

The respondents to the web survey are generally positive about this. The website, annual report and

other publications are excellent ways of communication and the first annual NLeSC symposium may be

called a success by bringing 300 researchers in the field of eScience in the Netherlands together. On the

other hand, a relative large number of people, that might get involved, dont know the center and / or its

resources. This is the conclusion of the committee after analyzing the answers to the web survey and

talking to the interviewees.

The committee is of the opinion that the visibility and network of NLeSC is at a sufficient level given the

short time that the center exists, but must be improved in the coming period when it comes to the

nature, role, internal workings of the center, and about modes of potential collaboration. The center has

not performed enough outreach to various potential allies and institutions about these issues, yet. As a

result, the Dutch research community is not clear about the nature, role, and internal workings of the

center, and about modes of potential collaboration. A clearer image of the center and its function and the

accessibility of the services is needed. There should, related to this, be more transparency of what the

center does: more information about the cross cutting and operational opportunities.


eScience integrator: Always room for improvement and again, although it is a bit early to tell how authoritative the

NleSC is or will become nationally and internationally.

Member of international advisory board: Might be true at national level but not international but insofar as I get the

drift, I think there are still many people who should know about the NLeSC who do not.

In particular, interviews with representatives from DANS, KNAW and COMMIT indicated the potential for

11

collaborations, earlier attempts to establish connections from these organizations to the NLeSC, and

general eagerness to share in work and results. However, there has been no action of the part on NLeSC

to respond to these requests, which has reinforced in some circles (not only among the people

interviewed, but generally within the country) that NLeSC is a rather Amsterdam-centered enterprise. In

the long term, this perception can be very damaging to the center.

b) International

ToR question: Are policy makers at the European level aware of NLeSC's work? Are partners in other

European countries actively identifying NLeSC as a potential research partner? Is NLeSC representing the

Netherlands at international eScience conferences? Does the international eScience community recognise

NLeSC as a point of contact for the Netherlands? How is the NLeSC performing or operating compared to

other similar international initiatives?

Currently the NLeSC is in touch with a number of international colleagues and related organizations

abroad, and the NLeSC has ambitions to work closely together with a number of related initiatives

abroad. But this is still in the process of discussing things and attending meetings. International

cooperation is hindered at the moment by the status of the NLeSC. The NLeSC has to obtain the approval

by the European commission to participate in European projects because it is currently not regarded as a

knowledge institute. The respondents to the web survey and the people the committee talked to are

divided in opinion on the question if the center should be able to apply for funding in Europe. Some think

that this should be encouraged and facilitated, others fear they will compete with researchers at Dutch

knowledge centers / universities. A suitable agreement might be that the NLeSC plays a key role in

European funding when the focus is around infrastructure and there is a need to have a national lead.

The rather unusual circumstances and operation of the NLeSC should be exploited to work together with

researchers nationwide to seek funding in ways that the researchers themselves cannot do in their

current situations.


Project leader: in my opinion it is relatively early for this claim/statement of international visibility. In

the Netherlands context NLeSC this is already more true than in international context is my estimation.

Project leader: I think the Center is too young to be able to claim such a very high international profile.

Respondents to the survey all agree that the center is too young to be able to claim a high international

profile. The committee agrees with this. The committee is under the impression that the international

visibility and network of NLeSC has to be improved coming years, but that national visibility is a priority

and that national visibility will also lead to international visibility.

c) Within Industry

ToR question: Is NLeSC developing partnerships with commercial organisations? Is Dutch industry

aware of NLeSC and its work? Is NLeSC seen as a serious potential partner by commercial organisations?

Is the NLeSC model recognised as innovative and valuable?

The last two years a number of industrial workshops have been organized by the NLeSC. This is a good

way to move forward and to develop networks within industry. The management team has indicated that

they want to develop strategic partnerships with industry the coming years and that a business plan will

be ready shortly.

12


Project leader: I am not aware of any collaborative effort between Dutch industry and the NLeSC

Some more work in coordinating with Industry would be beneficial

Industrial contact: For my sector, this has only recently become clear, but this is mainly the responsibility of the

sector itself.

The committee is of the opinion that the visibility and network in industry of NLeSC is currently relatively

weak. The committee is of the opinion that a lot of more work can be done and that there are ample

opportunities in the long run.

Some respondents to the web survey are critical about the relation to industry, but also point out that

the center is relatively new and deserves time to become more visible. A sense of business like thinking

has to be developed. Public private partnerships are often not easy to establish, especially if private

partners have to bring in cash. In kind participations are generally easier to establish. The center could

therefore also focus on a broker function, bringing researchers and industry together in eScience fields.

4. Do the center's activities have added value; is this the right model to

achieve the goals in the Netherlands?

4.1. Value-added premise

In the intentions formulated at the start it was stated that NLeSC should 1) on a science and problem

driven basis develop proof of concepts to support and accelerate scientific domains; and 2) strongly

collaborate with institutes, academic and commercial partners and with other entities relevant to NLeSC's

portfolio of activities, such as SURF, SURFsara, SURFnet, COMMIT and TARGET, for a solid

implementation throughout the entire innovation chain.

ToR questions: Does NLeSC provide added value by (I) preventing duplication, (II) supporting careers

of eScientists, (III) identifying and exploiting synergies, (IV) coordinating eScience activities, (V)

providing vision for the future of eScience and (VI) is NLeSC contributing sufficiently to the goal of

developing the Netherlands as a leader in the field of eScience? How does NLeSC's development to date

compare to the intentions formulated above? How is the balance of in-cash and in-kind contributions

currently assessed in NLeSC projects?

The committee has learned, not only from the documentation, but also from the discussions during the

interviews, about the value-added premise. The respondents to the web survey are in majority positive

about the added value premise and the committee agrees. There is also criticism: a person, for example,

wrote: So far the NLeSC has not been very effective in coordinating science activities. They seem to be

very much trying to define their own existence. This demand for a stronger vision has been heard also

during the interviews and seems to be a common theme / concern in all aspects that are discussed in

this report.


Project leader: In my opinion the aim of preventing duplication in source code is an illusion. In practice researchers

tend to (re)write code rather than use the codes written by another, and for a good reason; researchers like (and

have) to know in detail how the engine works and where it fails.

Project leader: So far the NLeSC has not been very effective in coordinating science activities.

Project leader: I think the eScience center is setup in a good way that it can indeed provide these added values..

Project leader: It could take a more active role in training MSc and PhD students to use open source and open data,

as science jobs may move more to data analysis rather than data acquisition.

13

An important part of the value of the NLeSC is to develop relevant techniques, algorithms, models, and

concepts that can be generalized and shared1 across projects and disciplines. Furthermore, a key aim2 is

To develop versatile and effective eScience tools. The eSTeP project is one effort in this direction that

the committee applauds. The review committee believes that these are important aims of the center and

should be fully embraced, supported, and publicized. The techniques, models, concepts, and tools should

complement those already available from other sources and normally be generated in response to

support of a specific discipline project and they should be developed in a way that they can be also

applied to other projects both within and without the eScience Center. Tools should normally be rolled

out in partnership with and transferred to SURFsara and/or commercial partners.

The link with relevant parties could be better is the general feeling within the committee after talking to

the various stakeholders, like DANS, KNAW, TARGET and COMMIT. The NLeSC still has to find its place in

the ecosystem and needs to open up and search for cooperation opportunities. Most respondents to the

web survey and people that were interviewed state there certainly is, currently and for the years to

come, a need for the NLeSC, and that there is much potential for added value.

The connection between the NLeSC and the other relevant organizations / parties seems lost at this

moment and needs to be more structural. There have been talks but nothing has come out of it.

Moreover, the relation with ICT research is weak and might cause problems in the longer run because

new techniques and solutions might not be used. There is a unique opportunity to link all ICT research

disciplines with other scientific disciplines. Recently an eScience platform has been established by NLeSC

in which the expertise of Dutch universities is bundled, and this is an excellent initiative according to the

committee. The management team during the interview mentioned the strong connection with computer

science and collaboration with ICT groups. The committee, however, still thinks that the bridge between

the ICT departments and the NLeSC could and should be stronger.

4.2. Sustainability and Governance

a. Sustainability

ToR question: Does the center provide an adequate model for the long-term stewardship of its own

developments, personnel and projects? Is NLeSC participating in long-term collaborations? Does the

center play a sufficiently active coordinating role in identifying opportunities and solutions for long-term

sustainability and stewardship of scientific software and data at a national level?

The respondents to the web survey in general are positive about this aspect, but also stress that a long-

term vision on sustainability needs to be developed. The committee after consulting the different

stakeholders during the interviews is not totally convinced about the model of sustainability: there is a

certain risk because a long-term vision is missing. Sustainability of the center can only be achieved with

a long-term plan and successful execution that then leads to wider acceptance within the science

community and its funding bodies. In this context, it is very important to find a CEO, who is widely

recognized as a leader and able to drive the development of such a vision.

1 Annual Plan and Budget 2011, page 5, Vision and mission, paragraph 2 2 ibid, Aim #3

14

Examples of answers to the web survey:

ESAC member: Strategy on a 10-year timescale is necessary for a growing and important area such as this, and the

timescale is currently limited to about three years.

Project leader: A long term plan seems to be missing completely. There is no stewardship or long term sustainability

of scientific software or data preservation. The NLeSC currently lacks the hardware and expertise for this. In my

opinion, the long-term sustainability of source code and the long-term storage of (simulation) data would be an

excellent task for the NLeSC.

Integrator: Given the increased relevance of Big Data and Datasharing and the fact that in most places the necessary

expertise for these endeavours is absent, there could be an important role for eScience to support and setup an

infrastructure that will allow external partners to conduct studies to generate large amounts of data, and focus on the

science without worrying about the data challenges. Perhaps it could invite universities to become partners in the

center.

There is a chance, it is even likely according to the committee, that there will be no role for the center in

ten years time. All research institutes by then should be using eScience in research, have their own

eScience centers, and therefore would not need a national center. How does the NLeSC leadership

respond to this claim? How would this influence the development of a vision for the future?

Some respondents argue that the center does not have the right expertise and resources at the moment

in the fields of scientific data and software stewardship. The committee, however, acknowledges the

report Data-Stewardship in the Big Data Era: Taking Care of Data and encourages the NLeSC to follow

up on it.

b. Personnel

ToR question: How appropriate is the staff policy? Is the model of the eScience engineers and the

eScience integrators the right model to achieve results?

The respondents to the web survey in general are positive about this aspect, but there is also some

criticism. The main critic is that the roles are not clear, there needs to be serious rethinking of the roles

especially when it comes to the relation between the project leaders, the NLeSC MT and the integrators.

How could or should the unclear relations in the projects be solved?

The stakeholders, during the interviews, were positive about the model and work of the engineers. The

model is innovative and interesting, and appears to be producing results. There has, however, to be

more clarity about their role in the different projects and their career inside or outside the center.

Training remains important and summer schools, for example, could be organized.

The work of the engineers and integrators should be made clearer to the public. These roles seem to

work very well for the center. It is most important to actively support the work and the careers especially

of the eScience engineers (who are key to the centers long-term success) by offering additional

incentives to publish, to present, and to collaborate with other groups beyond their existing project.


Project leader: There is confusion in the role of the eScience engineers and especially in the governance. The projects

are complex in many ways and it does not help if there is no clarity on who is supervising what. These aspects require

serious rethinking of the various roles.

ESAC member: More work needs to be done to progress career and esteem of e-science engineers especially in

University settings where promotion is often tied almost solely to publications.

15

Engineer: The role of integrators is not clear, there is no direct communication to NLeSC staff.

Engineer: I think the model works well. I think that the synergy between projects would improve if projects could

switch eScience engineers during the project's lifetime, depending on the expertise the project needs that point in its

development.

After having spoken with the integrators, the committee is of the opinion that the center needs more

integrators, and their division per disciplines needs to be better (e.g., one for humanities and social

science is not enough). The integrators can help to create more visibility in their domains (some of them

have more time for this than others). Another important aspect that was discussed during the meeting is

that the integrators only met once with the ESAC and would like to have more meetings because they

felt that this was very useful.

c. Research (funding)

ToR question: Should NLeSC develop its own research or research programme? Does NLeSC have a

viable plan for attracting new sources of funding and developing the eScience of the future? What options

exist? Are the conditions fulfilled to make use of these options? Should the center be able to attract

funding through competitive rounds from NWO or Horizion2020?

The respondents to the web survey are divided in their opinion about the issue of research funding.

Some indicate that it is a natural step for the NLeSC to do research and attract funding for this, others, a

majority, see a danger in NLeSC competing for research funds because it might obstruct or get in the

way of other Dutch research organizations, and because they regard the NLeSC principally as a

facilitating center.

The review committee carefully considered the complex issue of whether the NLeSC should be given a

formal research institute status. Presently it does not have the status of a knowledge / research institute

and it is not permitted to apply for funds in the Netherlands and the European Union. The most

important argument for such status is to allow the Center to participate in proposals and calls by the EU

and in the Netherlands and otherwise engage in joint activities with its projects. This would also benefit

the academic-track eScience engineers who should be developing a record of publications during their

time at the eScience Center. The most important argument against formal institute status is that it might

put the NLeSC into competition for research funding with its own customers.

As soon as NLeSC competes for funding with other NL research constituencies, it is highly probable that

the support for NLeSC from the wider Netherlands research community will cease and that the

sustainability of the center will become endangered. The committee is of the opinion that the NLeSC can

only perform research in areas which are strictly complementary to existing science organizations or in

strong collaborations with key constituencies. In particular, there will be occasions when integrators and

other researchers may not be able to propose effectively due to organizational or personnel constraints,

but the NLeSC can do so. In especially such cases, collaborative proposals would be ideal.

16


ESAC member: I think the NLeSC is in an excellent position to now attract European funds especially as the projects

are now started and coordinated.

I am not certain that NLeSC should develop its own research programme...or whether it should connect to existing

pan-European programs.

The NLeSC was setup to be an innovative service organisation to assist researchers. If researchers see an added

value, they will incorporate it in their project plans and reserve budget. NLeSC should not compete with the very

people whom they should support, i.e., the researchers.

Not sure, role of eScience Center might change if they go into competition with Universities (etc.) to attract funding

for projects/programmes. In my opinion NLeSC has now a more neutral role, which enables different mode of

cooperation. However, it is clear that NLeSC does need funding....

Once NLeSC develops its own research program it will be in competition with the eScience community in The

Netherlands. That will weaken its current role as facilitator and central point of contact.

NLeSC should not do research but serve research.

The Center should facilitate research groups in academia, not compete with them.

d. Governance / Organization

ToR question: Is the current set up of the governance/organisation appropriate for the center? What is

the ideal composition of the NLeSC Board? Should the center become an institute or foundation with its

own financial administration?

The respondents to the web survey in general are quite clear on the issue of becoming an institute: the

center should become an institute as an independent legal entity to be able to operate independently.

The question remains however what will be the focus of the institute: should this be research or

facilitating? (see above). The interviewees also stress that the center should become an institute with

more freedom to set its own goals.


Hard to judge, but I have the feeling the NLeSC management is sometimes unnecessarily slowed down due to the lack

of own legal entity.

The center should become an institute.

The center should become an independent foundation truly dedicated to facilitating science.

The board should give a much stronger mandate to the director. Dependent on such a decision will be my view on

how the structure of the center should become in the future.

The majority of the respondents to the web survey and interviewees think that the composition of the

Board needs to be reconsidered. The Board needs new blood, as one interviewee pointed out. It is well

understood that NWO and SURF are dominantly represented on the Board. But after 2 years of

operations, with the startup phase now successfully done, that the Board needs a broader direction and

representation from additional stakeholders in the community.

After having spoken with the stakeholders there are also concerns within the committee about the role of

the ESAC. This committee exchanges views with the executive team but not with the integrators and the

board. Therefore, it seems, that they are somewhat isolated and that the chain of information is broken.

17

5. How can eScience research and coordination best be implemented and

governed in the short to medium term?

5.1. About coordination of research The committee is of the opinion that the center made an excellent start. But now it is time to devote

more attention to creating an overarching vision and defining the intermediate- and longer-term

strategy. The committee considers it important that a balance be maintained between vertical domain-

oriented enrichment and horizontal cross-disciplinary technology development and sharing. For the

former, the committee suggests two effects over time: (1) bigger and richer results as the enhanced

science methods are applied in each discipline separately, and (2) methodological changes within them,

as the power of the enhanced science permeates individual research areas through the activities of the

integrators, acting as champions. For the latter, the committee would like to see new cross-cutting

results as information and techniques from one domain becomes available and are used in another.

As is natural at the outset, somewhat more emphasis has been placed so far on the vertical. The

committee recommends that in new calls for proposals, specific requests should be made for innovative

cross-disciplinary collaborations. Very important in making this work is to select proposal review

committees that understand the value of both (or more) sides of such proposals, and that do not require

traditional advances in any single discipline. At the same time, the funders should not impose too many

procedural requirements on new projects, but allow them to enter into the intellectual marketplace of the

NLeSC in a natural way and begin to share in and contribute to the engineer culture already present.

In the medium to longer term, the committee recommends that the strategic effects of funding decisions

be more clear, and that the calls details be aligned with the overall vision and coordinated with the

Centers management and Board.

5.2. About visibility and network First, the center should develop a clear strategy for networking, by informing relevant people and

meeting with potential partners at universities and government institutions across the country. The

center should no longer have the perception of being an Amsterdam in-group. Secondly, the Center could

support symposia (led by integrators in various fields). More integrators need to be brought in for

maximal dissemination of results, for example through workshops in different disciplines.

The committee also recommends that the participating members, specifically the integrators but also

other similar institutions, in the academic disciplines closest to NLeSC activities consider establishing

educational programmes at their universities, perhaps in collaboration with their Computer Science

departments. Focusing on the senior bachelors and masters levels, such programmes can interoperate

with and/or complement summer schools held by the engineers and integrators, and range from simple

courses in tool usage to more advanced ones in R&D methods in enhanced science in general.

5.3. About public private collaboration

Visibility within the industry seems to be too low, with respect to the mission of collaborating with

industry. This should be intensified, especially by initiating specific industry-oriented cooperation

activities that take the industrys eScience requirements into account.

The committee is of the opinion that NLESC could take a broker function, bringing researchers and

industry together: show industry what is being done, and invite industry to participate (in various ways:

providing people or resources or money) in return for some benefits (first right to results, training, data,

tools, etc.). Organize, for example, an Industry day for CEOs where you can affiliate possibilities. Invite

suggestions from industry about which kinds of projects and directions to address in new work.

18

In conclusion: develop a clear strategy together and with industry and in which the industrys voice is

clearly heard.

5.4. About the engineers Recognizing the crucial role of the engineers, the committee recommends that a variety of career path

alternatives be made available to them. For those oriented toward academia, the committee

recommends mentoring by one or two appropriate innovators. For those wishing to perform this function

long-term, the committee recommends creating status elevations, such as Data Scientist and other roles,

which may come with mini-sabbatical tours in research and industry labs every few years. For those

wishing to accompany their results into industry, the committee recommends appropriate support for

industry contact and training. In some cases, engineers may come from the domain institution and be

funded primarily from the outside.

The committee also recommends that the NLeSC investigates the feasibility of starting a culture in which

early-career academics and senior graduate students take a tour of duty among the engineers. Their

work in the center should be considered a prestigious academic rotation, with opportunities for

publications and scholarly recognition.

5.5. About governance

Currently the Board is too narrow, too divorced from the day-to-day operations of the Center, and too

distant from the Centers management. This is the conclusion of the committee after consulting the

stakeholders. The committee recommends to broaden the composition of the board. Ideally the board

should consist of representatives of NWO, SURF, the Dutch scientific community (a researcher who

applies eScience methods and an ICT-researcher) and an industry representative (for example a CEO of

a company that uses eScience on a large scale).

The advisory committee also needs a better representation of the different stakeholders. The committee

recommends the addition of relevant people in parallel and related activities, for example DANS and

COMMIT.

The committee recommends that the new Director be an ICT person, or someone with significant ICT

expertise, who also has at least one other scientific competency, and who is familiar with KNAW and

NWO.

5.6. About status of the NLeSC

As part of the vision, the Center should define itself more clearly. Three alternatives have been mentioned:

- A research center with its own researchers

- A funder that provides funding to projects closely aligned with integrators and realized by center

engineers who work with them

- An ideas marketplace that facilitates information and data exchange

The committee believes the Center should adopt all of these roles, but should focus largely on the second

one.

More specifically, the NLeSC should be limited to seek funding (through its research institute status) that

benefits its multidisciplinary eScience techniques, algorithms, models, software tools, training, and

advances in eScience infrastructure. Since these techniques, algorithms, models, software tools, training,

and advances in eScience infrastructure only should be developed in response to a specific project or

19

national initiative, the NLeSC should seek funding only in partnership with at least one other Netherlands

research institute. The NLeSC board should be empowered to make exceptions to these rules when there

is a clear and convincing case that an exception is in the best interest of the Netherlands and its

institutions as a whole. For example: an EU-wide call to support eScience Centers.

Any techniques, algorithms, models, software tools, training, and advances in eScience infrastructure

should normally be made widely available and not limited to eScience Center projects. Advances in

infrastructure, in particular, should normally be made available more widely through SURFsara.

Under conditions like those described above, the committee recommends formal institute status for the

NLeSC.

5.7. Conclusions and recommendations

The mid-term review has focused on three main evaluation questions. Please find below the main

conclusions and recommendations of the committee.

a. Is the NLeSC focussing on the right things?

Conclusions:

The quality of participating groups and scientists in the different NLeSC projects is excellent. After two

years considerable progress has been made with the current projects portfolio; this is an excellent

starting point to build upon the coming years.

The visibility and network of NLeSC can be improved. There is currently a lack of transparency, of a

clearly articulated long-term vision and structural approach towards other organizations. The committee

concludes that a clearer image of the center, its function and the accessibility of the services, and cross

cutting and operational side is needed.

The outreach to the outside network is only partially successful: the network surrounding the NLeSC is

gradually enlarging nationally and internationally. Still, the center could be more pro-active in

communicating and networking towards new partners.

The eScience Technology Platform (eSTeP) is considered by the committee to be an excellent initiative

in establishing cohesion between the programs and networks.

Recommendations:

The open calls and focused calls should be continued, and the committee would like to encourage the

center to give a stronger support to multidisciplinary and cross-cutting research by giving preference to

multidisciplinary proposals.

Emerging domains in Big Data in which the Netherlands can lead the way should be explored and used.

There is currently no longer-term vision. A vision must be developed and articulated of longer term

vision and strategy, specifically with regard to balance of breadth and depth of research portfolio. A clear

agenda has to be developed and followed. This has to take into account the balance of different

disciplines, the degree to which they need large data and large computation, investments into other

types of ICT enhancement.

In the medium to longer term, the committee recommends that the strategic effects of funding

decisions be more clear, and that the calls details be aligned with the overall vision and coordinated with

the centers management and Board.

b. Do the center's activities have added value; is this the right model to achieve the goals in

the Netherlands?

Conclusions:

The committee is positive about the added value, but there is also room for improvement. A stronger

vision of the NLeSC is missing; the committee is of the opinion that there are opportunities to take a

stronger lead to develop relevant techniques, algorithms, models, and concepts that can be generalized

20

and shared.

The committee is positive about the model of the engineers. The model is innovative and interesting,

and appears to be producing results. There has to be, however, more clarity about their role, how they fit

in the different projects and the possibilities for their own career inside or outside the center.

The committee is not totally convinced about the current model and work of the integrators: the work

and network of each integrator differs considerably per person. A clear set of goals and ambitions for the

integrators is missing.

The current governance structure is not transparent; for people outside the center the different roles

are not clear, and the Board is too distant from the center.

Recommendations:

The committee recommends that NLeSC writes down a strong vision and mission and acts on it.

The committee recommends that NLeSC focuses the coming years on developing relevant techniques,

algorithms, models, and concepts that can be generalized and shared.

The committee recommends NLeSC to focus on the role of funder (in kind and in cash) and secondarily

be a research center and a ideas marketplace. The review committee believes it is possible to achieve all

of these seemingly conflicting goals by carefully conscribing the areas in which the NLeSC would use

formal research institute status to seek funding.

The committee urges for a more clear-cut gap between integrator role and project leader as there

might be chances of conflict of interest and non-transparency in roles.

Ensuring personal development and mentoring by integrators for engineers who want to go to

academia is recommended.

The integrators role needs to have a more structural approach with clear goals and targets.

The Board should ideally be made up of five persons: one representative of the Dutch scientific

community applying eScience methods, one person representing ICT science, one Industry

representative, one NWO representative, and one SURF representative.

The Advisory board should also include representatives from parallel activities/organizations (for

example, COMMIT / DANS).

The new director must be experienced enough in not only ICT but also in collaborating with at least

one or two other disciplines so that he or she can make outreach and cross-disciplinary work easier.

c. How can eScience research and coordination best be implemented and governed in the

short to medium term?

Conclusions:

- The committee concludes that there is a need for the NLeSC, but it has to find its place in the Dutch

and international eScience ecosystem.

- It is important to show industry what is being done, and invite industry to participate (in various ways:

providing people or resources or money) in return for benefits (first right to results, training, data, etc.).

Recommendations:

- The committee recommends that NLeSC writes down a long term vision, strategy and mission and acts

on it.

- The committee recommends broadening the composition of the Board.

- The committee recommends that the NLeSC becomes an independent institute.

- The committee recommends NLeSC to focus on the role of funder (in kind and in cash, as is the case

now) and secondarily be a research center (when there is a clear and convincing case that this is in the

best interest of the Netherlands and its institutions as a whole) and an ideas marketplace.

21

Annex 1: NLESC-evaluation committee 2013

The NLeSC board has appointed a committee consisting of six members and a chair. Between them, the

members of the committee obtained a good picture of the Dutch context and the international position of

the Dutch research and information domain in the relevant area. Neither the members of the committee

nor the chair had any direct involvement. The secretary of this evaluation committee was provided by

NWO.

The composition of the committee has been as follows:

Chair

- Drs. Jan de Jeu (Chair), Vice President of the board of the University of Groningen

Members

- Prof. Wolfgang Gentzsch, Senior HPC Consultant for Governments & Industry

- Prof. Eduard Hovy, Professor at the Language Technologies Institute of Carnegie Mellon University

- Dr. Glenn Ricart, US Ignite - Next Generation Apps for America

- Prof. Milan Petkovic, Head of eScience department, Philips Research Europe

- Prof. Anne Trefethen, Professor of Scientific Computing and Chief Information Officer (CIO), Oxford

University

- Prof. Manfred Thaller, Professor in Computer Science for the Humanities, University of Kln

Secretariat

- Drs. Sjoerd Meihuizen (Secretary), Policy officer, Netherlands Organization for Scientific Research

(NWO)

22

Annex 2: Terms of Reference for the evaluation committee

The committee evaluated the NLsSC on the basis of all the reporting documents (covering primary

activities, subsidy programmes and networks), including self-evaluations and external evaluations as

appropriate. The committee basis its review and evaluation on the following main questions:


2. Do the center's activities have added value; is this the right model to achieve the goals in the

Netherlands?

3. How can eScience research and coordination best be implemented and governed in the short to

medium term?

The subset of questions, in order to answer the 3 main questions, is the following:

Quality of eScience Portfolio

a. Full Projects

Determined by scientific originality, breadth of subject, likelihood of delivering scientific breakthroughs,

depth of eScience involved and the long-term value of collaborations. Does the Full Projects portfolio

represent world-class application of eScience?

b. Pathfinding Projects

Determined by their scientific originality, likelihood of supporting scientific breakthroughs and potential to

develop reusable eScience technologies and initiate new long-term commitment. Does the Pathfinding

Projects portfolio represent world-class application of eScience?

Value-added premise

Does NLeSC provide added value by (I) preventing duplication, (II) supporting careers of eScientists,

(III) identifying and exploiting synergies, (IV) coordinating eScience activities, (V) providing vision for

the future of eScience and (VI) is NLeSC contributing sufficiently to the goal of developing the

Netherlands as a leader in the field of eScience?

Scope: in the intentions formulated at the start it was stated that NLeSC should 1) on a science and

problem driven basis develop proof of concepts to support and accelerate scientific domains; and 2)

strongly collaborate with institutes, academic and commercial partners and with other entities relevant to

NLeSC's portfolio of activities, such as SURF, SURFsara, SURFnet, COMMIT and TARGET, for a solid

implementation throughout the entire innovation chain.

In terms of scope, how does NLeSC's development to date compare to the intentions formulated above?

How is the balance of in-cash and in-kind contributions currently assessed in NLeSC projects?

Visibility and Network

a. National

Is NLeSC recognised as the driving force and primary center of excellence in the application of eScience

at a national level? Is NLeSC able to speak with authority on behalf of the Dutch eScience community?

Does the Dutch eScience community have a voice in NLeSC, and feel able to influence its direction? Is

NLeSC recognised as the central platform within the national eScience community?

b. International

Are policy makers at the European level aware of NLeSC's work? Are partners in other European

countries actively identifying NLeSC as a potential research partner? Is NLeSC representing the

Netherlands at international eScience conferences? Does the international eScience community recognise

NLeSC as a point of contact for the Netherlands? How is the NLeSC performing or operating compared to

other similar international initiatives?

c. Within industry

Is NLeSC developing partnerships with commercial organisations? Is Dutch industry aware of NLeSC and

its work? Is NLeSC seen as a serious potential partner by commercial organisations? Is the NLeSC model

recognised as innovative and valuable?

23

Sustainability and Governance

a. Sustainability

Does the center provide an adequate model for the long-term stewardship of its own developments,

personnel and projects? Is NLeSC participating in long-term collaborations? Does the center play a

sufficiently active coordinating role in identifying opportunities and solutions for long-term sustainability

and stewardship of scientific software and data at a national level?

b. Personnel

How appropriate is the staff policy? Is the model of the eScience engineers and the eScience integrators

the right model to achieve results?

c. Research (funding)

Should NLeSC develop its own research or research programme? Does NLeSC have a viable plan for

attracting new sources of funding and developing the eScience of the future? What options exist? Are the

conditions fulfilled to make use of these options? Should the center be able to attract funding through

competitive rounds from NWO or Horizion2020?

d. Governance / Organisation

Is the current set up of the governance/organisation appropriate for the center? What is the ideal

composition of the NLeSC Board? Should the center become an institute or foundation with its own

financial administration?

24

Annex 3: Responses to the web survey

The web survey consisted of open-ended and closed-ended questions. The web survey was sent to 50

people. Thirty two people responded: 7 project leaders, 4 ESAC members, 7 engineers, 5 colleagues

from related organizations in other countries, 3 integrators, 2 colleagues of other Dutch organizations, 2

industrial representatives and 2 members of the preparatory committee.

Below are the responses to the closed-ended questions. Some of the responses to the open questions

have been incorporated in the text above.

28

Annex 4: programme of the review

Programme NLeSC Review, 7-8 November 2013

Locations: NH Barbizon Palace, Amsterdam, and the Rosarium (Amstel Park)

7 November, The Rosarium

9.30 17.00: 1st annual NLeSC Symposium, Optimising Discovery in the Era of Big Data

17.45 18.15: Opening by chair

18.15 19.30: Working Dinner in Amsterdam with NLeSC Board

8 November, NH Barbizon Palace

9.00 Third group to be interviewed: 4 eScience Integrators

- Prof. Henri Bal (VU University Amsterdam) Computer Science

- Prof. Antal van den Bosch (Radboud University Nijmegen) Example-Based Models of Language

- Prof. Nick van de Giesen (Delft University of Technology) Water Resources Management

- Dr Jeff Templon (NIKHEF) Grid Computing

10.30 Fourth group to be interviewed: 4 leaders of NLeSC projects

- Prof. Guus Schreiber (University of Amsterdam) BiographyNed

- Prof. Willem Bouten (University of Amsterdam) Virtual Laboratories for Inspiration and Discovery in

Ecology

- Prof. Henk Dijkstra (Utrecht University) Predicting Local Sea-Level Changes

- Dr. Marco de Vos (ASTRON) Discovering Pulsars with the Low Frequency Array

12.00 Lunch

13.00 Fifth group to be interviewed: leaders of eScience centers and of other related organisations in

the Netherlands:

- Dr. Peter Doorn (DANS)

- Dr. Theo Mulder (Royal Netherlands Academy of Arts and Sciences)

- Prof. Edwin Valentijn (TARGET / University of Groningen)

- Prof. Barry Koren (Eindhoven University of Technology)

- Prof. Arnold Smeulders (IPN / COMMIT)

14.30 Interview with NLeSC Management Team

15.30 Prof. Arjen Doelman (Chair of the eScience Advisory Committee)

16.00 Discussion and main conclusions

18.00 Closing of the meeting

Part 2

Reflection on the Midterm Evaluation of the Netherlands eScience Center by its Board of Directors

The evaluation committee, appointed by the board, has conducted a broad review of the Netherlands

eScience Center during a two day session on November 7-8 2013. In response to their report, the board

expresses its gratitude to the committee for its work done and its useful and practical advice.

The board is very pleased with the positive outcome of the evaluation. This document summarizes the

findings of the committee and the first reactions of the board to the conclusions and recommendations

and will be made public together with the evaluation report.

1) Quality. The quality of the NLeSC-projects and their results are rated as excellent. The committee

was unanimously positive about NLeSC as a whole and concluded that the center definitely has

added value to the Dutch scientific landscape. The board recognizes and welcomes this

conclusion. It notes that the eScience Center has taken up a pioneering role from its onset, under

the joint responsibility for NLeSC of NWO and SURF. This approach is being innovative and

beneficial. The board will continue to encourage and support NLeSC to proceed with its

pioneering work enabling data and computational intensive science.

2) Governance. The evaluation committee was clear in its advice to move forward to a next phase of

governance. Initiating a center like NLeSC required close interaction between both founding

organizations at the highest level. This was achieved by direct representation of the board and

line management of both organizations within NLeSCs board. The board agrees that the next

phase of NLeSCs development requires closer, and more direct, links to NLeSCs research

domains and a board that can strongly support the route to scientific and organizational

leadership in eScience.

a. The board. The board welcomes the suggestions made by the committee to

recompose the board to increase the engagement of the board to NLeSCs activities

and future strategy. It has been proposed that future composition of the board

should include one representative of the Dutch scientific community applying

eScience methods, one person representing ICT science, one industry

representative, one NWO representative and one SURF representative. The board

notes that the representation of ICT science should not be taken too narrowly and

should be read as representing the technology domain, including methodologies

and software.

b. Executive committee. The board expresses its support for the excellent work done

by the executive committee in the past two years as reflected by the success of the

center since its inception.

c. eSAC. The position of the eSAC, the external eScience advisory committee, was

established as an independent organ, mainly composed of international experts. The

advice to have representatives of parallel organizations, such as DANS and

COMMIT take part in the eSAC will therefore be put in general context, and read as

a recommendation to give sufficient voice to such parties regarding the future

directions of NLeSC.

d. eScience Integrators. The concept of the eScience Integrators is considered

innovative and successful in ensuring early access to the scientific communities and

getting their direct involvement in NLeSC. The evaluation committee rightly notes,

however, that their roles are as yet rather undefined and that there is potential

unclarity of roles when integrators are also the primary investigators of NLeSC-

funded projects. The board will address this issue, and ask the executive committee

to better formalize the integrators roles, while keeping enough room for the

dynamics required for this group.

e. eScience Engineers. The board fully supports the credits given to the eScience

engineers, both to the model and their output. Regarding their career potential, the

board is of the opinion that continued efforts must be made to ensure that the

eScience engineers position is such that their appointment at NLeSC also

strengthens their career potential in academia and in industry.

3) Formal status and roles of NLeSC. NLeSC presently has a hybrid role of funder, research

organization and coordinating body at the national level. The committee recommends NLeSC to

focus on the role of funder (in kind and in cash) and secondarily be a research center and an ideas

market place. It believes it is possible to achieve all of these goals by performing research only in

areas which are strictly complementary to excisting science organizations and by seeking funding

only in partnership with at least one other Netherlands research institute. The evaluation

committee has pointed out that a formal status of research institute is lacking, which prevents

NLeSCs access to funding bodies in the Netherlands and beyond (mainly Europe). The board

agrees that this situation must be repaired urgently, as the first external funding opportunities

(such as those under Horizon 2020) are already due for April 2014.

The board acknowledges the coordinating activities led by NLeSC as did the evaluation

committee. In particular the formation of the Platform of Netherlands eScience centers is an

important step demonstrating clearly the national role allocated to NLeSC. The remark of the

committee that NLeSC is a rather Amsterdam-centered enterprise is, in the opinion of the

Board, not supported in fact. The personnel composition of NLeSC is very international and from

the number of allocated projects no Amsterdam bias can be concluded. The board is proud that

NLeSC has developed collaborations with important nearby parties such as KNAW, CWI, NIKHEF,

COMMIT, and the Amsterdam based Universities, VU and UvA. Care must be taken, however, to

address the false perception that NLeSC is Amsterdam focused.

4) Visibility, Industry and Public-Private partnerships. The board agrees with the committee as to the

importance of improved visibility and intensification of industrial links and partnerships with

private partners. However, given the short time since the foundation of NLeSC, the board thinks

that the visibility of NLeSC in the Dutch landscape is beyond expectation. The National eScience

Symposium demonstrated clearly a wealth of productivity and the significant attendance can only

be attributed to the national position that has been established. The numerous contacts

developed between NLeSC and potential industrial players, although ad hoc, are encouraging. The

board agrees with the recommendation of the committee to develop a coherent action plan for

long term co-operations with industrial partners. Participation in the public private partnerships -

part of NWO contribution to the Topsector policy- is obviously encouraged.

5) Future directions. The board agrees with the committee that eScience is becoming fundamental

to and integrated in all scientific disciplines and as a result universities and research centers will

develop their own eScience centers. Coordination, reuse of technology and preventing

fragmentation are key components of the mission of the eScience Center and this will remain

necessary in the developing Netherlands eScience landscape. The committee recommends that

NLeSC writes down a long term vision, strategy and mission, for example with regards to

balancing the breadth and depth of the research portfolio. The board follows this

recommendation and has asked the executive committee to compile a Strategy Document,

encompassing a multi-annual plan, to cover the period 2015-2020. This document should address

recommendations of the evaluation committee and opinions addressed in this response with the

goal to further embed NLeSC in the Dutch and international eScience ecosystem. The strategic

vision should indicate how NLeSC will cope with the continuous environmental changes in the

scientific and technological landscape, remain adaptive and address the needs of important

stakeholders. The Strategy Document will also address the ideal model for funding eScience

initiatives within the scientific domains as NLeSC balances its role as funder, scientific partner and

national coordinator. The board expects to receive the strategy document mid 2014 so that it can

be adopted in their policy.

On behalf of the Board of the Netherlands eScience Center

Ing. A.H. (Amandus) Lundqvist Prof. Dr. J.J. (Jos) Engelen

The Netherlands eScience Center Mid-Term Evaluation · The Netherlands eScience Center Mid-Term Evaluation December 2013 Including: Evaluation Netherlands eScience Center (NLeSC):

Documents

Dive+ NL eScience symposium 2015

End-to-End eScience

Workflows, Semantics & future eScience

Netherlands eScience Center · Research & Innovation in the...

Dropout Prevention Measures in the Netherlands, An...

Research evaluation in the Netherlands : a library...

OGF eScience Function

eScience, Education and Knowledge Management

“Grids and eScience” Mark Hayes Technical Director -...

Impact evaluation of Netherlands supported programmes in...

eScience – Grid Computing Graduate Lecture

Open Source Collaboration · 2018. 3. 12. · Open Source.....