ORION Workshop: XSEDE and Building a National/International Cyberinfrastructure

March 20, 2014

XSEDE and Building a National/International

Cyberinfrastructure

John Towns

PI and Project Director, XSEDE

Director, Collaborative Cyberinfrastructure Programs, NCSA

[email protected]

License terms • Please cite as: Towns, John. 2014 XSEDE and Building a

National/International Cyberinfrastructure, Presentation, Advanced Computing – Transforming Research: A Planning Workshop, 13 Feb 2014, Toronto, Ontario, Canada, http://www.slideshare.net/jtownsil/xsede-at-orionworkshopfeb2014.

• ORCID ID: http://orcid.org/0000-0001-7961-2277 • Except where otherwise noted, by inclusion of a source URL or some other

note, the contents of this presentation are © by the Board of Trustees of University of Illinois. This content is released under the Creative Commons Attribution 3.0 Unported license (http://creativecommons.org/licenses/by/3.0/). This license includes the following terms: You are free to share – copy and redistribute the material in any medium or format; and to adapt – remix, transform, and build upon the material for any purpose, even commercially.

• This can be done under the following conditions: attribution – You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.

Who is this guy? • John Towns

– PI and Project Director of XSEDE – Director, Collaborative eScience Program Office at NCSA – Co-Founder, Illinois Campus Cluster Program

• Background – just a mid-western boy from Missouri

• I don’t see a lot of folks from Missouri in the circles I am in

– failed physicist • general relativity

– failed computational scientist • numerical simulation of black hole spacetimes

– failed networked applications guy • NLANR-DAST: National Laboratory for Applied Network Research-

Distributed Applications Support Team

– but, a budding social engineer

• Wearing my XSEDE hat for this discussion

3

Who is this guy, part deux?

full disclosure/caveats/disclaimers

• To be clear – I am a member of the Compute Canada Board of Directors – I am not here in that capacity

• I am not in a position to speak for the CC Board

– I may make comments regarding the CC Board if appropriate

– I am open to comments from the community that I can carry back to the CC Board

• My capacity here – PI and Project director of XSEDE – someone with a lot of experience in building

institutional/regional/national/international cyberinfrastructure

4

Step-by-Step-Instructions

• How to create an open cyberinfrastructure ecosystem in 1,200 easy steps

5

. . . .

Boundary Conditions and Principles for XSEDE

• XSEDE inherited TeraGrid environment • XSEDE inherited TG community and their expectations

• Point of view has changed

– not an HPC/CS/tech play – about productivity and creating the environment necessary to be

productive • focus on the success of researchers!

• Finally figured out that the project must define a solution that is designed to evolve! – technologically and organizationally!

• Identify the greatest needs and start there – Don’t forget what you have learned – both good and bad!

• Oh yea… the researchers don’t care about your existence (per se) – they care about access to resources, services and support

6

XSEDE – accelerating scientific discovery

• XSEDE’s Vision: a world of digitally enabled researchers, engineers, and scholars participating in multidisciplinary collaborations to tackle society’s grand challenges

• XSEDE’s Mission: to substantially enhance the productivity of a growing community of researchers, engineers, and scholars through access to advanced digital services that support open research

7

XSEDE’s Strategic Goals

• Deepen and extend the use of the advanced digital research services ecosystem – deepen use by existing researchers, engineers, and scholars – extend use to new communities – prepare the current and next generation via education, training, and

outreach – raise the general awareness of the value of advanced digital services

• Advance the advanced digital research services ecosystem – create an open and evolving e-infrastructure – enhance the array of technical expertise and support services offered

• Sustain the advanced digital research services ecosystem – assure and maintain a reliable and secure infrastructure – provide excellent user support services – operate an effective and innovative virtual organization

8

What is XSEDE?

• An ecosystem of advanced digital services accelerating scientific discovery – support a growing portfolio of resources and services

• advanced computing, high-end visualization, data analysis, and other resources and services

• interoperability with other infrastructures

• A virtual organization (partnership!) providing – dynamic distributed infrastructure – support services, and technical expertise to enable

researchers engineers and scholars • addressing the most important and challenging problems facing

the nation and world

• A project funded by the National Science Foundation

9

XSEDE Factoids: high order bits

• 5 year, US$121M project – plus US$9M, 5 year Technology Investigation Service

• separate award from NSF

– option for additional 5 years of funding upon major review after PY3

• No funding for major hardware – coordination, support and creating a national/international

cyberinfrastructure – coordinate allocations, support, training and documentation for

>$100M of concurrent project awards from NSF

• ~140 FTE /~250 individuals funded across 20 partner institutions – this requires solid partnering!

10

Convenience requirements will always increase

Each generation of users

requires more convenience

than the former: thus we must

always be adding new layers of

software while maintaining and

extending existing reliability

and capability.

Change is the only Constant

– Heraclitis 535BC-475BC

11

No, his mind is not for rent

To any god or government.

Always hopeful, yet discontent,

He knows changes aren't permanent,

But change is.

– Rush - Tom Sawyer

Some Important Lessons Learned

• TeraGrid Project Final Report – http://hdl.handle.net/2142/43874 – a “lessons learned” document from the TeraGrid experience

• Some highlights that speak to partnership needs – “The work of managing such a large project needs to be carefully

distributed to make sure all the work gets done, but also needs to be well coordinated, to make sure there is a consistent message.”

– “TeraGrid lacked a structure to make contentious decisions and the authority to enforce those decisions even when some participants disagreed.”

– “The heterogeneity of the user environments on the various computing systems was a difficult challenge for all stakeholders.”

– “TeraGrid was able to sustain and scale-up projects through highly leveraged funding from other grants raised by the RPs, particularly in the education realm.”

12

Some Unexpected Challenges:

XSEDE is a socio-technical ecosystem

• Highly distributed organization – challenges in managing a project that involves

staff at 20 partner institutions

• A completely virtual organization – breaking new ground from an organizational

structure and management point of view

• Highly distributed engineering project – developing new methodologies to adapt

traditional practices to the unusual context of XSEDE

13

Some Musings on Cyberinfrastructure

Partnerships Work

• Partnerships can be a Good Thing – leverage of resources, services are personnel

– alignment of multiple efforts and funding steams to accomplish larger goals

– ultimately facilitate greater productivity of researchers!

• But there are practical pitfalls – socio-political issues often obscure the real goal

– strong individual personalities can upset the applecart in a variety of ways

– the importance of the mechanics of managing the partnership and maintaining accountability among the partners can never be underestimated

14

Strategies for Success in (Cyberinfrastructure)

Partnerships • Keep your eye on the ball!

– vision, mission, goals, KPIs, metrics – know why the partnership exists and how you will assess your collective

success – all partners must support the vision

• Know your stakeholders and their needs/interests/priorities – all partners are also stakeholders, not just the research community

being supported

• KISS principle – hierarchies of partnerships seriously complicate the management of the

whole enterprise • coordination of efforts, reporting, accountability, flowing money, …

– institutions must have the freedom to partner locally, regionally, nationally and internationally in order to pursue their objectives and priorities

• Make sure you can make decisions! – decision-making processes that do not require 100% consensus but are

also inclusive of the stakeholders are key

15

Questions?