Lenore Mullin Program Director CISE CCF Algorithmic Foundations National Science Foundation

CoProD 08 Friday, October 3, 2008

Architecture Aware Tensor-Based Architecture Aware Tensor-Based ComputingComputing

Challenges for the Computer Science and Mathematics Communities

CISE CCF Algorithmic Foundations: Moore’s Law and Verifiable, Scalable, Portable, and Reproducible Matrix and Tensor Software

Lenore MullinLenore MullinProgram DirectorProgram Director

CISE CCF Algorithmic FoundationsCISE CCF Algorithmic FoundationsNational Science FoundationNational Science Foundation

[email protected]@nsf.gov

2CoProD 08 Friday, October 3, 2008

OutlineOutline

• NSF and CISENSF and CISE• CCF: Algorithmic Foundations and CCF: Algorithmic Foundations and

BeyondBeyond• Challenges and Open QuestionsChallenges and Open Questions• ConclusionsConclusions


National Science FoundationNational Science Foundation

Administrative Offices

Directorate for BiologicalSciences

Directorate for Computer &Information Science & Engineering

Directorate for Social, Behavioral & Economic Sciences

Directorate for Education& Human Resources

Directorate for Engineering Office of International Science and Engineering

Office of the Director

National ScienceBoard

Office Cyberinfrastructure

Office ofInspector General

Directorate for Mathematical &

Physical Sciences

Directorate for Geosciences Office of Polar Programs


CISE GoalsCISE Goals1.1. EnableEnable the United States to remain the United States to remain

competitive in computing, competitive in computing, communications, and information science communications, and information science and engineeringand engineering

2.2. PromotePromote understanding of the principles understanding of the principles and uses of advanced computing, and uses of advanced computing, communications, and information systems communications, and information systems in service to societyin service to society

3.3. ContributeContribute to universal, transparent, and to universal, transparent, and affordable participation in an information-affordable participation in an information-based societybased society


Achieving CISE GoalsAchieving CISE Goals• CISE supports investigator initiated CISE supports investigator initiated

research in all areas of computer and research in all areas of computer and information science and engineeringinformation science and engineering

• CISE helps develop and maintain CISE helps develop and maintain cutting-edge national computing and cutting-edge national computing and information infrastructure for research information infrastructure for research and educationand education

• CISE contributes to the education and CISE contributes to the education and training of the next generation of training of the next generation of computer scientists and engineers. computer scientists and engineers.


Assistant Director: Jeannette Wing

Deputy Assist Dir: Deborah Crawford

Div Dir:

Taieb

Znati

Div Dir:

Haym Hirsh

CISE OrganizationCISE Organizationhttp://www.nsf.gov/cise/about/org_chart.jsphttp://www.nsf.gov/cise/about/org_chart.jsp

Div Dir:

Sampath

Kannan

http://www.nsf.gov/cise/about/org_chart.jsp


CCF: Computing andCCF: Computing andCommunication Foundations DivisionCommunication Foundations Division

http://www.nsf.gov/div/index.jsp?div=CCFhttp://www.nsf.gov/div/index.jsp?div=CCF

• Emerging Models and Technologies for Emerging Models and Technologies for ComputationComputation– Computational biology; quantum computing; nano-scale computing; Computational biology; quantum computing; nano-scale computing;

biologically-inspired computingbiologically-inspired computing

• Foundations of Computing Processes Foundations of Computing Processes and Artifactsand Artifacts– Advanced computation research; compilers; computer Advanced computation research; compilers; computer

architecture; design automation (micro/nano); graphics & architecture; design automation (micro/nano); graphics & visualization; software engineering & languagesvisualization; software engineering & languages

• Theoretical/Algorithmic Foundations Theoretical/Algorithmic Foundations – Computer science and communication theory; numeric Computer science and communication theory; numeric

symbolic/graphic computation; theory of computing; symbolic/graphic computation; theory of computing; computational algebra and geometry; signal processingcomputational algebra and geometry; signal processing

http://www.nsf.gov/div/index.jsp?div=CCF


Theoretical/Algorithmic FoundationsTheoretical/Algorithmic FoundationsNumeric, Symbolic and Algebraic ComputingNumeric, Symbolic and Algebraic Computing

• Investigations into new data structures and algorithms that yield Investigations into new data structures and algorithms that yield optimizations for particular applications are encouraged.optimizations for particular applications are encouraged.

• This includes the design and construction of high quality scientific This includes the design and construction of high quality scientific software ideally adept across numerous scientific domains. software ideally adept across numerous scientific domains. Tensors are pervasive throughout NSF disciplines. Tensors are pervasive throughout NSF disciplines.

• Specific research topics of interest include, but are not limited to, Specific research topics of interest include, but are not limited to, the following: the following: numerical linear and multi-linear algebras, numerical linear and multi-linear algebras, tensor tensor algebras and decompositions used in memory hierarchyalgebras and decompositions used in memory hierarchy mappings;mappings; linear and non-linear optimization; modeling and linear and non-linear optimization; modeling and simulation of complex processes; and numerical solutions of simulation of complex processes; and numerical solutions of differential equations and PDE’s. Research in numerical differential equations and PDE’s. Research in numerical computing and optimization has natural interdisciplinary computing and optimization has natural interdisciplinary applications. In fact, applications. In fact, this program seeks applications in science this program seeks applications in science and engineering whose basic problems actually require the and engineering whose basic problems actually require the development of new numerical and optimization methods.development of new numerical and optimization methods.


Theoretical/Theoretical/Algorithmic FoundationsAlgorithmic Foundations

Numeric, Symbolic, and Algebraic ComputingNumeric, Symbolic, and Algebraic Computing• Research focused on finding powerful methods for Research focused on finding powerful methods for

symbolically solving algebraic - numeric systems that combine symbolically solving algebraic - numeric systems that combine differential, integral and polynomial equations is required. differential, integral and polynomial equations is required. Interests include foundational research in algorithms and their Interests include foundational research in algorithms and their efficient execution. efficient execution.

• Basic research topics include: computational algebra and Basic research topics include: computational algebra and analysis, computational number theory and algebraic analysis, computational number theory and algebraic geometry, geometry, integration of numeric and symbolic techniques, integration of numeric and symbolic techniques, symbolic scientific applications and software.symbolic scientific applications and software. Fruitful Fruitful application areas for symbolic computation include the application areas for symbolic computation include the solution of complex equation sets.solution of complex equation sets.

• Symbolic/Numeric manipulation and Tensors: Symbolic/Numeric manipulation and Tensors: – composition of tensor operations(symbolic) and numeric composition of tensor operations(symbolic) and numeric

instantiation: e.g. SAGE, Matlab, Mathematica, Maple, Expression instantiation: e.g. SAGE, Matlab, Mathematica, Maple, Expression Templates, XML, compilers, interpreters, …Templates, XML, compilers, interpreters, …

– Tensors are n-d arraysTensors are n-d arrays


CCF: Theoretical/Algorithmic CCF: Theoretical/Algorithmic Foundations (AF)Foundations (AF)

Cluster supports research in the following areas: • Models of computationModels of computation• Computational complexityComputational complexity• Parallel and distributed computationParallel and distributed computation• Random and approximate algorithmsRandom and approximate algorithms• Algorithmic algebra, geometry, topology, and logicAlgorithmic algebra, geometry, topology, and logic• Computational optimizationComputational optimization• Techniques for representing, coding and transmitting informationTechniques for representing, coding and transmitting information• $30M/Year$30M/Year• New TF Program Solicitation New TF Program Solicitation NSF 08-518NSF 08-518

– Due Date March 12, 2008 - March 19, 2008 Due Date March 12, 2008 - March 19, 2008 http://www.nsf.gov/pubs/2008/nsf08518/nsf08518.htmhttp://www.nsf.gov/pubs/2008/nsf08518/nsf08518.htmTR Program Officers:TR Program Officers: John Cozzens, Lenore Mullin, Richard Biegel, John Cozzens, Lenore Mullin, Richard Biegel,

Sirin Tekinay, Robert Grafton, EK ParkSirin Tekinay, Robert Grafton, EK Park

http://www.nsf.gov/pubs/2008/nsf08518/nsf08518.htm


Theoretical/Algorithmic Foundations Theoretical/Algorithmic Foundations and BEYOND!!!and BEYOND!!!

• How can we create transformational How can we create transformational science when we can’t verify scientific science when we can’t verify scientific software?software?

• How can domain scientists doing How can domain scientists doing computational experiments achieve computational experiments achieve reproducibility:reproducibility:– Same answer and is that answer correct?Same answer and is that answer correct?– Are the resources used the same?Are the resources used the same?– Can the software scale to today’s and Can the software scale to today’s and

tomorrow’s hardware?tomorrow’s hardware?– Can we produce software that is optimal?Can we produce software that is optimal?


Theoretical/Algorithmic Foundations Theoretical/Algorithmic Foundations and BEYOND!!!and BEYOND!!!

• Optimality and Large Data SetsOptimality and Large Data Sets• Optimality and Data Locality across Optimality and Data Locality across

processor/memory hierarchyprocessor/memory hierarchy• Peta-Scale Computing and Beyond: Peta-Scale Computing and Beyond:

scalability and portabilityscalability and portability• Algebra of Arrays to build ANY Tensor Algebra of Arrays to build ANY Tensor

based applicationbased application– Must be a closed algebra without Must be a closed algebra without

anomalies for verificationanomalies for verification– No language today has such an algebraNo language today has such an algebra


Moore’s Law:Moore’s Law: Data Density Data Density DoublesDoubles every every 18 Months18 Months

EXCEPTEXCEPT Notice Notice flatteningflattening of slope due to Compilers of slope due to Compilers

1850 1950 20001900 2050

10-6

103

1

10-3

106

109

Babbage Engine

CMOS ICs

TX-2

ENIAC

Differential Analyzer

GeneralArchitecture

Lattice-GasArchitecture

Quantum Dots

Liquid NMR

Conve

ntion

al Com

puter

Roa

dmap

QC

Roa

dmap

MIPS

Year


Proebsting’s Law:Proebsting’s Law:CompilerCompiler Advances Advances Double Double Computing PowerComputing Power Every Every 18 Years18 Years

This means that This means that while while hardware hardware computing horsepowercomputing horsepower increases at increases at roughlyroughly 60%/year 60%/year, , compiler optimizations contribute compiler optimizations contribute onlyonly 4%. 4%.

1850 1950 20001900 2050

10-6

103

1

10-3

106

109

Babbage Engine

CMOS ICs

TX-2

ENIAC

Differential Analyzer

GeneralArchitecture

Lattice-GasArchitecture

Quantum Dots

Liquid NMR

Conve

ntion

al Com

puter

Roa

dmap

QC

Roa

dmap

MIPS

Year


What is Computational Science and What is Computational Science and Engineering?Engineering?

Mathematics

Physical Sciences and Biological Sciences

Computer Science and Engineering

X

X = The Intersection of Domain Sciences, Mathematics andComputer Science and Engineering


What can we do?What can we do?• Recent Award: for mini-symposia at the 2009 SIAM Annual

meeting (Lenore Cowen, Tufts: Uniting Discrete Methods, optimizations and CISE Community with the Community studying Matrix Operations, Tensors, Verifiable Computational Experiments and Scalability) in which Computer Scientists and students will be funded to attend and interact. This was initiated due to numerous tensor sessions at the 2008 SIAM Annual meeting.

– Tensor Decompositions Solving Fundamental Problems in Chemistry

– Tensor Decompositions for Large-Scale Date Applications– A Novel Higher-order Generalized Singular Value Decomposition for

Comparative Analysis of DNA Microarray Data from Different Organisms

– Tensor Algebraic Methods and Their Application to High-Dimensional Multi-Modal Data

– TensorFaces: Multilinear (Tensor) Decomposition of Image Ensembles

– Multilinear (Tensor) Independent Component Analysis– Modeling of Epileptic Seizures using Tensor Analysis– On a Generalization of Sylvester Methods for Symmetric Tensor

Decomposition


What can we do?What can we do?• A mini-symposium at the 2008 SIAM Annual meeting (MS3)

entitled Architecture-Aware Scientific Computing. Organizers and Presenters: L. Mullin (NSF) and Padma Raghavan (NSF PI).

• Plans to have an invitation only workshop with Frank Olken (IIS) are planned for spring 2009 to bring together experts in Knowledge Representation, Tensors, Algorithms and other related areas in Computer Science. Charles Van Loan, Cornell

• Recent Award: for a workshop at the Courant Institute to bring together Mathematicians and Computer Scientists to discuss scalable algorithms for PDEs on parallel, distributed, and multi-core algorithms. ODEs and PDEs can be represented as matrix and tensor operations. Numeric and symbolic environments are growing in popularity to combine verification and optimal implementations.

• Career Application 2008: BIO and CCF AF (Orly Alter: Integrative and Comparative Tensor Algebra Models of DNA Microarray Data from Different Studies of the Cell Cycle)


What can we do?What can we do?• Milestones in Computer Algebra 2008:

(Invitation only workshop) Systematic Tensor Simplification: a Diagramatic

Approach by A. D. Kennedy and T. Reiter. This workshop illustrated the need for combined numeric and symbolic environments to compute and symbolically prove correctness of designs. Numerous articles from this workshop discussed the need to combine environments, which was validated by an NSF supported workshop report written by E. Kaltofen (one of the organizers) November 2007 at NSF in Arlington. Symbolic/Numeric proposals entered in the 2008 NSG solicitation showed a 100% growth over 2007.


What can we do?What can we do?• SCAN 2008: expected outcomesSCAN 2008: expected outcomes

– Hardware and software support for verification toolsHardware and software support for verification tools– Theory, algorithms and arithmetic for verified numerical Theory, algorithms and arithmetic for verified numerical

computationscomputations– Supercomputing and reliabilitySupercomputing and reliability– Dynamical systems and verified numerical computationDynamical systems and verified numerical computation– Global optimization and verified numerical computationGlobal optimization and verified numerical computation– Programming tools for verified numerical computationProgramming tools for verified numerical computation– Computer aided proofsComputer aided proofs– Industrial and scientific applications of verified numerical Industrial and scientific applications of verified numerical

computationscomputations

• CoProD 08: expected outcomesCoProD 08: expected outcomes– Definition of new directions for combining numeric and Definition of new directions for combining numeric and

symbolic approaches in solving constraints and symbolic approaches in solving constraints and optimization problems in particular and in decision making optimization problems in particular and in decision making in general.in general.


What can we do?What can we do?• Matrix and Tensor operations are pervasive in science and Matrix and Tensor operations are pervasive in science and

engineeringengineering• Tensors are n-d arrays, but n-d arrays are more generalTensors are n-d arrays, but n-d arrays are more general• Generalized multi-dimensional Inner and Outer productsGeneralized multi-dimensional Inner and Outer products• Summations of multi-dimensional arraysSummations of multi-dimensional arrays• Projection operatorsProjection operators• AX=B like problemsAX=B like problems• Coupled differential and integral equations, eigenvalue Coupled differential and integral equations, eigenvalue

problems: generally translate to matrix problemsproblems: generally translate to matrix problems• Even non-linear operations: iterative solutionsEven non-linear operations: iterative solutions• Linear and Multilinear Algebra is not enough!Linear and Multilinear Algebra is not enough!

– Scalars, anomaliesScalars, anomalies• Existing languages are not enough!Existing languages are not enough!


Possible SolutionsPossible Solutions

• Identify a closed algebra that subsumes Identify a closed algebra that subsumes important matrix operationsimportant matrix operations

• Augment existing languages with this Augment existing languages with this algebra: optional usealgebra: optional use

• Solve a few important problems completelySolve a few important problems completely• Use the same algebra to map to processor Use the same algebra to map to processor

memory hierarchiesmemory hierarchies• Use the same algebra to abstract machinesUse the same algebra to abstract machines• These concepts proposed at Sandia These concepts proposed at Sandia

Workshop on Memory Hierarchy Workshop on Memory Hierarchy Optimizations for Scientific Software, Optimizations for Scientific Software, January 2008January 2008


Possible SolutionsPossible Solutions• Synergize Mathematicians, Synergize Mathematicians,

Computer Scientists and Domain Computer Scientists and Domain Scientists to collaborateScientists to collaborate

• Create a new community that solves Create a new community that solves these open questionsthese open questions– Revisit and reinventRevisit and reinvent

• Community then creates a research Community then creates a research base for funding agenciesbase for funding agencies

• Workshops and ColloquiaWorkshops and Colloquia– Supplements and/or new grantsSupplements and/or new grants


Numeric, Symbolic and Algebraic Numeric, Symbolic and Algebraic Computing Program in TFComputing Program in TF

• These issues appear in last year’s and These issues appear in last year’s and this year’s solicitationsthis year’s solicitations

• MPS and CISE cooperating programsMPS and CISE cooperating programs– Hope to develop new solicitationsHope to develop new solicitations

• Attend SIAM, SC, APS, MRS, etc. Attend SIAM, SC, APS, MRS, etc. – Raise the consciousness of computational Raise the consciousness of computational

scientists in these communitiesscientists in these communities• After solving small number of After solving small number of

algorithms within this algebra, identify algorithms within this algebra, identify what to do next.what to do next.– Can be used in existing programs.Can be used in existing programs.


Thank YouThank You

Questions?Questions?

Lenore MullinLenore MullinProgram DirectorProgram Director

National Science FoundationNational Science FoundationComputer & Information Science & Engineering Computer & Information Science & Engineering

DirectorateDirectorateDivision of Computer and Communications Foundations Division of Computer and Communications Foundations

Algorithmic Foundations ClusterAlgorithmic Foundations Cluster

[email protected]@nsf.gov

Lenore Mullin Program Director CISE CCF Algorithmic Foundations National Science Foundation

Documents