Introduction Technical Details Scaling and Summary Preforming Gather/Scatter Operations On A 2D Grid Using Tensor Contractions and CTF Peter Tatkowski Laboratory for Parallel Numerical Algorithms Department of Computer Science University of Illinois at Urbana-Champaign July 27, 2017 Peter Tatkowski Gather/Scatter and CTF
27
Embed
Preforming Gather/Scatter Operations On A 2D Grid Using ...lpna.cs.illinois.edu/talks/7_27_17.pdf · Preforming Gather/Scatter Operations On A 2D Grid Using Tensor Contractions and
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
IntroductionTechnical Details
Scaling and Summary
Preforming Gather/Scatter Operations On A2D Grid Using Tensor Contractions and CTF
Peter Tatkowski
Laboratory for Parallel Numerical AlgorithmsDepartment of Computer Science
Numerical method for solving problems of engineering andcompuational physicsInvolves taking a mesh and splitting it into many elements
Makes what used to be solving complicated differentialequations rather easy to approximate
This has many advantagesAccurate representation of complex geometryEasy representation of the total solutionCapture of local effects
Spectral Element Methods are similar to Finite ElementMethods, except that they use different nodes and differentdifferential equations
Peter Tatkowski Gather/Scatter and CTF
IntroductionTechnical Details
Scaling and Summary
Background KnowledgeProblem StatementMotivation
Unstructured Grids
I will focus on unstructured grid setupThis means that any individual element can be connectedwith any other element along its edgesGrid may be disconnected
This kind of setup allows us to take into account lots ofdifferent grid possibilities
Peter Tatkowski Gather/Scatter and CTF
IntroductionTechnical Details
Scaling and Summary
Background KnowledgeProblem StatementMotivation
Figure 1
Figure 1: An example of a Finite Element Method
Peter Tatkowski Gather/Scatter and CTF
IntroductionTechnical Details
Scaling and Summary
Background KnowledgeProblem StatementMotivation
Figure 2
Figure 2: 2 elements with n = 3 degrees of freedom
Deville, M.O., Fischer, P.F., Mund, E.H. (2002) High-OrderMethods for Incompressible Fluid Flow. New York, NY:Cambridge University Press.
Peter Tatkowski Gather/Scatter and CTF
IntroductionTechnical Details
Scaling and Summary
Background KnowledgeProblem StatementMotivation
Boundary Conditions
When preforming a Finite Element Method in 2D on anunstructured grid, certian degrees of freedom repeatthemselves
This replication happens because of a particular approachtaken where it is highly convinient
For each element:The edges can repeat at most twiceThe corners can repeat an infinite amount of times
Is there a way to account for this repetition using tensorcontractions?
Peter Tatkowski Gather/Scatter and CTF
IntroductionTechnical Details
Scaling and Summary
Background KnowledgeProblem StatementMotivation
Figure 3
Figure 3: Regular Grid
Peter Tatkowski Gather/Scatter and CTF
IntroductionTechnical Details
Scaling and Summary
Background KnowledgeProblem StatementMotivation
Figure 4
Figure 4: Irregular Grid
Peter Tatkowski Gather/Scatter and CTF
IntroductionTechnical Details
Scaling and Summary
Background KnowledgeProblem StatementMotivation
Problem Statement
After computing calculations on individual degrees offreedom in each element, can we find an effecient way tosum up each degree of freedom’s neighbors using tensorcontractions?Also, how does this scale with multiple processes?More focused on long term preformance rather than shortterm setupKey idea: since internal degrees of freedom don’t have anyneighbors, we can effectively ignore them in ourcalculations.
We can reduce the tensor, as shown in the details later
Peter Tatkowski Gather/Scatter and CTF
IntroductionTechnical Details
Scaling and Summary
Background KnowledgeProblem StatementMotivation
Motivation
Finite Element Analysis is used with many engineeringapplications
Computational fluid dynamicsCar crash simulationSeveral chemical, fluid, structural applications as well
Focus on implementing a simple, algebaic approach to theproblem with tensor contractions. Advantageous because:
Simple to implement and understandCan be computed with different meshes quicklyEasy to implement the methods using high-level linearalgebra programming abstractions (like CTF)Easy extension to higher dimensional domains
The grid u has dimensions elems,n,nelems is the number of finite elements in the gridn is the number of degrees of freedom in each direction ofthe mesh
Grid is this shape because it is unstrucutedEach wall can connect to only one other wallEach corner can have any number of neighbors
We need some way to determine connectivity betweenelementsFirst, number all elements in the mesh from [0,elems − 1]
We allocate a global array of dimension elems,4This shape allows us to account for each element and eachof its respective four edgesThis array allows us to determine each element’s neighborsIf an index contains a natural number, the current element’scorresponding wall neighbors the other element referencedby the indexIf an index contains a negative number, the currentelement’s corresponding wall doesn’t neighbor anything
The walls and corners in the tensor have an implicitnumbering to them based on its coordinates in the u tensorThis numbering will make it easier to not only computewall-wall connectivity, but also corner connectivity as wellIt is importaint that this numbering is maintained within thegrid, or else this method doesn’t work
Several checks in place in the code to make sure that themesh is correctly structured
Each wall can only connect with a certain set of otherwalls, as indicated below:
0 1 2 31 0 0 12 3 3 2
This is because the degrees of freedom will be backwardsif this is not honoredUser just has to make sure that all elements have thisconnectivity between other elements, works for all 2Dmeshes
Pabzxy contains 1 iff the wall corresponding to a,b isproperly indexed by either x or y , and the one that it is notindexed by has the same value as z. For example, on ann = 3 grid:
P0,0,1,1,2 would be one because index 1,2 is on wall 0 (0,0),and indicies z, x matchP1,0,2,0,2 would be one because index 0,2 is on wall 3 (1,0),and indicies z, y matchP0,1,1,1,2 would be zero because index 1,2 is on wall 0(0,0), but the index is supposed to be on wall 1 (0,1)
Pabzxy reduces the tensor to just be its walls, while Pcdzjkbrings the tensor back to its original shape (elems,n,n)
So far, we have only wall to wall connectivityHow do we achieve corner connectivity with just wallconnectivity?If the grid is set up correctly, the corner connectivity can bediscovered using an algorithm to traverse the walls inO(elems2), Ω(elems) time
Not necessarily a tight upper bound, but tight lower bound,depends on grid sturucture
This algorithm, if done over all corners, produces aelems,4, vector array, where the vector index can be ofvariable length
The variable length accounts for the variable amount ofelements one corner can connect to
This algorithm is then used to fill the Gcorners tensor, byfilling the appropriate indicies with 1Future work will involve a tensor-based version of thisalgorithm, allowing for maximum parallelization
The gather operation was tested by comparing every indexin our calculated tensor with what the values should havebeenThe true values were computed by adding up adjacentwalls, and the corners were evaluated using a similarmethod to filling the gather corner tensor.Meshes tested were Figure 4 and also a larger version ofFigure 3, both of which passed
Peter Tatkowski Gather/Scatter and CTF
IntroductionTechnical Details
Scaling and Summary
ScalingSummary
Scaling
Currently small scale testing being preformed right nowMainly done with changing the number of elements anddegrees of freedomChanging number of elements, elems, changes run timemore drastically than changing number of degrees offreedom, n
Will test scalability with multiple processes on Blue Watersin the forseeable future
Peter Tatkowski Gather/Scatter and CTF
IntroductionTechnical Details
Scaling and Summary
ScalingSummary
Summary
Two tensor contractions preform the gather operationOperation can be preformed efficently in parallelDesigned for many contractions, so setup time can beoverlooked
OutlookTensor version of Algorithm 1Test scaliability of this method on Blue WatersTransition this method over to 3D