Recent advances in modeling advanced accelerators: plasma based acceleration and e-clouds W.B.Mori, C.Huang, W.Lu, M.Zhou, M.Tzoufras, F.S.Tsung, V.K.Decyk.

Recent advances in modeling advanced accelerators: plasma based acceleration and e-clouds

W.B.Mori , C.Huang, W.Lu, M.Zhou, M.Tzoufras, F.S.Tsung, V.K.Decyk (UCLA)D.Bruhwiler, J. Cary, P. Messner, D.A.Dimtrov, C. Neiter (Tech-X)

T. Katsouleas, S.Deng, A.Ghalam (USC) E.Esarey, C.Geddes (LBL)

J.H.Cooley, T.M.Antonsen (U. Maryland)

Particle Accelerators Why Plasmas?

• Limited by peak power and breakdown

• 20-100 MeV/m

• No breakdown limit

• 10-100 GeV/m

Conventional Accelerators Plasma

Laser Wake Field Accelerator(LWFA, SMLWFA, PBWA)

A single short-pulse of photons

Plasma Wake Field Accelerator(PWFA)

A high energy electron bunch

Concepts For Plasma Based Accelerators

Drive beam Trailing beam

1. Wake excitation2. Evolution of driver and wake3. Loading the wake with particles

Physics necessitates the use of particle based methods:Many length and time scales for fields + particles--grand challenge!

Wake excitation is nonlinear: Trajectory crossingRosenzweig et al. 1990 Puhkov and Meyer-te-vehn 2002

Ion column provides ideal accelerating and focusing forces

Plasma Accelerator Progress and the “Accelerator Moore’s Law”

LOA,RAL

LBL ,RALOsaka

Slide 2

Courtesy of Tom Katsouleas

• Maxwell’s equations for field solver

• Lorentz force updates particle’s position and momentum

Interpolate to particles

Particle positions

Lorentz Force

push

partic

les weight to grid

z vi i,

ρn m n mj, ,,r

dpdt

E v B cr r r r= + ×

r rE Bn m n m, ,,

t

Computational cycle (at each step in time)

What Is a Fully Explicit Particle-in-cell Code?Not all PIC codes are the same!

Typical simulation parameters:~108-109 particles ~10-100 Gbytes~105 time steps ~104-105 cpu hours

Advanced accelerators:Before SciDAC

Beam-driven wake* Fully Explicitz ≤ .05 c/ωp

y, x ≤ .05 c/ωp

t ≤ .02 c/ωp

# grids i n z ≥350

# grids i n x, y ≥150# s tep s ≥2 x 105

Npar ticles ~.25 -1. x 10 8 (3D)

Part icle s x st ep s ~.5 x 10 13 (3D) - ≥ 50 00 h rs

*Laser -driven Ge V stage re quire s o n the orde r of (ωo/ωp)2=100 0 x longe r, however, the the re solution can us ua lly be re laxed (~200, 000 ho urs ).

5000+ node

hours for

each GeV of

energy

One 3D PIC

code

Accomplishments and highlights:Code development

• Four independent high-fidelity particle based codes– OSIRIS: Fully explicit PIC– VORPAL: Fully explicit PIC + ponderomotive guiding center– QuickPIC: quasi-static PIC + ponderomotive guiding center– UPIC: Framework for rapid construction of new

codes--QuickPIC is based on UPIC: FFT based

• Each code or Framework is fully parallelized. They each have load

balancing and particle sorting. Each production code has ionization packages for more realism. Effort was made to make codes scale to 1000+ processors.

OSIRIS:full parallel PIC for plasma accelerators

• Successfully applied to various LWFA and PWFA problems

– Mangles et al., Nature 431, 538 (2004).– Tsung et al., Phys. Rev. Lett., 93, 185002 (2004)– Blue et al., Phys. Rev. Lett., 90 214801 (2003)

• Code– Moving window– Parellized using domain decompostion– Two charge conserving deposition schemes– Current and field smoothing– Field + Impact Ionization– Static load balance.– Well tested

• Modern (object-oriented, Fortran 95 techniques)– Parallel (general domain decomposition) or Serial– Cross-platform (UNIX, Linux, AIX, OS X, MacMPIC)– Based on a well proven Fortran 77 code – Sophisticated 3D data diagnostics

• OSIRIS development team– UCLA(F. S. Tsung, J. W. Tonge), USC (S. Deng), IST (R. A.

Fonseca and L. O. Silva), Ecolé Polytechnique (J. C. Adam), and RAL (R. G. Evans).

– See http://exodus.physics.ucla.edu/

VORPAL – parallel PIC & related algorithms for advanced accelerators

• Successfully applied to various LWFA problems– Geddes et al., Nature 431, 538 (2004).– Cary et al., Phys. Plasmas (2005), in press (invited).

• Recently implemented algorithms– Ponderomotive guiding center treatment of laser pulses– PML (perfectly matched layer) absorbing BC’s– implicit 2nd-order & explicit 4th-order EM

• Many other capabilites/algorithms (only a sample here):– Impact & field ionization; secondary e- emission– Fluid methods for plasmas; hybrid PIC/fluid

• Modern (object-oriented, C++ template techniques)– Parallel (general domain decomposition) or Serial– Cross-platform (Linux, AIX, OS X, Windows)

• VORPAL development team– J. Cary (Tech-X/CU), C. Nieter, P. Messmer, D. Dimitrov,

J. Carlson, D. Bruhwiler, P. Stoltz, R. Busby, W. Wang, N. Xiang (CU), P. Schoessow, R. Trines (RAL)

– See http://www.txcorp.com/technologies/VORPAL/• Highly leveraged via SBIR funds: DOE, AFOSR, OSD

104

s(N)

VORPAL scales well to 1,000’s of processors

Colliding laser pulses

Particle beams

Code development:QuickPIC

Code features:• Based on UPIC parallel object-oriented plasma simulation Framework.• Underlying Fortran library is reliable and highly efficient• Multi-platform, Mac OS 9/X, Linux/Unix.• Dynamic load balancing

Model features:• Highly efficient quasi-static model for beam drivers• Ponderomotive guiding center + envelope model for laser drivers.• Can be 100+ times faster than conventional PIC with no loss in accuracy.• ADK model for field ionization.

Applications:• Simulations for PWFA experiments, E157/162/164/164X/167• Study of electron cloud effect in LHC.• Plasma afterburner design

Scalability:• Currently scales to ~32 processors• With pipelining should scale to 10,000+ processors

afterburner

hosing

E164X

2-D plasma slab

Beam (3-D):

Laser or particles

Wake (3-D)

1. initialize beam

2. solve ∇⊥2ϕ =ρ, ∇⊥

2ψ =ρe ⇒ Fp,ψ

3. pushplasma, storeψ

4. stepslabandrepeat2.

5. useψ togiantstepbeam

QuickPIC loop:

(1c2

∂2

∂t2−∇2)A =

4πc

j

(1c2

∂2

∂t2−∇2)φ=4πρ

−∇⊥2A =

4πc

j

−∇⊥2φ=4πρ

€

quasi - static : ∂s ≈ 0

Maxwell equations in Lorentz gauge Reduced Maxwell equations

])([:

:

//⊥⊥

⊥ ×+−

=

⎥⎦

⎤⎢⎣

⎡ ×+−=

BV

EP

BV

EP

cVcq

dd

electronsplasmaFor

ccq

ds

delectronsbeamFor

e

e

ee

bbb

ξ

Initialize beam

Call 2D routine

Deposition

3D loop end

Push beam particles

3D loop begin

Initialize plasma

Field Solver

Deposition

2D loop begin

2D loop end

Push plasma particles Iteration

s =z, ξ =ct−z

2∂∂s

−ik0 +∂∂ξ

⎛

⎝⎜

⎞

⎠⎟a−∇⊥

2a=k02χ pa=−k0

2 ωp2

ω02γp

a

Quasi-static Model including a laser driver

Laser envelope equation:

Parallelization for QuickPIC

zy

x

Node 3 Node 2 Node 1 Node 0

3D domain decompositionCommunication

Beam

xy

2D domain decomposition with dynamic load balancing

Node 3

Node 2

Node 0

Node 1

Beam

Plasma

2D loop timing

10

100

1000

1 10 100

CPUs

Excution time (Sec)

Dawson diff. nodes

Dawson same node

Nersc diff. nodes

Nersc same node

Dawson diff. nodes -- overhead removed

Dawson same node -- overhead removed

Network Overhead

• Scales up to 16-32 CPUs for small problem size.

• Network overhead dominates on Dawson cluster (GigE).

• 4 times performance boost with infiniband hardware.

• With pipelining should scale to 10,000+ processors

Accomplishments and highlights:Physics

• Development of new reduced models (QuickPIC) and benchmarking of codes (OSIRIS vs. Vorpal, QuickPIC vs. OSIRIS, Vorpal vs. Vorpal PG)

• Code validation (by adding more realism):– Modeling of PWFA experiments at SLAC in 3D: 4GeV energy gain in ~10cm

(OSIRIS and QuickPIC).– Identified self-ionization as a plasma source option in PWFA (OOPIC, Vorpal,

OSIRIS)– Modeling LWFA experiments at LBNL and RAL: 100MeV monoenergetic

beams in ~1mm (OSIRIS and VORPAL).

• New physics:– Modeling PWFA Afterburner (energy doubler) stages: From 50 to 100 GeV

and from 500 to 1000 GeV (QuickPIC).– Modeling possible 1GeV mono-energetic LWFA stages: with and without

external optical guiding (OSIRIS and VORPAL).

-3

-2

-1

0

1

2

-5 0 5 10

OSIRISQuickPIC

ξ( /cωp)

-3

-2

-1

0

1

2

3

-8 -6 -4 -2 0 2 4 6 8

OsirisQuickPIC (l=2)QuickPIC (l=4)

ξ ( /cωp)

-0.1

-0.05

0

0.05

0.1

-10 -5 0 5 10

Osiris

QuickPIC (l=2)

ξ ( /cωp)

-1

-0.5

0

0.5

1

-6 -4 -2 0 2 4 6

Osiris QuickPIC (l=2)

ξ ( /cωp)

e- driver e+ driver

e- driver with ionization laser driver

QuickPIC Benchmark: Full PIC vs. Quasi-static PIC

Benchmark for different drivers

Excellent agreement with full PIC code. More than 100 times time-savings. Successfully modeled current experiments. Explore possible designs for future experiments. Guide development on theory.

100+ CPU savings with “no” loss in accuracy

Code benchmarking:Vorpal fully explicit vs. ponderomotive

guiding center

• Removes fast time-scale of laser pulse– orders of magnitude faster than full

PIC– can simulate 3 cm LBNL plasma

channel in 2D in a few processor-hours

• Excellent comparison w/ 2D PIC

– good agreement seen for a0~1

• accelerating wake fields (upper fig.)• normalized particle velocities

(lower fig.)– particle trapping seen at larger values

of a0

€

Emax ~ 4GeV

(initial energy chirp

considered)

Modeling self-ionized PWFA experiment with QuickPIC

E164X experiment

QuickPIC simulation

-4

-2

0

2

4

6

8

0

2

4

6

8

10

12

0 100 200 300 400 500

(z μ )m

€

Emax ~ 5− 0.5(β − tron radiation) = 4.5GeV

Located in the FFTB

e-

N=1-2· 1010

σz=0.1 mm=30 E GeV

Ionizing Laser Pulse

(193 )nm Li Plasma

ne≈6 · 10 15 cm-3

L≈30 cm

CerenkovRadiator

Streak Camera(1 )ps resolution

-X RayDiagnostic

Optical TransitionRadiators Dump

25 m

∫Cdt

!Not to scale

Spectrometer

25 m

FFTB

Full-scale simulation with ionization of E-164xx is possible using a new code QuickPIC• Identical parameters to experiment including self-ionization: Agreement is very good!

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

0

+2

+4

-4

-2

0 +5-5X (mm)

Rel

ativ

e E

nerg

y (G

eV)

3 Nature papers (September 2004) where mono-energetic electron beams with energy near 100 MeV were measured. Supporting PIC simulations were presented.

SciDAC members were collaborators on two of these Nature publications and SciDAC codes were used.

Cover is a Vorpal simulation

Phys. Rev. Lett. by Tsung et al. (September 2004) where a peak energy of 0.8 GeV and a mono-energetic beam with an central energy of 280 MeV were reported in full scale 3D PIC simulations.

Recent highlights: LWFA simulations using full PIC

3D PIC Simulations with no fitting parameters:Nature papers, “agreement” with experiment: What is the metric for agreement?

• In experiments, the # of electrons in the spike is 1.4 108.

• In our 3D simulations, we estimate of 0.9 108 electrons in the bunch.

3D Simulations for: Nature V431, 541 (S.P.D Mangles et al)

0

5 107

1 108

1.5 108

20 40 60 80 100 120 140

3D PICExperiment

Energy (MeV)

f(E) (a.u.)

•Simulation Parameters–Laser:

• a0 = 4• W0=24.4 l=19.5 mm• wl/wp = 33

–Particles• 2x1x1 particles/cell• 500 million total

–Plasma length• L=.7cm• 300,000 timesteps

Full scale 3D LWFA simulation using OSIRIS:200TW, 40fs

4000 cells101.9 μm

256 cells80.9 μm

256 cells80.9 μm

State-of- the- art ultrashort laser pulse

0 = 800 nm, t = 30 fs

I = 3.4x1019 W/cm-2, W =19.5 μm

Laser propagation

Plasma Backgroundne = 1.5x1018 cm-3

Simulation ran for 75,000 hours on 200 G5 x-serve processors on

DAWSON (~5 Rayleigh lengths)

Simulation ran for 75,000 hours on 200 G5 x-serve processors on

DAWSON (~5 Rayleigh lengths)

Simulations are leading experiments:200TW 30fs laser----1.5 GeV beam in ~cm

• Laser blows out all plasma electrons leading to an ideal accelerating structure

• Isolated beams are self-injected.

• Beams become mono-energetic as they outrun the wake.

QuickTime™ and aMPEG-4 Video decompressor

are needed to see this picture.

OSIRIS simulation

One goal is to build a virtual accelerator:

A 100+ GeV-on-100+ GeV e-e+ Collider

Based on Plasma Afterburners

Afterburners

3 km

30 m

Advanced accelerator milestone:Full-scale simulation of a 1TeV afterburner is possible using QuickPIC

• We use parameters consistent with the International Linear Collider “design”

•We have modeled the beam propagating through ~25 meters of plasma!

QuickTime™ and aMPEG-4 Video decompressor

are needed to see this picture.

•Before SciDAC: 5,000,000+ node hours at NERSC (was not done)

•Because of SciDAC: 5,000 node hours on the DAWSON Cluster (2.3 Ghz x-serves)

Advanced accelerators:After SciDAC

• 3D modeling with realism-2 explicit PIC codes plus a parallel Framework• Code benchmarking• Code validation-Full scale 3D modeling of experiments• Efficient and high fidelity reduced description models: Rapid construction of

fully parallelized code• Extension of plasma techniques to conventional accelerator issues: e-cloud

• Rapid progress has resulted from:– Faster computers– New algorithms– Reuseable software

• Scientific discovery:• 3 Nature articles and 8 Phys. Rev. Lett.’s

Vision for the future:High fidelity modeling of .1 to 1TeV plasma accelerator stages

• Physics Goals:– A) Modeling 1to 10 GeV plasma accelerator stages: Predicting and designing near term

experiments.– B) Extend plasma accelerator stages to 250 GeV to 1TeV range: understand the physics and

scaling laws– C) Use plasma codes to definitively model e-cloud physics:

• 30 minutes of beam circulation time on LHC• ILC damping ring

• Software goals:– A) Add pipelining into QuickPIC: Allow QuickPIC to scale to 1000’s of processors.– B) Add self-trapped particles into QuickPIC and ponderomotive guiding center Vorpal packages.– C) Improve numerical dispersion* in OSIRIS and VORPAL.– D) Scale OSIRIS, VORPAL, QuickPIC to 10,000+ processors.– E) Merge reduced models and full models– F) Add circular and elliptical pipes* into QuickPIC and UPIC for e-cloud.– G) Add mesh refinement into* QuickPIC, OSIRIS, VORPAL,and UPIC.– H) Investigate the utility of fluid and Vlasov models.

* Working with APDEC ISIC

Pipelining: scaling quasi-static PIC to 10,000+ processors

beam

solve plasma response

update beam

Initial plasma slab

Without pipelining: Beam is not advanced until entire plasma response is determined

solve plasma response

update beam

solve plasma response

update beam

solve plasma response

update beam

solve plasma response

update beam

Initial plasma slab

beam

1 2 3 4

1 2 3 4

With pipelining: Each section is updated when its input is ready, the plasma slab flows in the pipeline.

Advanced accelerators:Goals

• Develop reusable software based on particle-in-cell methods that scales to 1000+ processors.

• Develop codes that use this reusable software and which include the necessary physics modules.

• Develop reduced description codes to reduce cpu and memory needs.

• Benchmark codes against each other and validate codes against experiments.

• Use validated codes to discover ways to scale plasma based accelerator methods to .1 to 1 TeV.