Thomas Nowotny Centre for Computational Neuroscience & Robotics Sussex Neuroscience Engineering & Informatics, University of Sussex Cambridge, 19-12-2013.

Thomas Nowotny

Centre for Computational Neuroscience & RoboticsSussex Neuroscience

Engineering & Informatics, University of Sussex

Cambridge, 19-12-2013

GPU enhanced neuronal networks (GeNN): Overview and New Developments

GeNN overviewPart I

Introduction

Fuelled by the games industry, graphics adaptors have become very powerful and are now using massively parallel graphical processing units (GPUs)

They offer a new platform for HPC

NVIDIA® TESLA C2050,(Image from NVIDIA product website)

From: Herb Sutter, “The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software”

NVIDIA® CUDATM

CUDATM= “Common Unified Device Architecture” Was introduced by NVIDIA® to allow main stream

developers to use massively parallel graphics chips for GPGPU without the restrictions of shaders

The first CUDA SDK was released Feb 2007 (according to Wikipedia)

CUDATM is supported on all newer NVIDIA cards

The CUDATM API

Each thread executes what is called a “kernel”, defined by a C-like function

CUDA is a Single-Instruction-Multiple-Data (SIMD) environment

Communication between threads is through shared memory spaces

n1

n2

Block 1 Block 2

Block 3 Block 4

Grid

1

nsyn

Synapse Blocks

Grid

2

Isyn= …

Model

Issues to consider Correct parallelization

Synchronization Atomics (no synchronization between blocks!)

Code structure Block size, number & alignment (unrolling) loops

Memory access patterns Use of different memories Memory bank access conflicts Coalesced memory access Host to/from GPU data transfers

Kernel /GPU

CUDA memory architecture

“Local Memory”

Device memory

Constant memory

Texture memory

Shared memory

GPU registers

Program /CPU

Cache/RegistersCache/Registers

Random Access Memory (RAM)Random Access Memory (RAM)

Graphics adaptor

Host computer

GPU Neural Network Simulators

Nageswaran et al. (UC Irvine, 2009): General simulator for Izhikevich neurons with delay, optimized for Izhikevich's thalamo-cortical model (C++ library)

Fidjeland et al. (Imperial, 2010) - Nemo: General simulator for Izhikevich neurons with delay, optimized for Izhikevich's thalamo-cortical model (C++ library)

Goodman & Brette (Ecole Normale Sup., 2009): GPU extensions to the Brian simulator (one-timestep grids with partial CPU involvement)

Mutch et al. (MIT, 2010) CNS simulator: Simulator for layered “cortical networks”, models can be defined by the user (used exclusively through a MatLab interface)

GeNN: Taking code generation seriously

The GPU enhanced Neuronal Network simulatorProvides a simple C++ API for specifying a neuronal network of interestGenerates optimised C++ and CUDA code for the model and for the detected hardware at compile time (e.g. grid/block organisation, HW capability, model parameters)GeNN can offer a large variety of different models – only the used ones actually enter the generated codeUsers can define their own model equaThe generated code is compiled with the native NVIDIA compiler nvcc (and all its optimisations).

GeNN flowchart

Example: Insect olfaction model

Performance

AL-MB: 50 % all-to-all

Individualconductances

spikes commu-nicated to host(dotted)

spike # commu-nicated to host (solid)

GPU

CPU

Examples of code-gen benefits

Synaptic connectivities:

model.addSynapsePopulation("PNKC", NSYNAPSE, DENSE, INDIVIDUALG, "PN", "KC", myPNKC_p);model.addSynapsePopulation("PNLHI", NSYNAPSE, ALLTOALL, INDIVIDUALG, "PN", "LHI", myPNLHI_p);model.addSynapsePopulation("LHIKC", NGRADSYNAPSE, SPARSE, INDIVIDUALG, "LHI", "KC", myLHIKC_p);

GeNN

Each connectivity type can generate entirely different code:

DENSE SPARSE

npre-neuron

n post

-neu

ron

connectivity matrix

sparse matrix represen-tation

n syna

pse

PostID gsyn

Pre#

npre-neuron

Examples of code-gen benefits

New neuron models:

int newModel= nModels.size();n.varNames.push_back(tS("V"));n.varTypes.push_back(tS("double"));n.pNames.push_back(tS("tau"));n.simCode= tS(“$(V) = $(V) + DT/ $(tau)* (- 40.0*$(V))");nModels.push_back(n);

GeNN

model.addNeuronPopulation("gp", 1000, newModel, p, ini);

GeNN

New DevelopmentsPart II

SpineML is an XML-based description format for networks of point neurons

It is a proposed extension of the NineML format, including the description of neural dynamics, network structure, and experimental procedures.

SpineML acts as an exchange format between creation tools and simulators, as well as between collaborators working on a shared model

SpineML

For more details see:Richmond P, Cope A, Gurney K, Allerton DJ. "From Model Specification to Simulation of Biologically Constrained Networks of Spiking Neurons" Neuroinformatics. 2013

Courtesy of Dr. Alex Cope

SpineCreator is one tool that can be used to author SpineML models

A Graphical User Interface allows models to be specified without any programming

Some features:Graphing3d visualisationExtensible simulator supportVersion control support

Available at https://github.com/SpineML/SpineCreator

SpineCreatorCourtesy of Dr. Alex Cope

SpineML can use code generation for simulator support, so it is very easy to add support for GeNN

XSLT translation scripts transform the model into GeNN user files, and the GeNN standard toolchain compiles and runs the simulation

Support for new neuron types (and soon synapses and weight update rules) is supported through XSLT translation to GeNN system files

SpineML and GeNNCourtesy of Dr. Alex Cope

BRIAN2 interface

BRIAN is a popular simulator for neuronal networks (http://briansimulator.org/)

BRIAN2 is in active development and will be entirely code-generation based

Users will be able to choose target devices, including Python (as before), C/C++, Android and GPU/GeNN

from brian2 import *from brian2.devices.genn import *set_device('genn');

##### Define the modeltau = 10*msecondeqs = '’’dV/dt = (-40*mV-V)/tau : volt # (unless-refractory)'''threshold = 'V>-50*mV'reset = 'V=-60*mV'refractory = 5*msecondN = 1000

##### Generate genn codeG = NeuronGroup(N, eqs, reset=reset, threshold=threshold, name='gp')M = SpikeMonitor(G)G2 = NeuronGroup(1, eqs, reset=reset, threshold=threshold, name='gp2')# Run the network for 0 seconds to generate the codenet = Network(G, M, G2)net.run(0*second)genn_device.build(net)

BRIAN2// setp 1: add variablesn.varNames.clear();n.varTypes.clear();n.varNames.push_back(tS("V"));n.varTypes.push_back(tS("double"));n.varNames.push_back(tS("lastspike"));n.varTypes.push_back(tS("double"));n.varNames.push_back(tS("not_refractory"));n.varTypes.push_back(tS("bool"));// step2: add parametersn.pNames.clear(); n.pNames.push_back(tS("tau"));n.pNames.push_back(tS("mV"));// step 3: add simcoden.simCode= tS(" $(not_refractory) = ($(not_refractory)) || (!(false));\n\const double _V = $(V) * exp(-(DT) / $(tau)) - 40.0 * $(mV) + 40.0 * $(mV) * exp(-(DT) / $(tau));\n\$(V) = _V;\n\");// step 4: add thresholder coden.thresholdCode= tS(" const double _cond = $(V) > -50 * $(mV);\n\”);// step 5: add resetter coden.resetCode= tS(" \n\ $(V) = -60 * $(mV);\n\");nModels.push_back(n);

GeNN

Proof of concept

from brian2 import *from brian2.devices.genn import *set_device('genn');

##### Define the modeltau = 10*msecondeqs = '’’dV/dt = (-40*mV-V)/tau : volt # (unless-refractory)'''threshold = 'V>-50*mV'reset = 'V=-60*mV'refractory = 5*msecondN = 1000

##### Generate genn code

G = NeuronGroup(N, eqs, reset=reset, threshold=threshold, name='gp')M = SpikeMonitor(G)G2 = NeuronGroup(1, eqs, reset=reset, threshold=threshold, name='gp2')# Run the network for 0 seconds to generate the codenet = Network(G, M, G2)net.run(0*second)genn_device.build(net)

BRIAN2

// setp 1: add variablesn.varNames.clear();n.varTypes.clear();n.varNames.push_back(tS("V"));n.varTypes.push_back(tS("double"));n.varNames.push_back(tS("lastspike"));n.varTypes.push_back(tS("double"));n.varNames.push_back(tS("not_refractory"));n.varTypes.push_back(tS("bool"));// step2: add parametersn.pNames.clear(); n.pNames.push_back(tS("tau"));n.pNames.push_back(tS("mV"));// step 3: add simcoden.simCode= tS(" $(not_refractory) = ($(not_refractory)) || (!(false));\n\const double _V = $(V) * exp(-(DT) / $(tau)) - 40.0 * $(mV) + 40.0 * $(mV) * exp(-(DT) / $(tau));\n\$(V) = _V;\n\");// step 4: add thresholder coden.thresholdCode= tS(" const double _cond = $(V) > -50 * $(mV);\n\”);// step 5: add resetter coden.resetCode= tS(" \n\ $(V) = -60 * $(mV);\n\");nModels.push_back(n);

GeNN

Summary

GeNN is a code-generation based simulator targeting GPUs with NVIDIA CUDA.

The native C/C++ interface is flexible and convenient for expert users.

A SpineCreator – SpineML – GeNN pipeline has been prototyped and will be used in the Green Brain project.

A BRIAN2 – GeNN interface is in development and will hopefully be released with 1st release of BRIAN2.

Acknowledgments

Ramon Huerta, Alex Cope, Esin Yavuz, James Turner NVIDIA (professor partnership: 2x Quadro FX 5800

cards) & hardware donation for Green Brain Funders:

http://www.sourceforge.net/projects/genn