Solving neural ODEs using fixed-point arithmetic with stochastic rounding Mantas Mikaitis [email protected] Collaboration with: Michael Hopkins, Dave Lester, Steve Furber @ CS Dept. NLA group meeting Manchester, 24 October 2019
Solving neural ODEs using fixed-point arithmetic with stochastic roundingMantas [email protected] with: Michael Hopkins, Dave Lester, Steve Furber @ CS Dept.
NLA group meetingManchester, 24 October 2019
2/23
Mantas Mikaitis Stochastic rounding in fixed-point arithmetic
https://en.wikipedia.org/wiki/White_stork
3/23
Mantas Mikaitis Stochastic rounding in fixed-point arithmetic
• Hardware and software for simulating large-scale biological neural networks• 1Million ARM968 (2004 ARM processors)
The SpiNNaker project
Images: Furtak, S. (2018). Neurons. In R. Biswas-Diener & E. Diener (Eds), Noba textbook series: Psychology
4/23
Mantas Mikaitis Stochastic rounding in fixed-point arithmetic
Generalized fixed-point number representation
2"# 2"$ …… 2# 2% .… …𝑠
𝑖 𝑝• 𝑖 - number of integer bits• 𝑝 - number of fractional bits (precision)• Range:
[−2+, 2+ − 2"-] (signed),[0, 2+ − 2"-] (unsigned)
• 𝜖 = 2"- (machine epsilon)• 𝜖 is a gap between any two consecutive
values• Accuracy is measured in 𝜖.• The maximum error is: 2
$
Converting to decimals:Given a binary number: < 𝑠|𝑢, 𝑖, 𝑝 > − 𝑠𝐼+"#𝐼+"$ ⋯ 𝐼%. 𝐹#𝐹$ ⋯𝐹-,if signed format and 𝑠 is set
𝑣𝑎𝑙𝑢𝑒 = ?@A%
+"#
𝐼@2@ +?CA#
-
𝐹C2"C − 𝑠2+, 𝑒𝑙𝑠𝑒 𝑣𝑎𝑙𝑢𝑒 = ?@A%
+
𝐼@2@ +?CA#
-
𝐹C2"C
𝑤
5/23
Mantas Mikaitis Stochastic rounding in fixed-point arithmetic
GCC types accum and long fract
• accum <s, 16, 15>:𝜖 = 2"#E ≈ 0.0000305176…
Range: [−2#M = −65536, 2#M − 2"#E ≈ 65535.99996948… ]
0 0.0000305176 0.00006103516-0.0000305176
• long fract <s, 0, 31>:𝜖 = 2"Q# ≈ 0.000000000465661
Range: [−1, 2% − 2"Q# ≈ 0.999999999534339… ]
00.000000000465661
0.000000000931322−0.000000000465661
𝜖
6/23
Mantas Mikaitis Stochastic rounding in fixed-point arithmetic
Izhikevich neuron model: Cortical spiking patterns (Izhikevich 2003)
(RK2 Midpoint ODE solver,Hopkins & Furber, 2015)
(on spike: 𝑉 = 𝑐,𝑈 = 𝑈 + 𝑑)
7/23
Mantas Mikaitis Stochastic rounding in fixed-point arithmetic
Error measure: spike time lag/lead (when I=4.775 at 60th ms and kept constant)
Arithmetic errors in spike lag
8/23
Mantas Mikaitis Stochastic rounding in fixed-point arithmetic
Errors when I=4.775 at 60th ms and kept constant (relative to FP64)
9/23
Mantas Mikaitis Stochastic rounding in fixed-point arithmetic
Some improvements to the constants• Instead of 0.04 -> 0.040008544921875 (error of 0.28𝜖).• GCC rounds down by default, returning an error of 0.72𝝐• Vague specification in the ISO/IEC 18037 standard:
“If the source value cannot be represented exactly by the fixed-point type, the source value is rounded to either the
closest fixed-point value greater than the source value (rounded up) or to the closest fixed-point value less than
the source value (rounded down).”
• Use fixed-point type < 𝑢, 0, 32 >, i.e. 0.040000000037252902984619140625.
• Requires multiplications to be done in mixed precision.
10/23
Mantas Mikaitis Stochastic rounding in fixed-point arithmeticM
SB
LSB
×
=
LSB
MSB
≫ (Round and saturate) Use these bits for rounding.
32-bit accum
64-bit answer
binary point
Fixed-point multiplier
11/23
Mantas Mikaitis Stochastic rounding in fixed-point arithmetic
Given the output from multiplier:
answer somewhere in this gapaccum range
ignore
Round-down (RD)
12/23
Mantas Mikaitis Stochastic rounding in fixed-point arithmetic
Given the output from multiplier:
?
answer somewhere in this gapaccum range
0 1
Round-to-nearest (RN)
13/23
Mantas Mikaitis Stochastic rounding in fixed-point arithmetic
Reduced error: running longer (0.1ms timestep).
14/23
Mantas Mikaitis Stochastic rounding in fixed-point arithmetic
Stochastic rounding (SR)
Given a real number 𝑥, a random number 𝑃 ∈ 0,1 from a uniform distribution and a fixed-point destination format < 𝑠, 𝑖, 𝑝 > with 𝜖 = 2"-,
Over many roundings:𝔼 𝑆𝑅 𝑥,< 𝑠, 𝑖, 𝑝 > = 𝑥.
𝑆𝑅 𝑥, < 𝑠, 𝑖, 𝑝 > =𝑥 𝑖𝑓 𝑃 ≥
𝑥 − 𝑥𝜖
,
𝑥 + 𝜖 𝑖𝑓 𝑃 <𝑥 − 𝑥𝜖
.
15/23
Mantas Mikaitis Stochastic rounding in fixed-point arithmetic
Given the output from the multiplier:
answer somewhere in this gapaccum range
Round-off bitsIf < round-off valueround up, else round down.
Use these bits as probability of
rounding up, [0,1).
SR at bit level
16/23
Mantas Mikaitis Stochastic rounding in fixed-point arithmetic
SR algorithm: by addition
17/23
Mantas Mikaitis Stochastic rounding in fixed-point arithmetic
Testing method of Izhikevich ODE solutions
• Four ODE solvers: RK2 Midpoint, RK2 Trapezoid, RK3 Heun, Chan-Tsai
• Two different neuron types (regular and fast spiking -RS/FS)
• Five arithmetics: IEEE 754 double (reference), float, fixed-point {round-down, round-to-nearest, stochastic}
• Run for ~1min to produce around 600 spikes.• For SR, repeat experiment 100 times with different
PRNG seeds• Report mean and std.dev. spike lag of each spike.
18/23
Mantas Mikaitis Stochastic rounding in fixed-point arithmetic
Results: 2nd order solvers (h=0.1ms)
Spike number
Spik
e la
g/le
ad c
ompa
red
to d
oubl
e re
f (m
s).
19/23
Mantas Mikaitis Stochastic rounding in fixed-point arithmetic
Results: 3rd order solvers (h=0.1ms)
Spike number
Spik
e la
g/le
ad c
ompa
red
to d
oubl
e re
f (m
s).
20/23
Mantas Mikaitis Stochastic rounding in fixed-point arithmetic
Comparison of SR resolutions
Given the output from multiplier:
Use SOME of these bits as probability
of rounding up, [0,1).
21/23
Mantas Mikaitis Stochastic rounding in fixed-point arithmetic
Psuedo-random number generation
• Marsaglia’s KISS64 (expensive but passes randomness tests)• Linear congruential generator (modular arithmetic by
overflow of 32-bit registers):
• For our tests, no significant difference (mean & std.dev.)!
22/23
Mantas Mikaitis Stochastic rounding in fixed-point arithmetic
Efficiency
• Fixed-point + SR 2.6× faster than soft-float and 4.2× faster than soft-double running RK2 Midpoint
• Previously: use fixed-point for speed, soft-float for accuracy• Now: use fixed-point for speed, fixed-point+SR for accuracy
23/23
Mantas Mikaitis Stochastic rounding in fixed-point arithmetic
Summary
• Stochastic rounding is very beneficial in solving this ODE• Fixed-point arithmetic similar in accuracy to double-precision• Only 6 random bits per multiplication required• Implemented in hardware on SpiNNaker2 (64/32bit -> 32/16bit
and float -> bfloat16).• Next questions:
1. Where else does SR apply?2. Is the pseudo-randomness in arithmetic a difficulty?3. Is SR less expensive than other approaches for similar
problems?
24/23
Mantas Mikaitis Stochastic rounding in fixed-point arithmetic
References
• Izhikevich. Simple Model of Spiking Neurons. 2003. IEEE Tran. Neur. Net.
• Hopkins & Furber. Accuracy and Efficiency in Fixed-Point Neural ODE Solvers. 2015. https://www.ncbi.nlm.nih.gov/pubmed/26313605
• Hopkins, Mikaitis, Lester & Furber. Stochastic rounding and reduced-precision arithmetic for solving neural ODEs. 2019. https://arxiv.org/pdf/1904.11263.pdf
• PD ISO/IEC TR 18037:2008, Programming languages. C. Extensions to support embedded processors. 2008. https://bsol.bsigroup.com/Bibliographic/BibliographicInfoData/000000000030185055
25/23
Mantas Mikaitis Stochastic rounding in fixed-point arithmetic
References
• Higham & Srikara. Simulating low precision floating point arithmetic. 2019. http://eprints.maths.manchester.ac.uk/2692/