MPI, Erlang and the web

parallel programmingwhy does it matter to web development

who i am

• german expat in NZ

• text editor addict since age 14

• CTO ideegeo Group Limited

• erlounge Wellington

erlounge Wellington

• diverse crowd

• more about parallel programming

• once a month

todays topics

• message passing infrastructure

• erlang

• open standards

scale web based projects

• one server

• separate functionality (some server)

• load balance (some more server)

loadbalance

• web servers is easy

• databases is harder

• business logic often not possible

the problem

• shared state

• shared data

• waiting for other processes

in general terms

global locking

sounds familiar?looks like a parallel programming issue to me

i tried to solve ...

• scalability issues

• scale to many servers

• asynchronous processes

• process queues

• callback queues

i wrote

• a simple perl based environment

• my own message queue

• my own messaging format

what i actually didwithout knowing it ...

i wrote a MPIok, a quite basic message passing infrastructure but it

was one

and to my big surprise

this thing workedbetter as expected

... but ...

i found erlang

erlang ...

• is a functional language

• has a built in MPI

• share nothing approach

feels like heaven on earth

issues

• not compatible to my existing stuff

• feel the pain of custom protocols

• rewrite or compat layer or ...

the usual thing happened

... i left it aside ...

... meanwhile ...

i started to play with

• CouchDB

• RabbitMQ

• ejabberd

all erlangand open standards based

the more I playedthe more I wished I could use them

the inevitable happened

iteration two

AMQP

open standard

• backed by major players

• around for some years

• RabbitMQ (erlang based)

JSON

open format

• used by many

• small footprint

• easy to read for humans and computers

erlang

fast and small

• compared to perl/php/ruby/...

• scales linear to 32 cores

• distributing processes is easy

perl

still in the system

• for compatibility

• for string processing

• for stuff where it makes sense

[your lang here]

language independent

• JSON is available in most languages

• AMQP is available in all major languages

• pick the one that makes sense

internet architectures

• scaling to many servers

• loadbalancing

• reducing global locks

multicore programming

• scaling to many cores

• loadbalancing

• reducing global locks

do we solve simmilar problems here?

where to find me

• @norbu09

• http://norbu09.org

• http://github.com/norbu09

• http://slideshare.net/norbu09

thanks

• http://www.flickr.com/photos/dranidis/

• http://www.flickr.com/photos/samuellkim/

• http://www.flickr.com/photos/tbcave/