Transient Detection and SQL Gijs Molenaar Anton Pannekoek Institute (UvA) @gijzelaerr [email protected]
May 31, 2015
Transient Detection and SQL
Gijs Molenaar !
Anton Pannekoek Institute (UvA) !
@gijzelaerr [email protected]
LOFAR
• Radio telescope
• ~10–240 MHz
• Software telescope
• omni-directional
AARTFAAC
• Lofar sub project
• 24/7 full sky transient detection
• ‘Changes’
• 99% Python code
Transients
• Super-nova
• Gamma-ray burst
• Pulsars
• exo-planets
• ?
Images
• 10-240 MHz
• Split in 10 bands
• 1 image cube per second
• future 10 images per second
Data
• Source
• Position
• Flux (intensity)
• Shape (2d gaussian)
Data rate
• 500 - 5000 inserts per second
• Higher in other modes
Calculations
• Associate in frequency
• Associate in time
Quite a lot of data
• move calculations to data (base)
nonosql
• Structured data
• Natural sharding by sky position
• MonetDB
Why MonetDB
• Dutch (CWI)
• Fast with huge datasets
• Developers next door
calculations to data
• No moving around data
• one huge big monolithic query per ‘operation’
Example
HORRIBLE• Huge monolithic queries
• hard to read
• hard to reason about
• hard to test
• hard to maintain
• hard to port
Solution?
• Functions?
NO!
• DB revisions, versioning hell
• For every code change new version
• Schema migrations
• Init functions on startup?
• DB specific
SOLUTION
Why?• Object Relational Mapper
• Better Query Builder Language!
• Support multiple database dialect
• Functions per dialect
!
• SQLAlchemy -> SQL is like C -> assembler
example
Why not earlier?
• Started by non programmers
• No MonetDB dialect for SQLAlchemy
sqlalchemy-monetdb
• pip install sqlalchemy-monetdb
• https://github.com/gijzelaerr/sqlalchemy-monetdb
• https://github.com/gijzelaerr/djonet
• (banana)
Questions?