Roy Williams Andrew Drake, Matthew Graham, Ashish Mahabal California Institute of Technology

Roy WilliamsAndrew Drake, Matthew Graham, Ashish Mahabal

California Institute of Technology

Skyalert and Event Processing

This talk defines these

VOEvent: Int’l standard representation of data for astronomical transient as XML

VOEventNet: A Protocol for pushing VOEvents, and its nodes

Skyalert: Software for managing VOEvents: web, subscriptions, repository

VOEvent Key Concepts

IVORN – unique identifier for event Instance of a Stream that also has IVORN How is it resolved?

Who author’s provenance

What data model designed by author May have URLs … permanence, calibration?

WhereWhen targeting in spacetime

Citation related events “followup", "supersedes" or "retraction”

3

VOEventLib

http://lib.skyalert.org/VOEventLib/

Python interface to VOEvent Code built automatically from schema

With ‘utility’ package Vutil

v = VOEvent.VOEvent(role='test', ivorn='ivo://testing#111') author = Author(contactName=['Donald Duck'])v.set_Who(Who(Author=author))

what = What()what.add_Param(Param(name='apple', value='123'))what.add_Param(Param(name='orange', value='124'))v.set_What(what)

s = stringVOEvent(v)print s

VOEventNet Key Concepts

Broker Broadcaster of received events May know which subscriber wants what

Subscriber Connects to Broker Long-term connection

Publisher Scientifically responsible Works with a broker May sign events with PGP

5

VOEventNet

a network of independent nodes exchanging VOEvents

with the VOEvent Transport Protocol (VTP)

LOFAR and 4piSkyrunning CometNASA events

running GCN/TAN

DC3.comrunning Dakota

Skyalertrunning Comet

CBATAAVSOCRTS1,2,3CSS NEO

VOEventNet defined

http://arxiv.org/abs/1206.0236

query

subscribe

relay

annotate

translate

translate

registry

publish

repository

VOEventNet Patterns

Query

VOEventNet

External annotator

Translate

Translate

Publish

repository

SkyalertOpen sourced VOEventNet node

Push messagesPush actionsWeb interfaceWeb services

Many types of events can be homogenized and interoperatedSupported by US National Science Foundation OCI-0915473

SkyalertAnnotation

Annotation can be:

• Newer observations or calibration

• Archive search for matches

• Machine-learning ‘decision’

• Follow-up observation

• Comment / Circular

VOEventNet

PublishAnnotation

Annotation

How can annotations be queued and clouded in case of high event rate?

Two lists of events

betelgeuse.ligo.caltech.edu

All ‘events’ from five high-energy missionsincluding ‘utility’ events

Received by VTP

skyalert.org

‘role=observation’ (astrophysics) events onlyXMPP and email translations

Skyalert Stream Registry... will be VO registry

http://www.skyalert.org/streams/

12

Skyalert Key Concepts

Stream Event semantics defined by author

What does Burst_SOD mean in the Fermi stream? Like a data dictionary, a schema, common metadata

Portfolio Event + associated data

As collection of VOEvents with the observation first

Query Define what is ‘interesting’ For Past or Future events Language, as XML, SQL tables (flex and fixed), KW dict,

triple store??

13

Concept: Stream as Event Template

Event author defines parameters and meaning in advance.stream event

made byperson in daytime

made byrobot system at night14

SWIFT portfolio

Notices from the satellite Circulars from the humans

15

Event DatabasesTechnology and Language Evaluations

Stardog SELECT ?s WHERE {?s voe:hasParam ?p . ?p voe:hasName "…"^^xsd:string . ?p voe:hasValue ?v .

FILTER (?v < …)}

MySQL fixed: with a separate table for each stream select id from … where … < …

MongoDB (NoSQL): ivorn, param name, value db.test.find({"param":…, "value":{"$lt":…}})

MySQL flex: with a params table select id from params where name = "…" and value < …

eXist native XML for $event in /voe:VOEvent where $event/What//Param[@name = '...']/@value < "..." return

<id>{$event/@ivorn}</id>

eXist XML

MySQL flex

Mongo noSQL

MySQL fixed

Stardog triples

0 10 20 30 40 50 60 70

Speed

Skyalert web servicesfrom django.utils import simplejsonimport urllib

sk = http://skyalert.org

# CRTS and CRTSCircular["First"]["eventClass"] == "Supernova"query = "285"

jtableurl = "%s/events/jtable/%s/" % (sk, query)

while 1: jsontable = urllib.urlopen(jtableurl).read() pytable = simplejson.loads(jsontable) list = pytable["list"] print "Found %d portfolios" % len(list) for pf in list: ivorn = pf[1]

dict = {} dict['ivorn'] = ivorn dict['JSON'] = 'on' params = urllib.urlencode(dict) resolveurl = "%s/resolve/" % sk jsondetail = urllib.urlopen(resolveurl, params).read() pydetail = simplejson.loads(jsondetail) # ivorn of the primary event of the portfolio should be here crtsevent = pydetail["portfolio"][ivorn] mag = crtsevent["FirstDetectionparams"]["magnitude"]

tok = ivorn.split('#') localivorn = tok[1] date = crtsevent[""]["ISOtime"] print localivorn, date, mag

jtableurl = jsonresult["next"] if jtableurl == "finished": break

1107030071174105491 2011-07-03T08:36:01 18.9354001107030091124153498 2011-07-03T08:01:30 18.7563991107031180794128821 2011-07-03T04:39:47 20.1698001107031070764142675 2011-07-03T04:07:39 18.1661001107021180094118167 2011-07-02T10:46:06 18.6014001107021041264109182 2011-07-02T09:31:06 15.8796001107011150104107901 2011-07-01T10:47:36 18.2796991107011120104119909 2011-07-01T10:46:15 17.8748001107010040834139996 2011-07-01T06:17:32 18.4836011107011040674116917 2011-07-01T04:13:08 19.630899

Magnitudes of CRTS supernovae

17

Who are the subscribers? What size telescope / Data mining / Citizens Is it the same as the “GCN followup” gang? How can we set up a clearinghouse of information?

How to build “simulated event stream” for LSST? So prototypes can be tested Remember average rate and burst rate

What are annotation services for LSST? Same as “common queries”

What is the security requirement? In testing, commissioning, observing

What language for alerts and history queries Triples, SQL, noSQL, XML, ...