EVENT DRIVEN CUBE MONITORING. David Prime & David Elliott SQLBits 6
Feb 23, 2016
EVENT DRIVEN CUBE MONITORING.David Prime & David ElliottSQLBits 6
WHO WE ARE.
Who we are– David Prime – Betfair Research– David Elliott – Information Management & Analytics Architect
Betfair– Launched June 2000 around an exchange betting platform– You can bet that an outcome will happen (back) or that it won't happen (lay). – You can choose the odds at which you want to play. – You can bet whilst the game is in play. – You can play on a range of products and games other than sports wagering
What this means in terms of data– Bets: >5 million bets daily– Latency: 99.9% bets processed <1 sec– More trades than all of the European stock exchanges combined
OUR OBJECTIVES.
Background– Early look at StreamInsight in Deep-dive– Architectural direction: EDSOA– Real Time requirements: Anti-Fraud, Legislation, Exposure Monitoring– BI / OI– Analytics API / Continuous ETL
Cube Monitoring– A good use-case and an opportunity to assess using SI with the rest of the BI stack– Provide light-weight aggregated usage information for the business
Real-Time– Alerts: name and shame greedy users, discover broken code– Aggregate session data– Using time windows to run complex monitoring scenarios
COMPLEX EVENT PROCESSING.
Betfair is awash with events
Your online business probably is too
COMPLEX EVENT PROCESSING.
STREAMINSIGHT.
What is StreamInsight?
NewIntegrated
FastImproving
OVERVIEW.
DB
Input adaptor
Outputadaptors
Alerting
Cubes
Trace events
CEP Server
Trace & Real-Time ETL
StreamInsight
WHAT DO WE WANT TO CAPTURE?
TRACE EVENTS.
ERROR
QUERY SUBCUBE
QUERY BEGIN
QUERY END
EXISTING SESSION
SESSION INTIALIZE
AUDITLOGIN
AUDIT LOGOUT
TRACING ANALYSIS SERVICES.
String connString = "Provider=MSOLAP;Data Source=bigbox;Initial Catalog=AdventureWorks Sample;Integrated Security=SSPI;";
// Create AS server objectserver = new Microsoft.AnalysisServices.Server();
// Connect serverserver.Connect(connString);
Trace trace = server.Traces.Add();
TraceEvent sessionInit = trace.Events.Add(TraceEventClass.SessionInitialize);sessionInit.Columns.Add(TraceColumn.TextData);sessionInit.Columns.Add(TraceColumn.ConnectionID);sessionInit.Columns.Add(TraceColumn.NTDomainName);sessionInit.Columns.Add(TraceColumn.NTUserName);sessionInit.Columns.Add(TraceColumn.ApplicationName);sessionInit.Columns.Add(TraceColumn.StartTime);sessionInit.Columns.Add(TraceColumn.CurrentTime);sessionInit.Columns.Add(TraceColumn.DatabaseName);
etc...
C#
TRACING ANALYSIS SERVICES.
trace.Update();
//engage the traces TraceEventHandler onTraceEvent = new TraceEventHandler(OnTraceEvent); TraceStoppedEventHandler onTraceStopped = new TraceStoppedEventHandler(OnTraceStopped);
trace.OnEvent += new TraceEventHandler(OnTraceEvent); trace.Stopped += new TraceStoppedEventHandler(OnTraceStopped); trace.Start();
C#
TRACING ANALYSIS SERVICES.
private void OnTraceEvent(object sender, TraceEventArgs e) {
siAdapter.PutEvent(e); //send the event out to streaminsight dbwriter.putEvent(e); //the dbwriter constructs a load of inserts based on the shape of the event //and dumps to our DB for cube-ness
switch (e.EventClass.ToString()) { case "SessionInitialize": break;
case "ExistingSession": break;
case "QueryEnd": break;
case "QuerySubcube": decodeQuery(e, querySubCubeID); break;
.
.
.
C#
NOW WHERE?
//filters out the events we want CepStream<QuerySumm> querySumm = from e in producer.AlterEventDuration(e => TimeSpan.FromMinutes(1)) where e.eventClass == "QueryEnd" select new QuerySumm { userName = e.userName, allTime = e.duration, cpuTime = e.cpuTime, startTime = e.startTime, endTime = e.endTime }; //detects slow queries so we can go and moan at the user CepStream<SlowAlert> slowProducer = from e in querySumm where e.allTime.Milliseconds > 1000 select new SlowAlert { userName = e.userName, allTime = e.allTime, cpuTime = e.cpuTime, startTime = e.startTime, endTime = e.endTime };
LINQ
STREAMINSIGHT.
LINQ
STREAMINSIGHT. //filters out the events we want CepStream<QuerySumm> querySumm = from e in producer.AlterEventDuration(e => TimeSpan.FromMinutes(1)) where e.eventClass == "QueryEnd" select new QuerySumm { userName = e.userName, allTime = e.duration, cpuTime = e.cpuTime, startTime = e.startTime, endTime = e.endTime }; //detects slow queries so we can go and moan at the user CepStream<SlowAlert> slowProducer = from e in querySumm where e.allTime.Milliseconds > 1000 select new SlowAlert { userName = e.userName, allTime = e.allTime, cpuTime = e.cpuTime, startTime = e.startTime, endTime = e.endTime };
AND THEN?
Output adaptors are nice
• Nagios• Splunk• Homebrew
• MORE CUBES :)
OUTPUT CUBE.
How do we do this?– FACTS– Dimensions
DEMO.
NEXT STEPS: SECURITY.
Security Monitoring / Auditing– Alerting on suspicious querying activity / disallowed querying
Alerting Reporting Analysis
– Provide an audit trail of querying on sensitive attributes– Regulatory Reporting– Dynamic Security
NEXT STEPS: PERFORMANCE.
Performance Recommendations– Provide data to enable assessment of ‘hot’ areas within the cubes
Alerting Reporting Analysis
– Feed into third party monitoring tools– Identify heavy users– Identify poorly performing queries for tuning– Automatic aggregation generation
EXTENDING THE FRAMEWORK.
QUESTIONS.