Multi-Tenant Log Analytics SaaS Service using Solr: Presented by Chirag Gupta & Srivatsan Parthasarathy, Microsoft

O C T O B E R 1 1 - 1 4 , 2 0 1 6 • B O S T O N , M A

Multi-Tenant Log Analytics SaaS Service using SOLR Chirag Gupta (Principal Software Engineer)

Srivatsan Parthasarathy (Partner Software Engineer) Microsoft Corp.

Solutions

Search & Analytics

Portal/API

Microsoft Log Analytics

Real-time search Patching

Configuration Change Tracking

Linux Solutions

Monitoring App Solutions

Community Gallery

Security Audit Data Protection Dashboards

Integrations/Collectors Log Analytics Service

Solution Platform

Microso'  Log  Analy0cs  Solu0on  Pla4orm

5

02

DEMO Microsoft Log Analytics

Aka OMS (Operations Management Suite)

6

02

•  50K+  customers  with  10M+  shards

Usage Statistics

•  20PB  searchable  documents  •  188  million  queries  per  week  

7

02 Challenges •  SOLR  Cloud  boDlenecks  

•  MulE-­‐tenant  support  

•  Performance  issues  with  wide  queries  

8

02 SOLR Cloud

IngesEon  

Azure  File  Shares

SOLR  Clusters

SOLR  

Query  

Cluster  State  

Microso'  Log  Analy0cs  Cloud

Gateway  

SOLR  Query  Merger  

SOLR  Cluster  

Query Inges0on

Predictable  Load Single  Customer/

Workload

SOLR  Workloads

Light  Load Moderate  Load Heavy  Load

Long  tail  of  large  and  small  customers Growing/shrinking  without  warning

IngesEon  

SOLR  Clusters

Azure  File  Shares

Cold  Cluster1  SOLR  

Query  

Microso'  Log  Analy0cs  Workload  Management

Cluster  State  

Gateway  

SOLR  Query  Merger  

Management  

SOLR  Cluster  

Wide  Range  Query

SOLR  Query

Query  

Cold  Cluster1  SOLR  

Hot  SOLR  Clusters

Azure  File  Shares

Cold  Cluster1  SOLR  

Cold  Cluster

Microso'  Log  Analy0cs  Cold  Query

Cluster  State  

Gateway  

SOLR  Query  Merger  

14

02 Recap •  Used  external  durable  storage  vs  replicaEon  

•  Central  scoped  config  with  ApplicaEon  Load  Balancing  vs  

Zookeeper  

•  Shard  Lifecycle  Management  as  a  separate  component  

•  Cold  Storage  based  Queries  

15

02 Next Steps •  Further  opEmize  SOLR  for  Log  AnalyEcs  Scenario  

•  IngesEon  pipeline  

•  AnalyEcs  queries  

16

02

Q/A

17

02

Thank You Check out Microsoft Log Analytics @ http://microsoft.com/oms Blogs @ https://blogs.technet.microsoft.com/msoms/