Top Banner
Copyright © 2015 Splunk Inc. Keisuke Noda Takeshi Suzuki Splunk as a Service at Rakuten 1
45

Splunk in Rakuten: Splunk as a Service for all

Feb 15, 2017

Download

Retail

Timur Bagirov
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Splunk in Rakuten: Splunk as a Service for all

Copyright  ©  2015  Splunk  Inc.  

Keisuke  Noda  Takeshi  Suzuki  

Splunk  as  a  Service  at  Rakuten  

1  

Page 2: Splunk in Rakuten: Splunk as a Service for all

Disclaimer  

2  

During  the  course  of  this  presentaEon,  we  may  make  forward  looking  statements  regarding  future  events  or  the  expected  performance  of  the  company.  We  cauEon  you  that  such  statements  reflect  our  current  expectaEons  and  esEmates  based  on  factors  currently  known  to  us  and  that  actual  events  or  results  could  differ  materially.  For  important  factors  that  may  cause  actual  results  to  differ  from  those  contained  in  our  forward-­‐looking  statements,  please  review  our  filings  with  the  SEC.  The  forward-­‐looking  statements  made  in  the  this  presentaEon  are  being  made  as  of  the  Eme  and  date  of  its  live  presentaEon.  If  reviewed  aQer  its  live  presentaEon,  this  presentaEon  may  not  contain  current  or  

accurate  informaEon.  We  do  not  assume  any  obligaEon  to  update  any  forward  looking  statements  we  may  make.    

 In  addiEon,  any  informaEon  about  our  roadmap  outlines  our  general  product  direcEon  and  is  subject  to  change  at  any  Eme  without  noEce.  It  is  for  informaEonal  purposes  only  and  shall  not,  be  incorporated  into  any  contract  or  other  commitment.  Splunk  undertakes  no  obligaEon  either  to  develop  the  features  

or  funcEonality  described  or  to  include  any  such  feature  or  funcEonality  in  a  future  release.  

Page 3: Splunk in Rakuten: Splunk as a Service for all

About  Us  

  Name  –  Keisuke  Noda  –  野田 啓介  

  PosiEon  –  Architect  /  Manager  –  Data  Store  PlaYorm  Group  

  Background  –  ApplicaEon  engineer  –  Database  engineer  

  Like  –  Massage  

3  

Page 4: Splunk in Rakuten: Splunk as a Service for all

About  Us  

4  

  Takeshi  Suzuki    鈴木 武    Tokyo      Rakuten,  Inc.    Security  OperaEons  Group    Security  engineer  /  Manager  

Page 5: Splunk in Rakuten: Splunk as a Service for all

About  Company  Founded:    February  7,  1997  IPO:    April  19,  2000  (JASDAQ  Stock  Exchange)  Office:    Rakuten  Tower  (Tokyo,  Japan)  Employees:          12,288  (as  of  June,  2015)    Market  Cap:        JPY  214,701  million  (as  of  Sep  15,  2015)  

5  

Page 6: Splunk in Rakuten: Splunk as a Service for all

About  Company  

6  

Page 7: Splunk in Rakuten: Splunk as a Service for all

Going  Global  

7  

Page 8: Splunk in Rakuten: Splunk as a Service for all

Agenda  •  Why  Splunk?  •  Why  is  Splunk  offered  as  a  Service?  •  Service  Overview  •  Our  Challenges  •  Current  Status  •  Case  Studies  •  What’s  Next?  •  Wrap  up  •  Q  and  A  

8  

Page 9: Splunk in Rakuten: Splunk as a Service for all

Why  Splunk?  Why  is  Splunk  offered  as  a  Service?  

9  

Page 10: Splunk in Rakuten: Splunk as a Service for all

Why  Splunk?  

•  Summer  2011…  I  discovered  Splunk  •  Cool  visuals  •  Looks  interesEng  

10  

Page 11: Splunk in Rakuten: Splunk as a Service for all

Why  Splunk?  

•  Self-­‐made  database  monitoring  system  •  Legacy  and  complex  system  

Batch  Server RDBMS Web  ApplicaEon

Add  a  column Modify  codes  

Modify  codes

Add  a  column

Input  the  Data  into  RDB   Store  the  Data   Visualize  the  Data  Output  

Database  Status  

Before  

11  

Page 12: Splunk in Rakuten: Splunk as a Service for all

Why  Splunk?  

•  Self-­‐made  database  monitoring  system  •  One  Splunk  is  simple  

Input  Data  /  Store  Data  /  Visualize  Data  Output  Database  Status  

Then,  Splunk  began  to  be  used  in  various  groups…

So  Easy!!

All  in  One!

Cool  Visuals!!

AQer  

12  

Page 13: Splunk in Rakuten: Splunk as a Service for all

Why  is  Splunk  offered  as  a  Service?  

.  .  .    Splunk  as  a  Service  was  born  

13  

•  Splunk  began  to  be  used  in  various  groups  •  There  were  so  many  repeEEve  operaEons            (such  as  license  management,  system  construcEons,  operaEons  …etc)  

If  there  is  one  big  plaYorm  and  everyone  can  use  it  without  management,  the  problem  will  be  solved.  

 In  addiEon,  it  may  have  many  other  benefits…  

Page 14: Splunk in Rakuten: Splunk as a Service for all

Service  Overview  

14  

Page 15: Splunk in Rakuten: Splunk as a Service for all

Service  Overview  

Corporate  IT   …  Merchant   Security  

Server  

Example  

15  

Dep.  A   Dep.  B  

Marketplace   Credit  Card  

E-­‐money  

Database  

Network  

Dep.  C   Dep.  D   Dep.  E  

•  Rakuten’s  organizaEon  •  There  are  so  many  departments  and  groups  

Page 16: Splunk in Rakuten: Splunk as a Service for all

Service  Overview  

Admin   User  

…  Network  

Security  

Credit  Card  

Corporate  IT  

My  Group  

16  

•  Groups  of  Splunk  as  a  Service  •  Admin  •  User  

Page 17: Splunk in Rakuten: Splunk as a Service for all

Service  Overview  

•  No  need  to  manage  Infrastructure  •  Design,  construcEon,  monitoring,  operaEon  and  license  

•  Easy  to  start  Splunking  in  a  few  minutes  without  detailed  configuraEon  

•  Charged  by  measured  rate  •  High  availability  

•  99.99%  upEme                  Rakuten  Splunk  as  a  Service

Will  be  talked  details  later

For  user  

17  

Page 18: Splunk in Rakuten: Splunk as a Service for all

Service  Design  

•  Environment  •  Private  Cloud  •  High  availability  •  On  Eme  delivery  •  Flexibility  

18  

Page 19: Splunk in Rakuten: Splunk as a Service for all

Service  Design  

•  System  configuraEon  •  v6.2.X  •  Using  an  indexer  cluster  •  Full  components  •  Newer  data  on  Low  latency  Storage  (Hotdb),  Older  data  on  Low  Cost  Storage  (Colddb)  

19  

Page 20: Splunk in Rakuten: Splunk as a Service for all

Service  Design  

•  Other  specificaEons  •  Splunk  account  is  created  for  each  user  •  1  user  =  1  group,  1  service,  or  1  project  •  Each  user  has  his/her  own  App  •  Basically  a  user  can  see  only  his/her  own  data  •  Accesses  are  controlled  by  tags  •  Users  can  choose  the  term  of  storage  retenEon  from  1  day  to  6  years  for  each  input  

•  Admin  does  not  do  backups  •  Dedicated  Search  Head  is  ready  for  users  who  need  

20  

Page 21: Splunk in Rakuten: Splunk as a Service for all

Service  OperaEons  

•  System  operaEons  •  Create  user  accounts  (rolls,  users  and  Apps)  •  Set  up  inputs  •  Install  external  Apps  •  Irregular  configuraEon  (props.conf,  transforms.conf,  limits.conf,  …etc)  

•  Service  operaEons  •  User  support  /  ConsultaEon  

•  Monitoring  •  Input  size  •  System  resources  (SoS  /  Unix  App  /  PandoraFMS)  

Admin  side  

21  

Page 22: Splunk in Rakuten: Splunk as a Service for all

Our  Challenges  

22  

Page 23: Splunk in Rakuten: Splunk as a Service for all

Our  Challenges  

1.  Easy  data  access  control  2.  CollaboraEon  with  internal  tools  3.  CollaboraEon  with  global  group  companies  4.  OperaEon  improvements  by  Rakuten  Splunk  Portal  Site  with  API  

23  

Page 24: Splunk in Rakuten: Splunk as a Service for all

Your  network  log  correlates  with  my  syslog.  I  want  to  see  your  logs!

Easy  Data  Access  Control  

  Demand  •  Users  want  to  see  other  user’s  data  

  Measure  •  Use  Tag  (tagUser::  host=<user’s  host>)  •  Add  whitelist  condiEon  in  case  of  sharing  data  

OK

Thank  you.  So  efficient!

User  B User  A

tag=tagA ,  tagB Admin

I’d  love  to.

24  

Page 25: Splunk in Rakuten: Splunk as a Service for all

CollaboraEon  with  Internal  Tools  

Lookup  Script   Internal  tools  

Script  

25  

  Demand  •  Users  want  to  use  some  internal  tools  and  informaEon  from  Splunk  

  Measure  •  Import  CMDB  data  (lookup)  •  Can  receive  direct  phone  calls  from  DC  staff  (alert  script)  •  Can  call  Hipchat/Slack  web  hooks  (alert  script)  

Page 26: Splunk in Rakuten: Splunk as a Service for all

CollaboraEon  with  global  group  companies  

26  

  Demand  •  Users  want  to  access  global  companies’  data  through  Splunk  

  Measure  •  Splunk  as  a  Service  in  the  USA  is  ready  •  Supervisors  can  see  the  whole  data  of  each  region  

Page 27: Splunk in Rakuten: Splunk as a Service for all

OperaEon  Improvements  

27  

  Demand  •  Users  want  to  start  Splunking  easily  in  a  short  Eme  •  Admin  wants  to  make  regular  operaEons  more  efficient  

  Measure  •  Made  Rakuten  Splunk  Portal  Site  for  operaEon  improvements  using  Splunk  REST  API!  

•  Easy  to  start  Splunking  in  just  a  few  minutes  

Page 28: Splunk in Rakuten: Splunk as a Service for all

Rakuten  Splunk  Portal  Site  

•  DemonstraEon  •  Easy  to  start  Splunking  1.  Create  Splunk  web  account  2.  Install  a  forwarder  3.  Set  up  &  deploy  Apps  

28  

Page 29: Splunk in Rakuten: Splunk as a Service for all

Rakuten  Splunk  Portal  Site  

•  Current  main  features  •  Manages  user’s  informaEon  (organizaEon,  emails,  etc..)  •  Creates  Splunk  web  accounts  (create  roles,  users,  and  Apps)  •  Manages  forwarders,  server  classes,  inputs  and  Apps  •  Deploys  Apps  to  users’  forwarders  •  Alerts  users  when  users’  forwarders  are  down  

Good  reputaEon

29  

Page 30: Splunk in Rakuten: Splunk as a Service for all

Rakuten  Splunk  Portal  Site  

•  Do  you  want  to  try  the  Portal  Site  on  your  environment?  

We  are  currently  developing  it  as  an  open-­‐source  project!!                Please  read  README  before  using  it.  

30  

To  be  prepared!!  

Page 31: Splunk in Rakuten: Splunk as a Service for all

Current  Status  

31  

Page 32: Splunk in Rakuten: Splunk as a Service for all

Current  Status  Indexed  Data  Size Availability  Rate

32  

Page 33: Splunk in Rakuten: Splunk as a Service for all

Current  Status  Input  Size #  of  Accounts

33  

Page 34: Splunk in Rakuten: Splunk as a Service for all

Case  Studies  

34  

Page 35: Splunk in Rakuten: Splunk as a Service for all

Case  Studies  

35  

Server  Real-­‐Eme  monitoring  TroubleshooEng  Usage  report

Database  Real-­‐Eme  monitoring  TroubleshooEng  Usage  report  Service  KPI  management

Security  IDS  real-­‐Eme  monitoring  Fraud  detecEon  

Private  Cloud  (RIaaS)  Real-­‐Eme  monitoring  Resource  management  

Applica[on  Real-­‐Eme  monitoring  Service  KPI  management  Performance  management

Storage  Real-­‐Eme  monitoring  Resource  management  Service  KPI  management

Network  Real-­‐Eme  monitoring  TroubleshooEng  Trend  analysis  

More…  

Page 36: Splunk in Rakuten: Splunk as a Service for all

Security  Monitoring  Security

36  

Page 37: Splunk in Rakuten: Splunk as a Service for all

Alert  Email

37  

Security  

Actual  avack  payload

DescripEon  of  avack  

Point  of  contact  

Data  comes  from  other  systems  

Page 38: Splunk in Rakuten: Splunk as a Service for all

Security  Monitoring  

•  Before  •  Analyze  by  Managed  Security  Service  Portal  •  Make  sure  the  right  person  handles  the  incident  

•  AQer  •  By  CMDB,  streamlined  escalaEon  flow  •  Shorten  Eme  for  iniEalizing  acEon  •  Detect  irregular  accesses  

Security

38  

Page 39: Splunk in Rakuten: Splunk as a Service for all

Case  Studies  

39  

Server  Real-­‐Eme  monitoring  TroubleshooEng  Usage  report

Database  Real-­‐Eme  monitoring  TroubleshooEng  Usage  report  Service  KPI  management

Security  IDS  real-­‐Eme  monitoring  Fraud  detecEon  

Private  Cloud  (RIaaS)  Real-­‐Eme  monitoring  Resource  management  

Applica[on  Real-­‐Eme  monitoring  Service  KPI  management  Performance  management

Storage  Real-­‐Eme  monitoring  Resource  management  Service  KPI  management

Network  Real-­‐Eme  monitoring  TroubleshooEng  Trend  analysis  

More…  

Page 40: Splunk in Rakuten: Splunk as a Service for all

Quote  

40  

“You  can’t  connect  the  dots  looking  forward;  you  can  only  connect  them  looking  backward.  So  you  have  to  trust  that  the  dots  will  somehow  connect  in  your  future.  You  have  to  trust  in  something”     -­‐  Steve  Jobs  

Page 41: Splunk in Rakuten: Splunk as a Service for all

Wrap  up  

41  

Page 42: Splunk in Rakuten: Splunk as a Service for all

What’s  Next?  •  Make  it  easier  to  get  Splunk  started  

•  Complete  automaEon  

•  Make  regular  operaEons  more  efficient  •  Change  frequent  operaEons  automaEcally  

•  Upgrade  to  v6.3  •  Enhance  Rakuten  Splunk  Portal  Site  •  Have  more  collaboraEon  with  global  group  companies  

42  

Page 43: Splunk in Rakuten: Splunk as a Service for all

Wrap  up  •  Rakuten  is  using  one  big  Splunk  as  a  Service  

•  PosiEve  advantages  for  user  •  No  need  to  manage  Infrastructure,  License,  and  detailed  configuraEon  •  Can  use  data  of  crossing  organizaEon  

•  PosiEve  advantages  for  admin  •  Can  manage  operaEons  and  license  efficiently  •  Have  many  saEsfied  users  

•  OperaEon  improvements  by  Splunk  Portal  Site  with  API  •  Can  start  Splunking  easily  in  a  few  minutes  

•  Many  different  types  of  users  are  using  Splunk,  and  hopefully  it  will  expand  globally  

43  

Page 44: Splunk in Rakuten: Splunk as a Service for all

QuesEons?  

44  

Page 45: Splunk in Rakuten: Splunk as a Service for all

THANK  YOU