Top Banner
Cloud monitoring using CHR and Big-Data
40

using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

Aug 31, 2018

Download

Documents

truongdien
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

Cloud monitoring using

CHR and Big-Data

Page 2: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

SummaryAbout Us

General background on CHR

CHR Amazon EC2 installing

CHR Use cases

Cloud monitoring elements

Reporting , Alerting and Trigger

Page 3: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

<whoami

2

Shlomi GutmanCTO of Voicenter (Israel)

VP of Cloud Products at QXIP (Amsterdam)

Page 4: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

3

Voicenter is A leading telecommunication technology company providing top-tier business telephony since 2007

We are delivering a ‘One-stop-shop’ solution for business all around the world

TelecomServices

PBX Call CenterSolution

Page 5: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

4

QXIP - Voice Capture Engineering & Development

QXIP {QuickSIP} is an R&D Company specializing in Open-Source and Commercial Voice Technology Development.

Page 6: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

5

What’s CHR?

Cloud Hosted Router (CHR) is a RouterOS version intended for running as a virtual machine.

It supports the x86 64-bit architecture and can be used on most of the popular hypervisors such as VMWare, Hyper-V, VirtualBox and others.

CHR has full RouterOS features enabled by default but has a different licensing model than other RouterOSversions.

Page 7: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

6

CHR Licensing License

Perpetual is a lifetime license -buy once, use forever .

It is possible to transfer a perpetual license to another CHR instance.

License Speed limit Price

Free 1Mbit FREE

P1 1Gbit $45

P10 10Gbit $95

P-Unlimited Unlimited $250

If the CHR instance will not be able to access the account server to renew the license ,it will behave as if the trial period has ran out and will not allow an upgrade of RouterOS to a newer version.

Page 8: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

CHR hosting environment

Page 9: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

8

Installing CHR on AWS – Step 1

Page 10: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

9

Select CHR Image (AMI)– Step 2

Page 11: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

10

Select Instance Tab – Step 3

Page 12: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

11

Setup your network – Step 4

Page 13: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

12

Configure – Step 5

Page 14: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

13

Installing CHR on AWS – Step 6

Page 15: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

15

Finally… Winbox ... IP… Connect ...

Page 16: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

16

Change Password !!!

Page 17: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

17

CHR - Use case Types

• Virtual Instance

Custom hardware

Management - Dude ,RADIUS

Labs setup

Page 18: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

18

Virtualization – CHR vs x86Why use the CHR instead of the traditional x86 VM?

• Optimized for Virtualization 64 bit support Fastpath support Driver support

• Paravirtualized NIC –Using the CHR allows us to use the a paravirtualized NIC which is capable of speeds beyond 10 Gbps. The E1000 NIC used in the x86 VM is only capable of 1Gbps.

• Future proof – The CHR will continue to be developed

Page 19: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

19

CHR - Use case Types

• Cloud Connectivity VPN cloud - Road Warrior

Direct Connect alternative

Secure distributed cloud environment

Page 20: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

20

CHR - Use case Types

• Cloud monitoring Cyber Defense

Billing Logic on Steroids

Centralized Log Analyze

Page 21: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch
Page 22: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

21

Cyber crime top 20 countries attracts

Page 23: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

22

IOT – the missing S

Page 24: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

23

General background on cyber attracts

Page 25: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

24

Who is behind cyber crime ?

Page 26: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

25

Page 27: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

26

Page 28: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

27

How to ship your data(Syslog) …..

Page 29: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

28

How to ship your data (NetFlow)

/ip traffic-flowset cache-entries=4M enabled=yes

interfaces=BRIDGE

/ip traffic-flow targetadd dst-address=66.66.66.66

port=1234 version=5

Page 30: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

29

Shipping Big Data Log• paStash is a tool to manage spaghetti I/O

with input, processors and output. • modules for all seasons and protocols.

https://github.com/sipcapture/paStash

Page 31: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

30

PaStash Config

Input plugins•File

•Syslog

•ZeroMQ

•Redis

•HTTP

•Websocket

•TCP / TLS

•Google app engine

•AMQP

•SQS

•NetFlow

•Freeswitch ESL

•Asterisk AMI

Outputs•ZeroMQ

•ElasticSearch

•Statsd

•Gelf

•File

•HTTP Post

•Websocket

•Redis

•Logio

•TCP / TLS

•AMQP

•SQS

•HEP

Filter plugins•Regex

•Grok

•Mutate Replace

•Grep

•Reverse DNS

•Compute field

•Compute hash

•Compute date field

•Split

•Rename

•Multiline

•Json fields

•Geoip

•Eval

•Bunyan

•HTTP Status Classifier

input {udp {host => 0.0.0.0port => 514type => syslog

}}

filter {regex {regex => /^(\S)/+/fields => [toto]

}}

output {elasticsearch {host => localhostport => 9200

}}

https://github.com/sipcapture/paStash/blob/master/docs/inputs/file.md
https://github.com/sipcapture/paStash/blob/master/docs/inputs/syslog.md
https://github.com/sipcapture/paStash/blob/master/docs/inputs/zeromq.md
https://github.com/sipcapture/paStash/blob/master/docs/inputs/redis.md
https://github.com/sipcapture/paStash/blob/master/docs/inputs/http.md
https://github.com/sipcapture/paStash/blob/master/docs/inputs/ws.md
https://github.com/sipcapture/paStash/blob/master/docs/inputs/tcp_tls.md
https://github.com/sipcapture/paStash/blob/master/docs/inputs/gae.md
https://github.com/sipcapture/paStash/blob/master/docs/inputs/amqp.md
https://github.com/sipcapture/paStash/blob/master/docs/inputs/sqs.md
https://github.com/sipcapture/paStash/blob/master/docs/inputs/netflow.md
https://github.com/sipcapture/paStash/blob/master/docs/inputs/esl.md
https://github.com/sipcapture/paStash/blob/master/docs/inputs/ami.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/zeromq.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/elasticsearch.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/statsd.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/gelf.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/file.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/http_post.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/ws.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/redis.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/logio.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/tcp_tls.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/amqp.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/sqs.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/hep.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/regex.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/grok.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/mutate_replace.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/grep.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/reverse_dns.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/compute_field.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/compute_hash.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/compute_date_field.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/split.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/rename.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/multiline.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/json_fields.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/geoip.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/eval.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/bunyan.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/http_status_classifier.md
Page 32: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

31

Parsing Mikrotik Netflow

Page 33: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

32

Parsing Mikrotik Log

Page 34: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

33

Mikrotik Netflow Dashboards

Page 35: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

33

Mikrotik Logs Dashboards

Page 36: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

34

ElasticsearchElasticsearch is a search engine based on Lucene. It provides a distributed,multitenant-capable

full-text search engine with an HTTP web interface and schema-free JSON documents.

Page 37: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

35

Siren alerting & reporting application

SENTINL extends Siren with Alerting and Reporting functionality to monitor, validate and inform users and administrators on data series changes using standard queries or join queries, programmable validators, transformers and messages to send out using a variety of configurable actions including sending action to the Mikrotik API as well as sending Emails, Slack Messages, API Webhooks, PDF Snapshots of Charts, creating new Documents and much more.

Page 38: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

33

Siren Alerting & Reporting App

Siren Enterprise provides many unique features and enables integrators to realize unique Business Intelligence creatures. With such power, automating workflows and being able to get notified with data detections quickly becomes a key requirement.

Page 39: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

[email protected] us today for a month free trial

Pushing your data out of the box

Page 40: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

THANK YOU!


Related Documents
Step By Step Konfigurasi Mikrotik HotSpot.pdf

Step By Step Konfigurasi Mikrotik HotSpot.pdf

Category: Documents
Elasticsearch Basics

Elasticsearch Basics

Category: Technology
Step by Step Konfigurasi Mikrotik HotSpot @ Simple Mode _ Sekedar Berbagi.

Step by Step Konfigurasi Mikrotik HotSpot @ Simple Mode _...

Category: Documents
MikroTik Labs - Jobsheet MikroTik

MikroTik Labs - Jobsheet MikroTik

Category: Documents
VPN PPTP client configuration – Mikrotik RouterOS7330130.ir/attachments/article/74/pptp clint mikrotik.pdfVPN PPTP client configuration – Mikrotik RouterOS In the first step, login

VPN PPTP client configuration – Mikrotik...

Category: Documents
Elasticsearch Documentation€¦ · Elasticsearch Documentation, Release 1.4.0 Official low-level client for Elasticsearch. Its goal is to provide common ground for all Elasticsearch-related

Elasticsearch Documentation€¦ · Elasticsearch...

Category: Documents
Step by Step Mikrotik

Step by Step Mikrotik

Category: Documents
[Step by Step] Buat Router Mudah Dengan MikroTikdslrouter.sourceforge.net/stuff/mikrotik/mikrotik/buat-router... · HotSpot Gateway with RADIUS authentication and accounting; ...

[Step by Step] Buat Router Mudah Dengan...

Category: Documents
[Step by Step] Buat Router Mudah Dengan MikroTik

[Step by Step] Buat Router Mudah Dengan MikroTik

Category: Documents
Step by Step Membangun VPN Server Dgn Mikrotik

Step by Step Membangun VPN Server Dgn Mikrotik

Category: Documents
STEP BY STEP INSTALLASI MIKROTIK ROUTER OS DI GNS3 ...ilmuti.org/wp-content/uploads/2017/05/STEP-BY-STEP-INSTALLASI... · adalah Router OS Mikrotik-5.20_Full_Lev6, rename menjadi

STEP BY STEP INSTALLASI MIKROTIK ROUTER OS DI GNS3...

Category: Documents
STEP BY STEP CARA KONFIGURASI HOTSPOT DI MIKROTIK …ilmuti.org/.../2018/03/RAFIK-STEP-BY-STEP-CARA-KONFIGURASI-HOTSPOT... · menambah pengetahuan kita cara konfigurasi hotspot di

STEP BY STEP CARA KONFIGURASI HOTSPOT DI MIKROTIK...

Category: Documents