QlikView Server Monitor - Kinesis to Redshift to Qlik - QWC Nov 2014

Enterprise Intelligence

Enterprise IntelligenceKinesis

to

Redshift

to

Qlik

Technology Partners:

http://www.ipc-global.com/company/about-us/


PresenterPete Zybrick – Enterprise Solutions Architect

Over 30 years of experience designing and delivering

complex software solutions.

From Bell Labs to Big Data, Pete has architected,

managed, tested and implemented large scale mission

critical systems directly responsible for billions of

dollars in annual transactions.

As the leader of the Big Data technical programs within

IPC Global, Pete is responsible for building a

framework of collaboration between IPC Global and our

technology partners, Cloudera and AWS.

Kinesis to Redshift to Qlik

http://www.ipc-global.com/


IPC Global’s AWS Certifications

AWS Certified Solutions Architect – Professional is a

certification that recognizes IT professionals that possess

unparalleled skills and the highest technical knowledge

necessary for designing, deploying, and managing

applications on the AWS platform.

AWS Certified SysOps Administrator – Associate Level is

a certification that validates IPC Global’s technical expertise

in deployment, management, and operations on the AWS

platform.


AWS Certified Developer – Associate Level is a

certification that validates IPC Global’s technical expertise in

developing and maintaining applications on the AWS

platform.



Objectives

• Provide QlikView Server Users with Near 100% Availability

• Provide QlikView Server Support Staff with Automation and Status Visibility

• Automated QlikView Service Restarts

• Capture QlikView Audit, Events, Performance and Session logs concurrently from multiple QlikView servers and store in central relational database

• Create a new Availability log, log message is written every 30 seconds describing the status of each QlikView service and Cluster Component (i.e. QMS, QDS, etc.)

• Process Rules against log messages in real time and take action(s)

• QlikView Dashboard to display key QlikView service availability metrics

• Scalable, Highly Available, Secure




AWS Technologies

• Virtual Private Cloud (VPC): Virtual Network to Manage Instances and Access

• Elastic Compute Cloud (EC2): Windows Server(s) and Linux instance(s)

• Kinesis/Kinesis Client Library (KCL): Distributed High Volume Message Delivery

• Simple Naming Service (SNS): Send Messages via Email, SMS and SQS

• Simple Queue Service (SQS): Point to Point Guaranteed Message Delivery

• Integrated Access Management (IAM): Security and Policy/Permission Management

• Redshift: Petabyte-Scale Columnar Relational Database






Scenarios

• Scenario 1: Basic Flow

• Scenario 2: Service Down, Restart Failure

• Scenario 3: Service Down, Restart Successful




Scenario 1: Basic Flow

MultiLogMgr

TailLog

QVS - First

Audit

Events

Sessions

Performance

QlikView

Server

ipcsvcmon Availability

QVS - Last

. . .

Audit

Events

Sessions

Performance MultiLogMgr

TailLog

QlikView

Server



Kinesis

MultiLogMgr.KinesisApp

Kinesis Client Library

RuleHandler

EC2 Instance 1



RuleHandler

EC2 Instance n

SNS

Redshift

SvcMon

Dashboard

mlmweb



Scenario 2: Service Down, Restart Failure

MultiLogMgr

TailLog

QVS - First

Audit

Events

Sessions

Performance

QlikView

Server



Kinesis



RuleHandler

EC2 Instance 1

SNS

Redshift

SQS

queue

service_name=QDS@uset1b-qlikview

service_status=CouldNotContact

QDS@uset1b-qlikview Down




MultiLogMgr

TailLog

QVS - First

Audit

Events

Sessions

Performance

QlikView

Server



Kinesis



RuleHandler

EC2 Instance 1

SNS

Redshift

SQS

queue


service_status=RestartAttempt




MultiLogMgr

TailLog

QVS - First

Audit

Events

Sessions

Performance

QlikView

Server



Kinesis



RuleHandler

EC2 Instance 1

SNS

Redshift

SQS

queue


service_status=Dead

QDS@uset1b-qlikview Dead

IPC Global 24x7 Support



Scenario 3: Service Down, Restart Successful

MultiLogMgr

TailLog

QVS - First

Audit

Events

Sessions

Performance

QlikView

Server



Kinesis



RuleHandler

EC2 Instance 1

SNS

Redshift

SQS

queue


service_status=CouldNotContact

QDS@uset1b-qlikview Down




MultiLogMgr

TailLog

QVS - First

Audit

Events

Sessions

Performance

QlikView

Server



Kinesis



RuleHandler

EC2 Instance 1

SNS

Redshift

SQS

queue


service_status=RestartAttempt




MultiLogMgr

TailLog

QVS - First

Audit

Events

Sessions

Performance

QlikView

Server



Kinesis



RuleHandler

EC2 Instance 1

SNS

Redshift

SQS

queue


service_status=RestartSuccessful



Dashboard

• Online Demo




Components

• On each QlikView Server• Dashboard

• ipcsvcmon: monitor QV services, write status to Availability log, process Restart requests (.Net, Windows Service)

• TailLog: tail the Audit, Availability, Events, Performance and Sessions logs, send to MultiLogMgr (Java, Windows Service)

• On Linux instance(s) • MultiLogMgr: process all inbound log messages from all source servers, write messages

to database, process rules, send emails/SMS messages, send ServiceStatus messages to ServiceMonitor (Java, linux daemon)

• mlmweb: Web application to maintain Rules and Source Systems (JSF2, JPA2, Tomcat7)




Summary

• Proactive

• Automation

• Visibility

• Near 100% Availability for IPC Global Hosted Clients






Technology Partners:

Kinesis

To

Redshift

To

Qlik



QlikView Server Monitor - Kinesis to Redshift to Qlik - QWC Nov 2014

Documents

aws platform

secure kinesis

successful kinesis

multiple qlikview servers

sqs simple queue service

log messages

big data technical programs

new availability log