Architecture for monitoring applications in Cloud

Architecture for monitoring ruby on rails applications

in Cloud Onkar kadam

[email protected]

Cloud Environment. Virtualization/Virtual machines. Monitoring Multiple hosts. Monitoring a Distributed environment. Central Monitoring Server. Real-time messaging. Selecting the metrics. Incident reports and notifications.

Challenges

Active Monitoring When the Monitoring server triggers the

check on a host then it is termed as active monitoring.

In an active monitoring environment , the central monitoring server performs periodic checks on hosts and retrieves metrics related to that service.

If the central monitoring server is within the cloud then its easy to monitor the hosts.

If the central monitoring server is at a remote location , ssh services are required to monitor virtual hosts.

Active Monitoring contd..

Scripts or programs on the virtual host perform tests and send the results to the central monitoring server.

Passive Monitoring

Active monitoring doesn’t put the load on the host to run monitoring scripts.

Passive monitoring is effective in case of failures. Passive monitoring fit well in the case of reporting problems to the central monitoring server in real time.

Active vs Passive

The need of an efficient messaging protocol for sending performance metrics to the central monitoring server.

For Real Time Application performance monitoring a fast messaging protocol is a must.

Leveraging already available messaging services to be used in the architecture

RabbitMQ , ActiveMQ , Apache Kafka.

Real-Time Messaging

Architecture -1

In the architecture, the monitoring agent will calculate the performance metrics and publish it on RabbitMQ.

The virtual hosts are the producers application performance metrics and the Central monitoring server is the consumer.

The Central monitoring server will fetch the metrics from the queue of RabbitMQ.

Architecture 1 –contd..

Architecture 2 – Apache Kafka

Instead of point-to-point pipelines use of a central data pipeline.

The producers, in our case the virtual machines hosting the rails application, produce metrics and publish data on the kafka cluster with a key in the message which specifies the source of the metric.

The consumer reads the messages from the kafka cluster(commit log) and then represents the data in graphs for the end user.

Architecture 2 –contd…

Website activity tracking (page views, searches , or other action users perform)

Real time processing and real time monitoring.

Aggregating statistics from applications and produce operational monitoring data.

Apache Kafka –Monitoring Use case

Two ways: Ruby plug-in or a full-fledged rails application’.

The main goal of the agent is to collect critical metrics from the host and send it to the central monitoring server, hence a plug-in will be more suitable and lightweight.

In cloud environment it is necessary to have a lightweight plug-in as the size of an instance varies.

Monitoring Agent Architecture

Monitoring Agent – contd..

Monitoring web servers is important as it is the front end of any web application.

We can monitor the web server by parsing the log files.

Monitoring Web server.

Action Controller receives incoming requests and hands off each request to a particular action. It maps URL to specific actions.

Rails comes with a number of built-in logging facilities.

Use of after_filter to invoke custom logging method for each request.

It will help the monitor to inspect the request cycle in real time.

Monitoring.

Using “~/projects$ ruby script/console –s” we can inspect the relationships between the objects.

Giving the client a view of the these relationships in the front end of the application monitoring agent.

Monitoring Model Relationships

Use of “stats rake” task to generate statistics about the Rails project.

A “Httperf” like tool can be used to generate real time web server performance statistics for a real world web traffic.

A front end abstraction can be used to represent the statistics collected in an graphical view.

Monitoring the web server cache.

Monitoring

The most important part of a rails application is the database.

The most important thing to monitor is the mapping between the database and Active records.

To maintain the integrity of ORM, a change in active record object should cause an immediate change in the corresponding row of a table in a database.

Database Monitoring

Periodically issue a MySql “Show status” command and return.

Metrics such as Mysql traffic, No of transactions between Controller and database, no of queries ,no of update queries, no of delete queries, no of times a sql command is issued, query cache statistics, etc

Database Monitoring

There are various methods an monitoring application can be deployed in the cloud.

One is providing the source code and installing it after a virtual instance with the web application is up an running.

Creating a custom image with the monitoring agent already installed, such that the users only select the image and deploy their applications and start monitoring their application. (http://docs.aws.amazon.com/gettingstarted/latest/wah-linux/getting-started-create-custom-ami.html)

Deployment of the agent in Cloud

The main important decision is to perform active or passive monitoring or combination of both.

To collect statistics about an application it is necessary to dig deeper in the PAAS technology used by the application, for eg. a rails application will have a different set of performance metrics than a java based web application , but both applications will have a default set of metrics which are valid for both type of applications.

Conclusion

The agent should be as light as possible so that it will not affect the web application performance.

Real- time messaging so that the metrics reflect in the monitor in real time

Real time data processing to generate the statistics.

Conclusion contd..

A self aware scaling tool for web applications in cloud.

A performance metric predicting tool based on the past performance metric and current time of the day.

Future Work

THANK YOU