R11 Event Management Scalability Tests -. © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced.

r11 Event ManagementScalability Tests

-

© 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Objectives

- To run scalability tests to ascertain scalability limits such as the number of events that can be processed with different MRA, impact of AMS, AEC, etc

Test 1- MRA Processing on Local Node


Test 1 - Goal

- Analyze impact of MRA with different levels of Message Storm.

- Any MRA with the exception of FORWARD or ALERT


Test 1a – Half Million

- Generate message storm with half a million events and review latency period on the last event

- MRA = SendOper

- This is to simulate event storm that may be generated by back bone router going down


Test 1a - 500,000 Events Storm with SendOper MRA

Latency:

2 mins 26 seconds


Test 1a

- Confirm half millions events processed by Event Management


Test 1b - 250,000 Events Storm with SendOper MRA

125,000 Events

Latency 40 seconds

50,000 Events

Latency 21 seconds


Test 1b - 250,000 Events Storm with SendOper MRA

250,000 Events

Latency 74 seconds


Test1b - Repeated

- Repeat Test1b on a different server to verify the latency period is consistent


Test1b - Repeated

250,000 Events

Latency 74 seconds50,000 Events

Latency 21 seconds


Conclusion

- MRA without Forward can process approximately half million events message storm with last event latency of approx 2 mins.

- 50,000 Event message storm has no significant delay

Test 2- MRA processing on local node with steady flow of events


MRA Test

- Generate 500 events per second and process one MRA for each on the local node

- Analyze if any latency


Generate 500 events per second


500 Events Per Second

First Last

No Latency


MRA Test

- Generate 500 events per second and process two MRA per event on the local node

- Analyze latency


2 MRA Test – 500 Events per Second

Confirms no latency by adding extra MRA


MRA Test

- Generate 1800 events per second and accelerate by 10% compound every 30 second

- Analyze latency


Accelerated Event Storm


One Minute Analysis


Latency

LastFirst

860010 events in 345 seconds , average of 2495 events per seconds


Conclusion

- 2495 events per second for local node MRA processing with exception of Forward, Alert or command

Test 3- Event Storm Forward to Central Node


Test 3 - Goal

- Analyze impact of FORWARD MRA with different levels of Message Storm.

- On the central console, “No ALERTS” (AMS)


r11 Configuration

Server_229 Server_230

Server_405

Central Server

Event Agent Event Agent


Central Server Machine Spec


Test 3

- Forward One Million Events from Server 229 to Central Console

- At the same time forward another million Events from Server 230 to Central Console


Shows 1 million events each from 2 nodes


Latency – 2 Millions Forward - One million events forwarded from each of the 2 servers. In

total 2 millions events forwarded to the central node.

Latency of 1 hour 12 mins

Latency of 1 hour 12 mins

Server 229Server 230


Conclusion

- Forward MRA can process 1 million events each from 2 different server but will have significant delay

Test 4- Steady Flow Events with Forward to Central Node


Test 4 - Goal

- Analyze impact of FORWARD MRA with accelerated Event Message storm

- On the central console, “No ALERTS” (AMS)


Test 4

- Generate 200 events per second from each server and accelerate the rate by 20 events every 60 seconds.


Generate Message Storm

Server 229

Server 230


Confirm Counts


Reports shows Events per minute


5 mins – Server One- In 5 mins 30 seconds, 72,300 events forwarded from

Server 230.

- Average 220 events per seconds forwarded from server 230

Server 230

Accelerated rate of 300

Latency: 30 seconds


5 mins - Server Two- In 5mins, 72,300 events forwarded from Server 229 and in

total 144600 events received by the central node.

- Average 433 events per second received by the central node

Server 229

Accelerated rate of 300

Latency: 31 seconds


10 mins - Server One

- In 13 mins, 74000 events forwarded from Server 230.

Server 230

Latency 3 min 16 secs

Accelerated rate of 380 per second


10 mins - Server Two

- In 13 mins, 174000 events forwarded from Server 229, average of 220 events per second

- 348,000 Events received by the central node in 13 minutes, average of 437 events per second

Server 229

Latency 3 min 16 secs

Accelerated rate of 380 per second


Events processed per minute

This shows the number of events received by the central node


Conclusion

- In 13 mins, 174000 events forwarded from each node.

- The maximum latency of 3 mins 16 seconds

- On Average, 220 events per second can be forwarded from each node without any delay

- Analysis of Events processed per minute shows on average of 445 events per seconds received by the central node.

Test 4 Extension- “Forward” MRA


Test 4 Extension

- Test 4 confirms 440 events per second received from 2 nodes.

- Does this mean, one can FORWARD 440 events per second from a single node only?

- No!!!.

- If you forward 440 events from one node, you will experience significant latency


Test 4 Extension

- Generate a steady flow 221 events per second and forward it to a central node


Test 4 Extension

Shows 221 events per second forwarded from a single node to the central node with latency of 9 seconds


Test 4 Extension

- Generate 440 events per second from a single node and forward it to the central node.

- No events forwarded from any other nodes except this one.


Generate 440 Events per second


440 Event per second Forward from one Node

78581 events in 6 minutes , generating average of 218 events per second


Conclusion

- Average of 220 events per second can be forwarded from each node.

Test 5- “ALERT” MRA


Test 5 - Goal

- Analyze impact of ALERT MRA

- Test5a – Event Storm 10,000 events

- Test5b – 20 events per second with Consolidate


DIA Zones


MCC Alert Queue Display


Alert MRA


Alert Queues


Test4a –10,000 Events Storm with Alert MRA

After 1000 Events

Latency : 37 Seconds



After 2500 Events

Latency : 3 mins 4 seconds



10,000 Events

Latency of approx 34 mins


Conclusion

- If message storm is expected, another layer to handle message storm should be used. That layer should handle the message storm. Example, use AEC to handle message storm and get AEC to generate message which triggers ALERT MRA

Test 5b- “ALERT” MRA- Controlled Rate


Test 5b – Controlled Rate

- Generate 20 events per second for 60 seconds.


Alert Queue - Consolidate

- Events on the Alert Queue are consolidated to cut down the number of entries on MCC


Test 5b – After 800 events

Latency of 67 seconds


Test 5b – After 1200 Events

Latency: 2 mins 49 seconds


Conclusion

- For ALERT MRA, filtering should be used if message storm is expected.


Conclusion

- 2495 events per second for local node MRA processing with exception of Forward, Alert or command

- On Average, 220 events per second can be forwarded from each node without any significant delay

- Message storm of 50,000 events has no significant delay

- 250,000 events storm is handled with latency of approx 74 seconds.

- 500,000 message storm is handled with latency of approx 2 mins on the last event


Questions??

- Mail to:

- Yatin Dawada- [email protected]

- Bill Merrow

- [email protected]

R11 Event Management Scalability Tests -. © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced.

Documents