FUTURESTACK13: Wake Up! A New Day Dawns for Alerts in New Relic from Bill Kayser, Distinguished Engineer at New Relic

Wake Up!A New Day Dawns for Alerts in New Relic

Bill Kayser

Wednesday, November 6, 13

You are always monitoring all the things.

WelcomeBill Kayser

@bravokingbkayser@newrelic.com

Covered in this talk:Overview of alerting in New RelicAlert Policies, other new featuresTips for getting the most out of alerts

What do we expect from alerting systems?

Photo credit: U.S. Coast Guard / Foter / CC BY-NC-ND

Moehre1992 CC BY-SA 3.0

Randy Chiu CC BY 2.0

Zinnmann CC BY-SA 3.0Wednesday, November 6, 13

What to Watch

Define Conditions

Applications

Define Conditions

Applications

KeyTransactions

Define Conditions

Applications

KeyTransactions

Servers

What Action to Take

37 Signals’ Campfire

What Action to Take

Mobile Alerts

What Action to Take

Mobile Alerts

What Action to Take

37 Signals’ CampfireNotification Channels

Mobile Alerts

Thresholds and Notifications - Today

New in New Relic:Alert Policies

New Feature: Alert PoliciesAn Alert Policy specifies:

What to watchWhat action to take

Alert Policies

Alert PoliciesPolicies apply to:

ApplicationsKey TransactionsServers

Alert Policies

Threshold settings and other configuration

Alert Policies

Notification channels

Alert Policies

Server policies are mapped to servers

Notification channels

ops-puppet-01

Application Alert Policy

Server Alert Policy

Key Transaction Alert Policy

Editing Alert Policies

Alert Policies

Named Policies

Alert Policies

Default Policy

Default Policies

Applications with defaults

Default Policies Default settings

Mapping Policies

Defining Channels

Channel SettingsDefine distinct endpoints for each channel type

Designate channels to be used strictly for downtime alerts

Notification GroupsDefine a set of notification channels for use across multiple policies

Include e-mail addresses, PagerDuty services, HipChat rooms, etc

Policies: Putting it All Together

Examining Alerts

What’s the Problem?Problems are events

Critical: tell me right away

What’s the Problem?Problems are events

Critical: tell me right awayCaution: note it in supporting detail

Alerts in the Event List

Alert History

Tips for Leveraging New Alerts

‘Alarm fatigue’worries hospitals

San Francisco Chronicle, October 23, 2013

“...a patient’s life could be hanging in the balance.”

Avoid Threshold SensitivityIncrease the lag time for triggering problems

Avoid Threshold Sensitivity

Increase the lag time for triggering downtime

Improve Threshold SensitivityAdd substring requirement to downtime settings

Improve Threshold Sensitivity

T = 800 ms

Apdex Score: 0.84 - 0.91

Mean Response Time

Narrow the ScopeDefine Key Transactions

Watch error rate and apdexCustomize apdex T value

Key Transaction Policies

Ignore Transaction NoiseJava: NewRelic.ignoreTransaction(), @NewRelicIgnoreTransaction

Ruby: newrelic_ignore [ :only => action..] [ :except => action...]

DotNet: NewRelic.Api.Agent.NewRelic.IgnoreTransaction();

PHP: newrelic_ignore_transaction ( )

Python: newrelic.agent.ignore_transaction(flag=True)

Stop waking me up

Stop waking me up(Wake up Jonathan instead)

Alerts API

Alerts V2: Summary of ImprovementsAlert PoliciesNotification ChannelsNotification GroupsServer DowntimeLag time settingNew API

What do we expect from alerting? Get the right people involved at the appropriate time

Use well defined notification channelsGroup channels together for easy re-useTake advantage of PagerDuty

High signal, low noiseAlert on key transactionsEdit lag timesDistinguishing critical and caution problemsAdjusting apdex T

What do we expect from alerting Flexibility

Define policies according to specific operational requirementsUse defaults, define overridesAPI for integration, management and auditing

SimplicityWatch only the things you care about most

Migrating to use Alert PoliciesAccounts will be upgraded in November

One policy will be created for each application and key transactionOne policy will be created for each unique set of threshold settings across all servers

Existing alert behavior is preserved

Bill Kayserbkayser@newrelic.com@bravoking

Thanks!

FUTURESTACK13: Wake Up! A New Day Dawns for Alerts in New Relic from Bill Kayser, Distinguished Engineer at New Relic

alert policieswednesday

alert policies policies

new relic alert policies

server alert policywednesday

default policieswednesday

alert policiesopspuppet

server policies

notifications todaywednesday

Technology

New Relic Fact Sheet · Lew Cirne founded New Relic in 2008...

New Relic WhitePaper Monitoring Mobile Apps

Getting Started with New Relic:

Writing New Relic Plugins: NSQ

FUTURESTACK13: Nerd Life Balance from Nick Floyd, Senior...

New Relic Online Technical Community...New Relic Online...

FUTURESTACK13: Setting up New Relic's Infrastructure from...

APM for Enterprise WhitePaper from New Relic

New Relic Business Enablement Team stephen@newrelic

Ebook DevOps | Por New Relic

New Relic infographic...Of The New Relic Platform for Cloud....

New Relic Partner Program Overview · 2020-04-12 · New...

New Relic Unveils New Relic Insights™: Software Analytics....

Hp bsm new relic integration

Autoscaling, Chef and New Relic

New Relic .NET Agent Overview