NHS Choices: Managing complex infrastructure to deliver critical online services
Presented by Andy Callow, Head of Technology Delivery
2
About HSCIC
3
The Health and Social Care Information Centre (HSCIC) was formed in April 2013 as an executive non-departmental public body
and the national provider of information, data and IT systems for patients, service
users, clinicians, commissioners, analysts and researchers in health and social care.
About HSCIC
4
• We have responsibility for a number of national systems, including Spine, NHSMail, Electronic Referrals and GPs Systems of Choice and NHS Choices to name just a few.
• April 2016 - Announced moving to new operating name NHS Digital
HSCIC - What We Do
5
We provide information, data and IT Systems for commissioners, analysts and clinicians in health and social care. Our work includes…
• Setting standards that protect patient’s confidential information, reduce bureaucracy and improve data quality
• Operating essential technology services that support the health and care system
• Collecting, analysing and publishing national data and statistical information that helps inform decision making
• Developing the next generation of national data and information systems
About NHS Choices
6
• UK’s biggest health website• More than 14,000 regularly
updated articles and hundreds of videos and
interactive tools• Compare health services in
England
6
About NHS Choices
7
About NHS Choices
8
Our Build
9
Our Original Challenge
10
Good coverage
48MVisits/ Month
Massive Logs
Nov 2010
• Tried out free version
June 2011
• 5GB/day license
Today• Multiple use
cases
Future• Wider
use
Our Splunk Journey
11
Live Performance Stats on Display
Live Performance Stats on Display
• Public Health England campaigns dashboard:
13
Live Performance Stats on Display
14
Live Performance Stats on Display
15
Live Performance Stats on Display
16
Internet
CDN
Web Application Firewall/Reverse Proxy
Cloud Provider
Search Provider (SaaS)
Legacy Network (IaaS)
Load Balancers
Windows Web
Servers
Linux Web Servers
Database Clusters
Cloud Web Services (PaaS)
Cloud Database Services (PaaS)
Splunk
Database Load Balancers
Indexers
Single Search Head
Searching
Splunk TopologyReporting Alerting Dashboards
NHS Choices
Data segmented by user
Our Data Sources
19
Online Services
Web Server Logs
CDN (Akamai)
Windows Event Logs
Mail Server Logs
Performance Counters from Windows & LINUX
RSS Logs
Searching Reporting Alerting Dashboards
Our Use Cases
Troubleshooting
Monitoring Impact of
Changes to the Website
Managing Unpredictable
Traffic
Reporting to Management
20
Troubleshooting
21
Troubleshooting: Root Cause Analysis
DDOS attack blocked by CDN
Reviewed data for previous year
Investigation = no previous attack
Answer the same day
Troubleshooting: Root Cause Analysis
Page rating averages dropping
Used Splunk to investigate
Identified set of IP addresses in Germany
Blocked IPs & introduced dashboard
Troubleshooting: Root Cause Analysis
24
Troubleshooting – using alerts
25
Monitoring Impact of Changes
26
Managing Unpredictable Traffic
27
Managing Unpredictable Traffic
28
Reporting to Management
29
MonitoringData
combined over period
GroupedResults
grouped into interest areas
Availability Actual vs Target
Future Plans
30
Analytics team: Monitor real time
searches
Partner team: Track usage of content
provided to syndication
partners
Product Owners: Build product
specific dashboards
DevOps team: ingest data from
other popular tools
• Make data available to the business
Value from Splunk
More informed decisions
Identifying business value in the data we already have
Alerts and reports free up
time
Correlate previously
unconnected events
Real-time and historical analysis
31
Thank you
32