Page 1
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
HLC304
Building IoT Applications
with AWS and Amazon Echo
November 28, 2016
Chris McCurdy - AWS Specialist SA in Healthcare and Life Sciences
Nitin Gujral – Directory of Software Engineering and Innovation at Boston Children's
Page 2
What to Expect from the Session
• What is IoT and what are some patterns that are being
used in industry?
• What are some alternatives if I just want health sensor-
related data?
• How is KidsMD using Alexa to help improve home care?
Page 8
What does AWS IoT consist of?
Security First• All traffic to and from AWS IoT must be encrypted
over Transport Layer Security (TLS)• AWS Managed Certificates
• Certificate Signing Request (CSR) Certificates
• Just-in-time (JIT) Registration
• Protect your certificates• chroot
• Security Enhanced Linux (SELinux)
• One-Time Programmable (OTP) Fuses
• Trusted Platform Modules
• Smartcards
• Locks and Boxes
• FIPS-style hardware
Page 9
What does AWS IoT consist of?
Thing Gateway
Page 10
What does AWS IoT consist of?
Thing Gateway
Supported Protocols
HTTPS, Websockets, Secure MQTT
What is MQTT?
A lightweight pub/sub protocol, designed to minimize network bandwidth and
device resource requirements. MQTT supports TLS for encryption.
Page 11
What does AWS IoT consist of?
Thing Gateway var device = awsIot.device({
keyPath: 'private_key.pem',
certPath: 'certificate.pem',
caPath: 'root.pem',
clientId: clientId,
region: 'us-east-1' });
var button = new groveSensor.GroveButton(6);
function readButtonSensorValue() {
var buttonState = button.value();
if (buttonState == 1) {
device.publish(topic, JSON.stringify({
SubmitTime: (new Date()).toJSON()}));
}
}
setInterval(readButtonSensorValue, 100);
Page 12
What does AWS IoT consist of?
IoT Rules Engine
Amazon
Kinesis
Amazon S3
bucket
Page 13
What does AWS IoT consist of?
IoT Rule Engine• Augment or filter data received from a device.
• Write data received from a device to an Amazon DynamoDB
database.
• Save a file to Amazon S3.
• Send a push notification to all users using Amazon SNS.
• Publish data to an Amazon SQS queue.
• Invoke a Lambda function to extract data.
• Process messages from a large number of devices using
Amazon Kinesis.
• Send data to the Amazon Elasticsearch Service.
• Capture an Amazon CloudWatch metric.
• Change an Amazon CloudWatch alarm.
• Send the data to Amazon Machine Learning to make
predictions based on an Amazon ML model.
Page 14
What does AWS IoT consist of?
IoT Rule Engine
{
"sql": "SELECT * FROM 'Sensor/DispensingSensor'",
"ruleDisabled": false,
"actions": [
{
"kinesis": {
"roleArn": "arn:aws:iam::XX:role/sensor_status_role",
"streamName": "sensor_status_stream",
"partitionKey": "${ClientID}"
}
}]
}
Page 15
What does AWS IoT consist of?
Thing Registry
• Associate MQTT Client IDs
• Associate Certificates
• Specify up to 3 custom attributes
Page 16
What does AWS IoT consist of?
Thing Registry
{
"version": 3,
"thingName": "PortableDispenser",
"defaultClientId": "abc123",
"thingTypeName": "Dispenser",
"attributes": {
"model": "123",
"dispensingVolume": "75"
}
}
Page 17
What does AWS IoT consist of?
Thing Registry
response = client.update_thing(
thingName=PortableDispenser',
attributePayload={
'attributes': {
'dispensingVolume': '150'
},
},
expectedVersion=2,
)
Page 18
What does AWS IoT consist of?
Thing Shadow
• Store and retrieve current and future state of the
thing
• Simple JSON Document
• Get and set the state of a thing over MQTT or
HTTP
• Access the shadow regardless of whether the
thing is connected to the Internet
Page 19
What does AWS IoT consist of?
Thing Shadow{
"state" : {
“desired" : {
"sensor" : "ON"
},
"reported" : {
"sensor" : "OFF"
},
"delta" : {
"sensor" : "ON"
} },
"version" : 10
}
Page 20
What does AWS IoT consist of?
Thing Shadow
data = {
"state" : {
"desired" : {
"switch" : "on"
}
}
}
response = client.update_thing_shadow(
thingName = PortableDispenser',
payload = mypayload
)
Page 21
Other services in an scalable IoT architecture
Kinesis
• Streams are made of Shards
• Each Shard ingests data up to 1 MB/sec, and up to 1000
TPS
• Each Shard emits up to 2 MB/sec
• All data is stored for 24 hours – 7 days
• Scale Kinesis streams by splitting or merging Shards
• Replay data inside of 24Hr -7days Window
Page 22
Other services in an scalable IoT architecture
Amazon DynamoDB
• Provisioned at the table level• Write capacity units (WCUs) are measured in 1 KB per
second
• Read capacity units (RCUs) are measured in 4 KB per
second
• RCUs measure strictly consistent reads
• Eventually consistent reads cost ½ of constant
reads
• Read and write throughput limits are independent
• Increase as necessary, decrease at most 4 times per
UTC day
Page 23
Other services in an scalable IoT architecture
Lambda
• AWS Lambda offers 23 "power levels"
• Higher levels offer more memory and more CPU
power
• 128 MB, lowest CPU power
• 1.5 GB, highest CPU power
• Compute price scales with the power level
• Duration ranging from 100 ms to 5 minutes
Page 24
Example IoT architecture
IoT MQTT
protocol
IoT
certificateIoT
ruleIoT
topic
Amazon
Kinesis
AWS
Lambda Amazon
DynamoDB
Amazon
SNS
Alexa
Medication Status
monitoring device
Medication Status Backend
Node.js
AWS
LambdaAWS
Lambda
Page 25
Elephant in the room
http://nos.twnsnd.co/post/104252656546/elephants-tea-party-robur-tea-room-24-march
Amazon
Kinesis
AWS
Lambda
Amazon
DynamoDB
Amazon
SNS
Alexa
AWS IoT
HIPAA Eligible Not HIPAA Eligible
Page 26
Example IoT (PHI) Architecture
IoT MQTT
protocol
IoT
certificateIoT
ruleIoT
topic
Amazon
Kinesis Amazon
DynamoDB
Alexa
Medication Status
monitoring device
Medication Status Backend
Node.js
AWS
Lambda
Amazon
S3
Amazon EC2
client
Amazon EC2
AWS
Lambda
SQS
Page 27
Validic Digital Health Platform
Clinical, Fitness, Wellness and
Nutritional Devices, Wearables,
Applications, and Equipment
Healthcare Providers, Clinical
Research, Wellness Apps, Insurance,
Academia
Page 28
Validic architecture
3rd Party Devices,
Apps, and APIs
Validic Core Platform
VitalSnap™ OCR
Image Algorithm
Calibration
Validic
Standardization
Services
Health Data
Retrieval
Services
Admin
Database
Primary
Database
Replica
Databases
Validic API
Healthcare Systems,
Clinical Trials Platforms,
Wellness Applications
Patient / Individual
Page 29
Alexa Voice Service in Healthcare
Nitin Gujral
Director of Software Engineering, Innovation and Digital Health
Page 30
Boston Children’s Hospital
#1 ranked pediatric hospital
nationwide (8 out of 10
specialties)
400 plus bed; children's
hospital affiliated with Harvard
Medical School
Largest recipient of NIH funds
in pediatric-focused research
Page 31
Innovation and Digital Health AcceleratorShaping the future of health care and extending Boston Children’s pediatric
leadership globally
BCH DataClinical Expertise HCIT Development
Products Platforms Ventures
Boston Children’s Expertise & Capabilities:
+Industry Partners & Startups, like:
=To launch disruptive health care focused:
Learn more:
bostonchildrens.org/accelerator
GE GrandRoundsCerner
IBM Watson Health Rock Health
Page 33
Ask the Internet? –Parents are actively looking for guidance—traditional medical content sites lack
the trustworthy and readily accessible health guidance.
Although I will ALWAYS go with my gut about my kids
being sick, I typically ask the internet to confirm my
suspicion about what might be happening with them.
– Female, 36, Kids ages 1 & 4
Source: 3rd Party consumer research survey (n=200)
Page 34
Decision support in consumers’ hands
Boston Children’s
Data & Expertise
Withings
Raiing
Page 35
Kids.MD™ is delivering consumer facing digital health tools to patients and families across digital mediums.
Kids.MD™
powered by:
Boston Children’s Hospital
web
mobile
voice
device
research
platform
Page 36
“Alexa, ask Kids.MD”
Page 37
Kids.MD – Alexa Skill Architecture
AVS +
Kids.MD
“Alexa, ask Kids.MD about fever”
Skill identified in Alexa
voice service
Request sent to Kids.MD
clinical decision support
Sends
‘recommendations’ back
to Alexa voice service
Alexa converts ‘text’ to
spoken words and
streams audio
Or access
via mobile &
web app
Kids.MD™
Electronic Health
Record
Patient
Portal
Page 38
Kids.MD on AWS
Kids.MD™
Amazon RDS Amazon S3
AWS Lambda API Gateway Amazon EC2
Amazon Dynamo DB
Page 39
Kids.MD Skill: Development
Page 40
Node Module - alexa-sdk
Handlers for intents
Page 41
Node Module - alexa-sdk
Custom events
Page 42
Node Module - alexa-sdk
State Machine
Context persistence using DynamoDB
Page 43
Node Module - alexa-sdk
Setting context
Response
Page 44
Project structure
• Structure backend code; e.g.,
handlers, model and other assets
- including tests
• Organize Alexa Skills Kit speech
assets
Page 45
Debugging
Locally debug AWS Lambda code
Reference:
Blog by Nathan Grice (https://developer.amazon.com/public/community/post/Tx24Z2QZP5RRTG1/New-Alexa-
Technical-Tutorial-Debugging-AWS-Lambda-Code-Locally
• within any IDE
supporting
Node.JS
• simulate AVS
request /
response
Page 46
Unit testing
Javascript testing with Jasmine (a behavior driven framework)
https://jasmine.github.io/
NPM module - jasmine or jasmine-node
Ability to mock requests and responses
Reference:
Article by Jeremy Bohrer: https://medium.com/@jjbskir/unit-testing-an-amazon-alexa-skill-with-node-js-and-
jasmine-98982544471f
Page 47
Integration testing – developer tools
Page 48
Lambda deployment
• NPM package
• Leverages AWS SDK
• Packaging for Lambda
• Upload to Lambda
• Supports unit testing as well
https://github.com/Tim-B/grunt-aws-lambda
Page 49
Kids.MD Skill: technical
considerations
Page 50
States and decision trees
Page 51
Managing context in an Alexa skill
• short time frame for the duration of skill invocation
• access to user and application info
• persistent context for a user
Page 52
Intent Schema Slot types
• better voice recognition
• more accuracy with user response
• supports free form responses
Custom Slot Types
Built-in Slot Types
• e.g. AMAZON.NUMBER, AMAZON.DATE,
AMAZON.DURATION
Page 53
Speech Synthesis Markup Language (SSML)
• for more natural speech
• pauses and predefined formats (characters, digits, date, telephone,
address, etc.)
<s>Let us start again.</s>
<p>Would you like to hear this information again?</p>
<break strength='weak'/>
<say-as interpret-as="digits">101.4</say-as>
Page 54
Speech Synthesis Markup Language (SSML)
• improved pronunciation using “phoneme”
• specifically for medical terminology
<phoneme alphabet=\"ipa\"
ph=\"sɪmp.təms\">symptoms</phoneme>
<phoneme alphabet=\"ipa\" ph=\"moətrɪŋ\">Motrin</phoneme>
Page 55
Logging, auditing, and monitoring
• AWS Lambda CloudWatch logs and metrics
• Custom error logs
• Custom auditing of user interactions
• Monitoring of various service components
Page 56
Account linking
• Link an Alexa user to the user of your system
• Enables contextual interactions based on user’s preferences
• Performed at the time of skill installation
• Using OAuth authorization flow
Reference:
https://developer.amazon.com/public/community/post/Tx3CX1ETRZZ2NPC/Alexa-Account-Linking-5-
Steps-to-Seamlessly-Link-Your-Alexa-Skill-with-Login-wit
Page 57
Images and audio in Alexa Skills
• Display images in Cards
• JPG or PNG (less than 2 MB) Need CORS support for image URLS
• SSML Audio Tags <audio>
• MP3 support
• 90 sec limit
• Audio Streaming
• Leverage built-in intents for playback
Reference: https://developer.amazon.com/public/community/post/Tx1DSINBM8LUNHY/New-Alexa-Skills-Kit-ASK-
Feature-Audio-Streaming-in-Alexa-Skills
Page 58
Voice design – lessons learned
• Maintain intuitive conversation
• provide guidance via prompts
• Balance the amount of verbal responses
• Effectively utilize Companion App(s)
• with images, audio and streaming sound
Page 59
Some enterprise deployment challenges
Security
Connectivity
Page 60
Voice in healthcare A new way for patients and clinicians to access health information and
personalized care in various settings
@ home
@ inpatient
@ outpatient
by patients & families
by clinicians
Context: Users:
Page 61
Photo by John Blanding for The Boston Globe
Example usage:
• Home Health Hubs
• At the Bedside for Patient Care
• Virtual Assistant for Clinicians
Voice in healthcare – a new mediumStreamline and ease delivery of health care services
Page 62
“Alexa, ask Children’s Hospital”
Page 63
Growth of “Internet of Healthcare”
Page 64
Promise of connected healthcare
Portable
Proactive
Patient Centric
Page 66
Remember to complete
your evaluations!