Docker Containers OTA for IoT Gateways Shashank Jain/ SAP HCP
Docker Containers OTA for IoT Gateways
Shashank Jain/ SAP HCP
Customer
About Me
• Development Architect @SAP Bangalore
• Cloud Enthusiast
• Email : [email protected]
• Twitter : @smjain
Customer
Motivation
• IoT Device Explosion
• Container Boom
• Using Developer Workflows like Heroku and Other PaaS for IoT
• A Gateway Centric Solution to Merge Both Worlds
Customer
Docker Docker Everywhere
Customer
What is Docker?
• Docker is a tool created by the folks at dotCloud to make using Linux Containers (LXC) easier to use.
• Linux Containers are basically light weight Virtual Machines (VM). A Linux container runs Unix processes with strong guarantees of isolation across servers.
• Your software runs repeatedly everywhere because its Container includes all of its dependencies
Customer
Docker Architecture
Customer
Docker File System
Customer
Docker vs VMs
Size:
VMs are very large which makes them impractical to store and transfer.
VM: You have a container image that is 1GB in size. If you wanted to use a Full VM, you would need to have 1GB times x number of VMs you want.
Docker: With LXC you can share the bulk of the 1GB. It means that If you have 1000 containers you still might only have a little over 1GB of space for the containers OS, assuming they are all running the same OS image.
Customer
Docker vs VMs
Performance:
VM: A full virtualized system usually takes minutes to start.
Docker: LXC containers take seconds, and most times less then a second.
Customer
Docker Advantage
Throwable Sandboxes: Create a container in a minute to test your stuffs and tear it
down.
Fine Application Delivery: Containers allow you to package just about any application.
You could add the dependencies of the application in the container itself. Ex: MySQL
service.
Reusability: Docker makes containers reusable.
Uniformity: Development and production Environments
Customer
Some Terminologies on Docker
Image
An image is a read only layer used to build a container. They do not change.
* Container
Is basically a self contained runtime environment that is built using one or more images.
You can commit your changes to a container and create an image.
* Docker index / registry
These are public or private servers where people can upload their repositories so they
can easily share what they made.
Customer
Docker Under The Hoods Lets see what docker is doing here.
1. Downloaded the image from docker repo
2. Generated a new LXC container
3. Created a new file system
4. Mounted a read/write layer
5. Allocated network interface
6. Setup IP
7. Setup NATting
8 Executed the bash shell in the container
Customer
SAP IoT Architecture
Customer
SAP IoT Architecture
Control Plane
• Sets up the Rules and Data flows
• Currently uses the Camel Routes as the mechanism
Data Plane
• Responsible for Using the rules/flows and sending data upstream to the Cloud
Customer
IoT Gateway/Connector
In general the IoT Connector should exhibit the following properties
• The connector should have the capability to support variety of protocols out of the box and also the
capability to easily introduce adapters for enabling communication via new/custom protocols.
• The connector should allow easy pluggabilty of modules like filtering/Edge Stream Processing
(majorly on the fly) , into the existing runtime
• The connector should allow if needed a store and forward mechanism for Guaranteed delivery
• The connector should have proper failure handling primitives built in.
• The connector should allow proper usage of the hardware and implement a capability to apply back
pressure to allow bounded latency and prevent abrupt crashing of the runtime due to resource
exhaustion.
Customer
IoT Gateway Data Pipeline
Customer
Currency of Deployment For Gateways
Yesterday
OSGi Components
Advantages
• Lifecycle managed by OSGi
• Easy to introduce new component in runtime
Challenges
• Limited Flexibility in defining Flows
• Coupled to Specific Technologies
Customer
Currency of Deployment For Gateways
Today
Camel Routes orchestrating OSGi components
Advantages
• Ability to define new Data flows/Routes on the fly
• Can run multiple data flows on a single gateway
Challenges Coupled to Specific Set of Technologies like Java/Camel, Choice of DB
Customer
Currency of Deployment For Gateways
Future
Docker Images
Advantages
• Deploy the complete pipeline instead of Components
• Immutability Rules
Challenges
• Deploying Images on low ended machines
Customer
Docker OTA For IoT Gateways Opportunity
• Reduce overall deployment downtime for IoT landscapes.
• Achieve an immutable deployment strategy using containers.
• Increase support driven development
• Allow polyglot development based on skills of development teams.
Customer
Rationale • The heterogeneous nature of IoT Services demands for stronger CI
capabilities
• Current CI landscapes work as follows
• The subsystems, subcomponents, and services that are interconnected are owned by different groups and third party vendors. Inability to upgrade online, just one dependent subcomponent could impact the whole application.
• Considerable effort and coordination among different teams is needed to set the right configuration data populated across all the systems/ devices.
Customer
PROPOSED FLOW
Customer
Benefits
By Introducing Immutability we can do
Canary Deployments
• Apply changes to a selected fleet of Gateways
• If All works well, Apply the changes to all the gateways
• If there are issues we can rollback to the last know good image.
Customer
Challenges
• Adapters are customer Specific.
• The protocols are propertiery and native at times.
• Some protocols cant be shared and have IP around them.
Customer
Possible Solutions Break the solution into 3 parts
1. A Customer specific Adapter Image
2. A Extended image
3. Bind the images via zeromq
This allows
1. Customer to write adapter in Language of their choice
2. SAP to write the rest of data pipeline in the language/framework they want
Customer
Flow
Customer
Current Status
Working on a Prototype for the above flow
• Using 2 runtimes
1. Java based Data Pipeline
2. Golang Based Data Pipeline
• Should have the ability to switch via a click.
Customer