Mistral OpenStack Meetup Feb 5

© MIRANTIS 2014© MIRANTIS 2014 !1

Mistral Workflow as a Service

Renat AkhmerovSenior Software Engineer

© MIRANTIS 2014 !2

Agenda

• Project Mission

• Project users

• Use Cases

• Main Features

• DSL, Data Flow concept, Architecture

• Current status

• Live Demo

• Roadmap and Contribution

© MIRANTIS 2014 !3

Workflows all Around

Find a job

Make money

Stanford, Harvard, etc.

© MIRANTIS 2014 !4

Project Mission

Provide a mechanism to define and execute tasks and workflows without writing the code

© MIRANTIS 2014 !5

Who are Mistral Users?

• OpenStack developers: Nova, Heat, Murano, Solum etc.

• Administrators: capabilities like Cloud Cron to schedule periodic helper cloud tasks (local VM processes, backups etc.)

• Enterprise application developers: automate business processes consisting of multiple distributed processing steps

© MIRANTIS 2014 !6

Use Cases

• Cloud Cron: a system administrator can schedule cloud tasks for periodical execution

• Cloud Environment Deployment: tools like Heat can represent deployment steps as a workflow and use Mistral to run it

• Live Migration: on CPU close to 100% run specific migration workflow

• Long-Running Business Process

• Big Data Analysis & Reporting

© MIRANTIS 2014 !7

Cloud Cron

• Start a workflow on event (timer)

• Single point of configuration and control for helper periodic jobs

• Wide range of tasks: local processes, OpenStack services (Nova, Cinder, Swift etc.), email, SMS, custom actions

• Monitoring & Management

• HA & Scalability

© MIRANTIS 2014 !8

Main Features

• RESTful Web service

• Based on Domain Specific Language (DSL)

• Runs workflows consisting of synchronous and asynchronous tasks

• Tasks associated with pluggable actions

• Built-in actions: HTTP request, AMQP message

• Runs workflows on event (timer, ceilometer alarm)

• Provides history and state of currently running workflows

• HA and Scalability for workflow executions

© MIRANTIS 2014 !9

Mistral Workflow

• Graph of tasks

• Task Actions (HTTP, AMQP, SSH, email, etc.)

• Transitions

• Dependencies

• Data Flow

• Conditions

• Events (triggers)

T1

T5

T3

T4T2

start

T6

T7

depends

T8

T9

© MIRANTIS 2014 !10

Mistral DSL Capabilities

• Task dependencies (task3 depends on task2 and task1)

• Direct transitions (on task1 success execute task2, on error task3)

• Conditional expressions in dependencies and direct transitions (based on Data Flow)

• Task action declarations (signals over HTTP, AMQP)

• Standard actions (std:repeat, std:email)

• Cron events (run workflow hourly)

© MIRANTIS 2014 !11

DSL Snippet

Workflow: tasks: runJob: requires: createVM: $.vm_ip != null action: DemoApp:runJob on-success: deleteVM on-error: sendJobError ! deleteVM: action: Nova:deleteVM on-finish: end ! sendJobError: action: std:email parameters: address: [email protected] subject: Workflow error body: Error occurred while running a VM job in execution {$.execution.id} on-finish: deleteVM

© MIRANTIS 2014 !12

Data Flow (concept)

RunVM input:

image_id

SendMsg input:

person.email

IsMale input:

person.name

GoOn “image_id”: “123”, “person”: { “name”: “John”, “email”: “[email protected]” }

“image_id”: “123”, “person”: { “name”: “John”, “email”: “[email protected]” }, “vm_ip”: “10.0.0.3”

“image_id”: “123”, “person”: { “name”: “John”, “email”: “[email protected]” }, “vm_ip”: “10.0.0.3”

“image_id”: “123”, “person”: { “name”: “John”, “email”: “[email protected]” }, “vm_ip”: “10.0.0.3”, “male”: “true”

“image_id”: “123”, “person”: { “name”: “John”, “email”: “[email protected]” }, “vm_ip”: “10.0.0.3”, “sent_msg”: “true”

“image_id”: “123”, “person”: { “name”: “John”, “email”: “[email protected]” }, “vm_ip”: “10.0.0.3”, “male”: “true”, “sent_msg”: “true”

- task input - task result (added into context)

© MIRANTIS 2014 !13

Architecture

Scheduler

Task Executor

API ServerEngine

Task Executor

!...

workflows executions

tasks events

Workflow Queue

Task Queue

© MIRANTIS 2014 !14

Current Status

• Basic task dependency based model works

• Pilot is currently being worked on

• Launchpad home page o https://launchpad.net/mistral

• Code in stackforge code repos o https://github.com/stackforge/mistral o https://github.com/stackforge/mistral-extra o https://github.com/stackforge/python-mistralclient

• Description on OpenStack Wiki o wiki.openstack.org/wiki/Mistral

• Contributors: Mirantis, StackStorm

© MIRANTIS 2014 !15

Live Demo

MistralDemo App

1. Upload workbook

2. Run

work

flow (ta

sk4)

HTTP

3

4

4

5

REST endpoint

Execute Backup

Backup User Data

Backup Service Data

Put Service on

hold

© MIRANTIS 2014 !16

Mistral Icehouse Roadmap

• End of February, 2014. Pilot. o Basic DSL capabilities (dependencies, direct transitions, conditionals,

cron events) o Keystone Integration o REST API o Python Client

• April, 2014. Release 0.1 o Overall stability o Refined DSL o Mistral CLI o Horizon Dashboard (Workbook CRUD operations, workflow execution

state, history)

© MIRANTIS 2014 !17

Contribution Opportunities

• Blueprints for additional capabilities

• Seed initial dev team

• Web UI (history, workflow lifecycle management)

• Ceilometer integration

• Integration with deployment tools (Puppet, Chef)

• Advanced capabilities of Python Client (building workflow step-by-step locally, callbacks etc.)

• Workflow rollback

• CLI for Mistral API

• CLI for Mistral action handlers

© MIRANTIS 2014 !18

How to Become a Contributor?

Sign OpenStack

CLA

Buy a computer

Learn Python

Register at Launchpad

Install Python, git etc.

clone mistral repos

Make changes

Send patches

no

Tired?

Have coffee

yes

Cont

ribut

or lif

e-cy

cle

© MIRANTIS 2014 !19

Thanks for Attending!

Q & AProject: launchpad.net/mistral

IRC: #openstack-mistral

Me: Renat Akhmerov

Email: [email protected]