Top Banner
© 2019 KNIME AG. All rights reserved. Sharing and Deploying Data Science with KNIME Server Roland Burger Marten Pfannenschmidt KNIME
38

Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

May 20, 2020

Download

Documents

dariahiddleston
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved.

Sharing and Deploying Data Science with KNIME Server

Roland Burger

Marten Pfannenschmidt

KNIME

Page 2: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 2

KNIME Server

• Collaboration– Share Expertise and Templates/Blueprints

• Automation– Schedule, Monitor, Update

• Deployment– To Humans: Analytical Applications (“Guided Analytics”)

– To Machines: via REST and custom APIs

• Management– Manage distributed setups

– Integrate with existing enterprise setup (authentication)

Page 3: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 3

Delivering Data Science

• Statistics• Mining• AI/ML/DL Models • R/Python Scripts• Legacy Languages

Marketing and Sales Decisions

Financial Decisions

Manufacturing Decisions

IT Decisions

Product R&D Decisions

• Data Access• Data Blending• Transformation• In-database• “In” Big Data

Apps

Data Blending

Data Analytics

Predictive Analytics

Machine Learning

Artificial Intelligence

• Analytics as a service

• Guided Analytics• Analytical Apps &

Dashboards

Data Science

Page 4: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 4

Creating Data Science: KNIME Analytics Platform

Partner

Extensions

KNIME Analytics Platform

Community

Extensions

KNIME

Extensions

KNIME

Integrations

Data Blending

Data Analytics

Predictive Analytics

Machine Learning

Artificial Intelligence

Data EngineersData Analysts

Data ScientistsML/AI Engineers

App DevelopersData Science

Page 5: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 5

Productionizing Data Science: KNIME Server

Marketing and Sales

Financial

Manufacturing

IT

Product R&D

DeployManageAutomateCollaborate

KNIME Server

Data Science Teams IT Teams Business Teams Apps

REST API

Web Applications

Mobile Applications

Cloud-based Services

Guided Analytic Apps

Page 6: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 6

Better Decision-making, Faster!

KNIMEIntegrations

KNIMEAnalytics Platform

KNIME Server

Load > Integrate > Transform > Analyze > Visualize

KNIMEExtensions

CommunityExtensions

PartnerExtensions

Collaborate Automate Manage Deploy

Page 7: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 7

KNIME Server

Management

Deployment

Configuration &Versioning

RESTful Services

Client & User PreferencesSecurity

Guided Analytic AppsWeb Front-ends

Collaboration

Workflow Hub Workflows

Shared Repository, Versioning

Metanodes, Nodes Data

AutomationDistributed Executors

Remote, Client ControlProvisioning

Page 8: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 8

• Connect to KNIME Server from local KNIME Analytics Platform– Authentication possible via LDAP

/ Active Directory

– Centralized administration and security

• Interact with KNIME Server:– Upload workflows for sharing and

collaboration with colleagues

– Execute workflows, create schedules, set permissions, …

– Organize your work in workflow groups (directories)

Collaboration

Workflows

Workflow Groups

Data

Metanode Templates

Jobs

Click to add new mountpoint

Page 9: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 9

• Right-click -> Deploy to Server– Share workflows, data,

etc. for collaboration

– Continuous delivery in all steps of workflow life cycle

Deploy workflows to KNIME Server

Page 10: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 10

KNIME Workflow Hub

• A place to share knowledge about Workflows.

– Share

– Discover

– Search

– Rate

– Comment

– Version

Page 11: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 11

Metanode Templates

• Create template from metanode and store on server

• Templates are linked -> check for new version and apply updates automatically

• Increase efficiency through reusable workflow components

• Ensure compliance with company policies

Page 12: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 12

Remote Execution

• Execute workflows directly on KNIME Server

• Scale beyond limitations of desktop machines

Page 13: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 13

Scheduled Execution

• Automate workflow execution in regular intervals

• Orchestrate job pipeline by using Call Workflow nodes

Page 14: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 14

Remote Workflow Editor

• Monitor progress of execution on server (almost) live• Very helpful for longer-running jobs

What‘s my workflow doing right now?

Page 15: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 15

Remote Workflow Editor

• Change node configurations directly in the remote job– Resume execution if

there was an error

– Try different configuration options without re-executing the full job

– Add nodes, change node connections, all live and in the secure server environment

Page 16: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 16

KNIME WebPortal

• All workflows on KNIME Server available as web apps

• Step-by-step execution of workflows from any browser

• Simple, clean interface for end users

– Guided Analytics

• Customize layout to match corporate design

Page 17: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 17

Guided Analytics: Interactive Data Science

Interaction Points

Page 18: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 18

Inside an Interaction Point

Page 19: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 19

KNIME Server REST API

• Enables external integration

• Build applications around KNIME Server

• e.g. for Microservices and real time scoring

Output in server responseInput data

Deploy KNIME workflows as web services

Page 20: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 20

KNIME Server REST API: Job Pool

KNIME Workflow for e.g. sentiment prediction, churn score, chemical property

Pre-loaded jobs pooled and waiting

for work

Workflow immediately

executed

External REST call

No “load latency” for client

Results

Take from pool

Pooled jobsKNIME Executor

KNIME SERVER

Page 21: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 21

KNIME Server REST API

• Swagger UI: Explore API definition of workflow in browser

• Provides all information needed to interact with API

Page 22: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 22

KNIME Server REST API

• Programmatically control KNIME Server– Upload/download/delete

resources

• Staging from Dev to Prod

– Upload licenses

– Empty trash/restore items

– Execute workflows

– Schedule jobs

– Set permissions

– Create users & groups, etc.All server functionality available via REST API

Page 23: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 23

Version Control

• Create snapshots of workflow revisions

• Allow roll-back to previous versions

• Full workflow history available in client and on Workflow Hub

Page 24: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 24

• Inspect differences between versions of the same workflow

Workflow Difference

Highlight differences:- Nodes included/excluded- Node configurations

Page 25: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 25

Permission Management

• Manage permissions for workflows and data on KNIME Server

• Control which groups can access, modify, or execute a workflow

• Use LDAP / Active Directory groups for centralized administration

Page 26: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 26

Managing Preferences – local

KNIME Analytics Platform

KNIME Analytics Platform

KNIME Analytics Platform

How do I install

database drivers?

How do I configure Python?

KNIME WebsiteInternal knowledgebase

Page 27: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 27

Managing Preferences – KNIME Server

Marketing R&DFinance

Linux and macOSPythonR

Windows 7OracleMS Access

Windows 10HiveSpark

• Different departments/teams have different requirements

• Multiple OS deployments• Windows 7• Windows 10• Linux• macOS

Page 28: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 28

Managing Preferences – KNIME Server

KNIME Server

Profiles can include: Preferences, drivers, and more

• Client-profiles• Python-Linux• Python-macOS• R-Linux• R-macOS• Databases-Win7• Big Data-Win10

client-profiles

Big Data-Win10

Databases-Win7

Python-macOS (etc)

Marketing Finance R&D

Page 29: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 29

Client Customization Profiles

Page 30: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 30

KNIME Server – Admin made easy

Page 31: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 31

Distributed Executors

• Need more workflow horsepower?

• KNIME Server currently supports ‘Scale Up’

• KNIME Server Distributed Executors allows ‘Scale Out’

https://upload.wikimedia.org/wikipedia/commons/thumb/8/8b/Server.svg/2000px-Server.svg.pnghttps://upload.wikimedia.org/wikipedia/commons/5/53/Server-multiple.svg

Page 32: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 32

Distributed Executors

KNIME Server

Executor 1

Executor 2

Executor 3

RabbitMQ

KNIME Server + Distributed Executors

Requests to execute

workflows

Page 33: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 33

Distributed Executors

KNIME Server

Executor 1

RabbitMQ

KNIME Server + Distributed Executors

Request to execute

workflow

Scripted launch on AWS CloudFormation

Page 34: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 34

Distributed Executors

KNIME Server

Executor 1

Executor 2

RabbitMQ

KNIME Server + Distributed Executors

More requests to

execute workflows

Launching

CPU load increases

AWS Auto-scaling Group

Page 35: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 35

Distributed Executors

KNIME Server

Executor 1

Executor 2

RabbitMQ

KNIME Server + Distributed Executors

Requests to execute

workflows

AWS Auto-scaling Group

Page 36: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 36

KNIME Software – On premise and in the cloud

KNIMEAnalytics Platform

KNIME Server

KNIMEAnalytics Platform

KNIME Server

KNIMEAnalytics Platform

KNIME Server

Page 37: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

© 2019 KNIME AG. All rights reserved. 37

KNIME Server

Management

Deployment

Configuration &Versioning

RESTful Services

Client & User PreferencesSecurity

Guided Analytic AppsWeb Front-ends

Collaboration

Workflow Hub Workflows

Shared Repository, Versioning

Metanodes, Nodes Data

AutomationDistributed Executors

Remote, Client ControlProvisioning

Ready to get started with KNIME Server? Contact us for a free trial:

[email protected]

Page 38: Sharing and Deploying Data Science with KNIME Server · •AI/ML/DL Models •R/Python Scripts •Legacy Languages Marketing and Sales Decisions Financial Decisions Manufacturing

38© 2019 KNIME AG. All rights reserved.

The KNIME® trademark and logo and OPEN FOR INNOVATION® trademark are used by KNIME AG under license from KNIME GmbH, and are registered in the United States.

KNIME® is also registered in Germany.