Cloud Computing with APL - Dyalog Ltd.6 Cloud Computing with APL Cloud Computing: Definitions Cloud Computing = "Using someone elses computer" • SAAS (Software As A Service): You

Cloud Computingwith APL

Morten Kromberg, CXO, Dyalog

Cloud Computing Dockerwith APL

Morten Kromberg, CXO, Dyalog

2

Cloud Computing with APL

Cloud Computing: Definitions

3


Cloud Computing: DefinitionsCloud Computing = "Using someone elses computer"

4



• SAAS (Software As A Service): You use "Software" like gmail, dropbox, etc - with no idea of where it is running, or where your data is stored

5




• PAAS: Someone hosts your application on a "Platform" which runs specific development tools (php, mysql, ASP.NET, Wordpress...)

6





• IAAS : Someone hosts your Virtual Machine on their "Infrastructure". You can install anything you like.

7





• IAAS : Someone hosts your Virtual Machine on their "Infrastructure". You can install anything you like.

This talk is about installing and running Dyalog APL on IAAS.

8


Using IAAS

9


Using IAAS• Pick an IAAS provider

10


Using IAAS• Pick an IAAS provider• Upload or Create a Virtual Machine

11


Using IAAS• Pick an IAAS provider• Upload or Create a Virtual Machine

• You save the hassle ofo Buying [a] big enough computer[s]o Maintaining / replacing the hardwareo Paying for a fast internet connection

12


Using IAAS – the Hard Part

13


Using IAAS – the Hard Part• Picking a provider:

o Amazon, Microsoft, Google, DigitalOcean, Oracle, RackSpace, Netrepid, IBM/Redhat, GreenCloud, Alibaba, Openstack, ...

o Can't help you with that

14


Using IAAS – the Hard Part• Picking a provider:

o Amazon, Microsoft, Google, DigitalOcean, Oracle, RackSpace, Netrepid, IBM/Redhat, GreenCloud, Alibaba, Openstack, ...

o Can't help you with that

• Installing the software that you want to run on the Virtual Machine:o APL Interpreter, Web Server or Service Framework,

Database System, other tools ...o This is where Containers are "Pure Magic"

15


FROM ubuntu:18.04

ADD ./dyalog-unicode_17.0.34604_amd64.deb /ADD /myapp/v7/test /myapp

RUN dpkg -i /dyalog*.debRUN git clone https://github.com/dyalog/JSONServer /JSS

ENV RIDE_INIT="SERVE:*:4502"ENV CodeLocation=/myapp

CMD dyalog /JSS/JSONServer.dws

Containers Solve the Distribution Problem

16


FROM ubuntu:18.04






Base Image

17


FROM ubuntu:18.04






Base Image

Files to Add

18


FROM ubuntu:18.04






Base Image

Files to Add

Run during Build

19


FROM ubuntu:18.04






Base Image

Files to Add

Run during Build

Environment Vars

20


FROM ubuntu:18.04






Base Image

Files to Add

Run during Build

Environment Vars

Run at Startup

21


FROM ubuntu:18.04






Base Image

Files to Add

Run during Build

Environment Vars

Run at Startup

This "Dockerfile" completely describes a machine which will run "myapp".

22


FROM ubuntu:18.04






Base Image

Files to Add

Run during Build

Environment Vars

Run at Startup

This "Dockerfile" completely describes a machine which will run "myapp".

Your Code

23


FROM ubuntu:18.04

ADD ./dyalog-unicode_17.0.34604_amd64.deb /

RUN dpkg -i /dyalog*.debRUN git clone https://github.com/dyalog/JSONServer /JSSRUN git clone https://github.com/myco/myapp /myapp




Base Image

Files to Add

Run during Build

Environment Vars

Run at Startup

Uses GitHub to load the source code for "myapp".

Your Code

24


25


FROM ubuntu:18.04ADD ./dyalog-unicode_17.0.34604_amd64.deb /ADD /myapp/v7/test /myappRUN dpkg -i /dyalog*.debRUN git clone https://github.com/dyalog/JSONServer /JSSENV RIDE_INIT="SERVE:*:4502"ENV CodeLocation=/myappCMD dyalog /JSS/JSONServer.dws

Building and Running the Docker ImageDockerfile

26



Building and Running the Docker Image

Build

Dockerfile

docker build –t myco/myapp-test .

27



Building and Running the Docker Image

Build

Run

Dockerfile

docker run -p 8081:8080 -v /somefolder:/data –e DEBUG=1 myco/myapp-test


28


docker run syntax & common switches

Switch Description-p hhhh:cccc Map TCP port cccc in container to hhhh on host-e name=value Set environment variable inside the container-v /hfolder:/cfolder Mount /hfolder in container as /cfolder-t Allocate a pseudo-TTY-i Keep stdin open even if not attached--rm Discard changes when container terminates


docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

29


Distributing the Image: DockerHubBuild

Rundocker run -p 8081:8080 -v /somefolder:/data –e DEBUG=1 myco/myapp-test


30





We can "push" the image to DockerHub:

31





Pushdocker login

docker push myco/myapp-test


32





Pushdocker login



Now, the following will work on ANY computer that has Docker installed(assuming myco/myapp-test is a PUBLIC container)

33





Pushdocker login



Now, the following will work on ANY computer that has Docker installed(assuming myco/myapp-test is a PUBLIC container)

34


GitHub for source code distribution.Code can be loaded at Image Build time,OR when a Container is started.

DockerHub for container distribution.

+

= Simple distribution of applications and toolsto ANY machine – including IAAS VMs.

35


36


37


Containers are STUNNING technology!

38


Containers are STUNNING technology!In addition to making distribution very simple:

39


Containers are STUNNING technology!In addition to making distribution very simple:• Containers allow several applications to share

the same host but remain isolated from each other

40




• The effect is similar to Virtual Machines but the Operating System kernel is shared

41





• Containers start and stop Containers in secondso (the Operating System does not need to "Boot Up")

42





• Containers start and stop Containers in secondso (the Operating System does not need to "Boot Up")

• Containers consume MUCH less resources than VMs

43


Containers are STUNNING technology!ZDNET:Docker is hotter than hot because it makes it possible to get far more apps running on the same old servers and it also makes it very easy to package and ship programs.

http://www.zdnet.com/article/what-is-docker-and-why-is-it-so-darn-popular/

http://www.zdnet.com/article/what-is-docker-and-why-is-it-so-darn-popular/

44


Containers & Docker

From:http://www.zdnet.com/article/what-is-docker-and-why-is-it-so-darn-popular/

45


Containers & Docker

From:http://www.zdnet.com/article/what-is-docker-and-why-is-it-so-darn-popular/

46


Linux

47


Linux• Container technology works best with

Linux, due to the size of the kernel

48



Linux, due to the size of the kernel• Windows kernels are getting smaller

but are still 10-20x as large as Linux (~0.5-1Gb vs 50Mb).

49



Linux, due to the size of the kernel• Windows kernels are getting smaller

but are still 10-20x as large as Linux (~0.5-1Gb vs 50Mb).

• Good News: Your Dyalog APL code will run unchanged under Linux.o So long as it doesn't call Windows APIs

50


Docker for Windows• Docker for Windows uses Microsoft

Hyper-V to run either Linux or Windows virtual machines.

• It provides the same command line interface as Docker under Linuxdocker build –t myco/myapp-test .



51


52


53


54


55


Public Dyalog ContainersThese currently for experimentation only and are based on UNSUPPORTED NON-COMMERCIAL Dyalog 17.1.All run full development interpreters in interactive terminal mode.

dyalog/dyalog:17.1-dbg• Linux + Dyalog APL Interpreterdyalog/jsonserver:dbg• dyalog:17.1-dbg + JSONServerdyalog/miserver:dbg• dyalog:17.1-dbg + MiServerdyalog/jupyter• dyalog:17.1-dbg + Python, Anaconda & Jupyter Notebook

56


FROM ubuntu:18.04ADD ./dyalog-unicode_17.0.34604_amd64.deb /RUN dpkg -i /dyalog*.debRUN git clone https://github.com/dyalog/JSONServer /JSSADD /myapp/v7/test /myappENV RIDE_INIT="SERVE:*:4502"ENV CodeLocation=/myappCMD dyalog /JSS/JSONServer.dws

Benefits of Public ContainersWithout Public Containers

57




FROM dyalog/jsonserver:dbgADD /myapp/v7/test /myappENV RIDE_INIT="SERVE:*:4502"ENV CodeLocation=/myappCMD dyalog /JSS/JSONServer.dws

With Public Containers

58




FROM dyalog/jsonserver:dbgADD /myapp/v7/test /myappENV RIDE_INIT="SERVE:*:4502"ENV CodeLocation=/myappCMD dyalog /JSS/JSONServer.dws

With Public Containers

docker run –p 8080:8080 –p 4502:4502 –v /myapp/v7/test:/myapp –e RIDE_INIT="SERVE:*:4502" –e CodeLocation=/myapp dyalog/jsonserver

Or even without a Dockerfile

59


FROM dyalog/jsonserver:dbg

ENV MAXWS=256MENV CodeLocation=/appENV Port=8080

ENV Secure=1ENV SSLValidation=64 ENV RootCertDir=/certs/ca ENV ServerCertFile=/certs/server/myserver-cert.pem ENV ServerKeyFile=/certs/server/myserver-key.pem

ADD test-certs /certs

ADD backend /app

CMD dyalog /JSONServer/Distribution/JSONServer.dws

Demo: Secure JSONServer

Runs ZodiacService backend as a secure service

60






ADD backend /app


Demo: Secure JSONServerAPL+JSONServer included


61






ADD backend /app



Basic JSONServerSettings


62






ADD backend /app




Secure Options


63






ADD backend /app




Secure Options


Add Certificates

64






ADD backend /app




Secure Options

Application Code


Add Certificates

65






ADD backend /app




Secure Options

Application Code

Start JSONServer


Add Certificates

66


67


68


Demo Time

69


Demo TimeOn each machine, we have already:

70


Demo TimeOn each machine, we have already:• Installed git

71


Demo TimeOn each machine, we have already:• Installed git yum install git

72


Demo TimeOn each machine, we have already:• Installed git• Installed docker

yum install git

73


Demo TimeOn each machine, we have already:• Installed git• Installed docker yum install -y docker

usermod -a -G docker mary

yum install git

74


Demo TimeOn each machine, we have already:• Installed git• Installed docker• Installed the

Docker Util Scripts

yum install -y dockerusermod -a -G docker mary

yum install git

75



Docker Util Scripts

• (and put them on the PATH)


yum install git

76



Docker Util Scripts

• (and put them on the PATH)


yum install git

git clone https://github.com/mkromberg/docker-utils

77


Demo Time• Build the "secure" service• Push it to DockerHub• Login to an AWS EC2 instance • Start the service• Test it from a Web Browser

78


79


80


81


82


83


Dyalog Public Scripts

84


Dyalog Public Scriptsgit clone https://github.com/mkromberg/dyalog-docker

85



In parallel folders bashscripts and winscripts:

86



In parallel folders bashscripts and winscripts:dyalog-c folder [rideport]• Starts container dyalog/dyalog:17.1-dbg

folder is always mounted as /app in the container

rideport is the optional port that RIDE can be attached to

87




jsonserver-c folder [[httpport] [rideport]]• Starts container dyalog/jsonserver-dbg

folder is always mounted as /app in the containerhttpport is the application port that is always exposed by json- & mi-serversrideport is the optional port that RIDE can be attached to

88





miserver-c folder [[httpport] [rideport]]• Starts container dyalog/miserver-dbg


89





miserver-c folder [[httpport] [rideport]]• Starts container dyalog/miserver-dbg

jupyter-c [folder[/notebook]] [httpport]• Starts container dyalog/jupyter (Jupyter notebook server)


90


Demo Time

91


Demo Time

Let's build

92


Demo Time

Let's build an APL Based

93


Demo Time

Let's build an APL BasedWeb Site

94


Demo Time

Let's build an APL BasedWeb Site From Zero

95


Demo Time

Let's build an APL BasedWeb Site From Zero

In ABOUT 2 minutes...

96


docker-compose (multiple services)

97


docker-compose (multiple services)

98


scaling (replicated services)

99



100



101


Ideas for Future ContainersRuntime and Development/Debug versions of all containers.

dyalog/tamstat• Runs HTML/JS version of Tamstat "anywhere"• Looks for data in mapped folder /data

dyalog/isolate• Runs an isolate server• If /workspace.dws is found,

each isolate will be intialised from it• /isolate.config will set security rules and other options

102


Conclusion• It is already easy to deploy APL applications

to the cloud (and debug them there)

• Many more public containers and tools to come.o Also "Premium Images" that you can run on cloud

systems and pay for Dyalog APL "indirectly" throuhg the service provider.

• Follow the Dyalog Webinar series for more news and examples

Cloud Computing with APL - Dyalog Ltd.6 Cloud Computing with APL Cloud Computing: Definitions Cloud Computing = "Using someone elses computer" • SAAS (Software As A Service): You

Documents