On development of distributed optimization modeling ...grid2012.jinr.ru/docs/Voloshinov-RESTOpt-GRID2012-VoloshinovVV.… · Voloshinov V. Vladimir, Smirnov S.A. Supported by Federal

On development of distributed optimization modeling systems in

the REST architectural style

5th International Conference"Distributed Computing and Grid-technologies in Science and

Education"

Voloshinov V. Vladimir, Smirnov S.A.Supported by Federal special purpose program

“Research and development in the priority fields of Russian science and technology complex in 2007-2013"

(Agreement # 07.514.11.4024)

Center of Grid-technologies & Distributed Computing, Institute for System Analysis RAS, Moscow

JINR, Dubna, 2012

#2

http://dcs.isa.ru Support optimization modeling approaches in the practice of scientific researches

● Optimization solvers (LP/MILP, NLP/MINLP) as REST-services

● Translators of optimization modeling languages (AMPL, A Modeling Language for Math. Programming) - as REST-services

● Integration of all above and others utilities (e.g. visualization) into workflows - a new domain-specific distributed applications (available as REST-services)

Support of scientific collaboration in Web2.0 style (e-Science) regarding "publication" of various solvers

REST-services for optimization modeling. General purposes.

#3

Urgency of "service-oriented" optimizationSince 2004, project Optimization Services, www.optimizationservices.org, under the aegis of COIN-OR (IBM) www.COIN-OR.org/projects/OS.xml

COIN solvers !!!AMPL, GAMS - !!!XML-RPC, WSDL, BPEL - ???

#4

Our approachhttp://dcs.isa.ru/drupal/ru/development/mathcloud/optimizationServices

REST - as an architectural styleRESTful-(web)-services - as a middle-wareHTTP as a transport protocol, JSON (JavaScript Object Notation) as a messages format (plain text), HTML+JavaScript for Web User Interface (WUI)

MathCloud (aka WebSolve, http://code.google.com/p/websolve/) - as a middleware and software toolkit

AMPL and GNU MathProg - optimization modeling and algorithms (high-level) description

AMPL-compatible solvers (LP/MILP, NLP, MINLP), GNU MathProg (LP/MILP, GLPK, GNU LP Kit)

#5

JAX-RS(Jersey)

Service-B

Service-A

Service-J (Python)

Service-G

Jobs/Results

WAITINGRUNNINGDONEERROR

ServerResource

ServiceResource

JobResource

FileResource

ServiceManager

JobManager

service.conf

Java Adapter

Console Adapter

Grid Adapter

Cluster Adapter

Java App

Console App

gLite (EGGE)

Port. Batch. Sys

Web-сервер(Jetty)

Service-C

MathCloud toolkit. Service container.

#6

MathCloud toolkit. Workflow Management System (WFMS).

Workfloweditor WUI(browser)

WFMSeditor

interfaceWFs as

a services

WF JSONdescriptor WF running

engine

WFMS service / server

RESTService REST

Service

RESTService

Optimization modeling standards

Finite-dimension mathematical programming problems (LP/MILP, NLP/MINLP)

f o p , x minx,

f i p , x0 i∈ I ,g j p , x =0 j∈J x∈Mp∈

x - variables,

p - parameters,

I, J - indices,

M - additional variables constraints (positive/negative, boolean, integer, ranges)Π - check constraints on parameters

∇ x f o p , x ,∇ x f i p , x i∈ I ,∇ x g j p , x j∈J ;∇ xx f o p , x ,∇ xx f i p , x i∈ I ,∇ xx g j p , x j∈J .

Numerical methods (solvers) also requires procedures for first and second derivatives (Jacobians & Hessians):

#7

The existing approach - usage of AML-system

AML - Algebraic Model Languages.

Common features:✔ Convenient (symbolic "TeX-like") description of object & constraints

functions ✔ Separation of "symbolic/abstract" models and numerical data for

multivariate computation (parameter sweeping)✔ Automatic differention (Jacobian & Hessian)✔ Support of "Lagrange formalism" - access to variables and duals found

as a result of solution✔ Unified open-source (even for "commercial" AMLs) API for solvers'

developers

#8

There are a number of AMLs

Incomplete list:

AMPL - A Modeling Language for Mathematical Programming, AT&T Bell Laboratories, D.M. Gay, Brian W. Kernighan, since 1980- , http://www.ampl.comхGAMS - General Algebraic Modeling System, International Bank for Reconstruction and Development, since 1980-x, http://www.gams.com

OPL - Optimization Programing Lang., IBM, ILOG CPLEX (LP, QP, ...), CP Optimizer, http://www-01.ibm.com/

GNU MathProg - "subset" of AMPL for GLPK, GNU LP Kit, Andrey Makhorin, MAI, since 2000, http://www.gnu.org/software/glpk/

Zimpl - since 2004, http://zimpl.zib.de/ (LP, MILP, NLP ?) Konrad-Zuse-Zentrum für Informationstechnik Berlin (ZIB)

#9

General scheme of AMLs usage

"Symbolic" opt. model

file *.mod

f o p , xminx,

f i p , x0 i∈I ,g j p , x=0 j∈J x∈M

Parameters' values

param p : ...;set I : ...;set J : ...;

file *.dat

AML-scriptmodel *.mod;data *.dat;option solver ipopt;solve;display _var, _dvar;printf ...

AML-translator ampl.exegams.exe.

Problem's data as a stub fileAMPL "stub", *.nl, *.sol

GAMS data exch., *.gdx

AML API

AMPL/Solver interfaceLibrary

GAMS....

SolversCPLEXLpsolveMinosKnitroSnoptGurobiMosek...

COIN-OPIpoptBonmin...

AMPL and GAMS - most popular de-facto standards

#10

GLPK

Set of "atomic" RESTful-optimization services

"Atomic" REST-services deployed in a single MathCloud Everest container

Prepare input data & processing output (solution):ampl-stub - generate AMPL-stub from model and data ampl-pre-opt - more complex AMPL-stub generation (model, data, AMPL- script)ampl-post-opt - processing solution (model, data, solution AMPL- format, AMPL-script)

Solver services (via LPSOLVE, IPOPT, BONMIN other AMPL-solvers):optimization-service-{command | cluster | grid} - to solve LP/MILP, NLP/MINLP problems presnted by their AMPL-stub, respectively on dedicated server, cluster, grid-node

Set of GLPK services (GLPK includes GNU MathProg translator):glpk-{command | cluster | grid} - full scheme of optimization: model, data, pre opt. GMP script -> solution -> post opt. GMP respectively on dedicated server, cluster, grid-node

#11

Web-interface of RESTful-optimization services

#12

#13

Set of "composite" RESTful-optimization services

http://dcs.isa.ru/drupal/ru/development/mathcloud/optimizationServicesComposite services are implemented as WORKFLOWS of atomic ones.

"Full optimization cycle" services (ampl-pre-opt, optimization-*, ampl-post-opt):ampl-optimization-service-{command | cluster | grid} - full scheme of AMPL-optimization: model, data, pre opt. AMPL script -> solution -> post opt. AMPL respectively by solvers on dedicated server, cluster, grid-node

mcl-control - "enhanced" AMPL-translator, enables running any (!) AMPL-algorithm in distributed mode; all stubs are sent to a pool off optimization-service-* and solutions are brought back to AMPL (and so on); Includes simple task manager (Python) for load balance

... a number of demo & test services

Workflow for ampl-optimization-service-(command). MathCloud WF Editor

#14

Auto generated web-interface for composite ampl-optimization-service-(cluster)

#15

Web-interface "inherits" WUI of atomic services

Automatic implementation by WFMS

Mcl-control logic. "Distributed mode" of any AMPL algorithms

#16

Take any any AMPL-algorithm and run it in distributed mode: - all stubs are sent to a pool off optimization-service-*;- solutions are brought back to AMPL (and so on);- includes simple task manager (Python) for load balance in heterogeneous "environment", i.e. unknown task complexity and optimization services performance

Load balancing algorithm

#17

Unknown in advance complexity of problems in the queueUnknown performance of opt.-services-* in the pool

“Greedy Round-Robin” - ?? Fast, simple and only ~50% worse than optimal scheduling

Some of problems (complex ones) may be fetched by a number of workers

Mcl-control architecture. MathCloud API (Python)

#18

Everest atomic + Python + a number of AMPL "macroses"

#19

{ "name": "mcl-control","description": "Distributed running of AMPL-скрипт ","inputs": {

"model.mod": {"type": "file","title": "AMPL model (*.mod)","optional": true

},"data.dat": {

"type": "file","title": "Data-file (data.dat)","optional": true

},"runner.amp": {

"type": "file","title": "AMPL algorithm, use( printf ...>,>>(outputFilePath))"

},"solvers": {

"type": "string","title": "List of optimization-service-*"

}},

MathCloud JSON-description of mcl-control (inputs)

#20

# run.sh, bash script ...ln -s stdout stdout.txtln -s stderr stderr.txt{echo "BASE_URL=\"$1/services/$SERVICE_NAME/job$(basename $PWD)\""echo 'SOLVER_URLS=['cat solvers.txt | tr '\r' '\n' | sed -e '/^[[:space:]]*$/d' \

-e 's/[[:space:]]//g' -e 's/^\(.*\)$/"\1",/' echo ']'} > mcl_config.pyln $SERVICE_DIR/*.amp $SERVICE_DIR/*.py .ampl runner.amp

"outputs": {"userout.txt": {"type": "file","title": "Results printfed by user, userout.txt" },

"stderr.txt": {"type": "file","title": "System error during running runner.ampl, stderr.txt"},

"ampllog.txt": {"type": "file","title": "Log-file of AMPL-translator, ampllog.txt"} },

"implementation": {"adapter": "command","command": "bash ../../services/mcl-control/run.sh

...

MathCloud JSON-description of mcl-control (outputs) and impl.

Transport problem with block structure

#21

A number of products must be delivered from initial placements (offers) to consumers (demands) over transport set of limited carrying capacity.O - set of initial placements, D - set of demands, P - set of productsSupplyo,p - initial values of product p in oDemandd,p - requirement on p in dco,d,p - cost of transportation (o->d) for unit of product p

lo,d - capacity of (o->d)

, , , , , ,, ,

, , ,

,, ,

, , ,

min over { }, (total transport cost) s.t.

( , ) (products delivery constraints)

( , ) (products supply constraints)

o d p o d p o d po O d D p P

o d p d po O

o po d pd D

o d p o d

c x x

x Demand d D p P

x Supply o O p P

x l

∈ ∈ ∈

∈

∈

⋅ →

= ∈ ∈

= ∈ ∈

=

∑∑∑

, ,

( ) (transport capacity)

0p P

o d p

p P

x∈

∈

≥∑

Classical problem with block structure to demonstrate decomposition algorithms (Dantzig-Wolfe, Bendres etc) . AMPL-algorithm http://www.ampl.com/NEW/LOOP2/ multi2.mod, multi2.run, multi.dat

Original AMPL DW-algorithm (multi2.run) is not parallel

#22

Original AMPL-algorithm http://www.ampl.com/NEW/LOOP2/ multi2.run uses subsequent for cycle for subproblems solving

...for {p in PROD} { printf "\nPRODUCT %s\n\n", p; solve SubII[p]; ... if Reduced_Cost[p] < - 0.00001 then { /* change subproblems parameters */; ... };...

Modified (multi2_mclTest-remote.amp) is parallel

#23

Just replace fragments of original AMPL code. “Map-reduce style”for {p in PROD} { printf "\nPRODUCT %s\n\n", p; solve SubII[p]; ... if Reduced_Cost[p] < - 0.00001 then { /* change subproblems parameters */; ... };for {p in PROD} { printf "\nPRODUCT %s ==> stub \n\n", p; problem SubI[p]; let __mcl_probName := ("SubI_" & p); commands mcl_write_problem_stub.amp; # Generates sub-problems AMPL-stubs} commands mcl_solve_problems_list.amp; # Parallel solving of SubII_*for {p in PROD} { printf "\nPRODUCT %s <== solution\n\n", p; # solve SubI[p] problem SubI[p]; solution ("SubI_" & p & ".sol");if Reduced_Cost[p] < - 0.00001 then { /* change subproblems parameters */; ... };

mcl-control runs multi2_mclTest-remote.amp

#24

Start and finish web-forms

Fragments of userout.txt

#25“Fast” optimization-services may solve more problems than “slow” ones

26

Optimization in processing of experimental data

#26

Fine structure of carbon films deposited in thermonuclear reactor TOKAMAK T-10 by results of synchrotron X-ray scattering diffraction

Experimental data (on scattering angles)

=

2sin4 xq θ

λπ

27

Unexpected interesting result

#27

Dominance of toroidal spatial forms of carbon has been revealed (7 toroidal form over ~500 candidates for all criteria !)A. B. Kukushkin, V. S. Neverov, N. L. Marusov, I. B. Semenov, B. N. Kolbasov, V. V. Voloshinov, A. P. Afanasiev, A. S. Tarasov, V. G. Stankevich, Svechnikov "Few-nanometer-wide carbon toroids in the hydrocarbon films deposited in tokamak T-10" // Chemical Physics Letters (14 March 2011) doi:10.1016/j.cplett.2011.03.036

28

Geodesic data processing REST-serviceREST-service (secured access) on the base on AMPL translator and Ipopt (NLP) solver

29

Geodesic data processing via convex NLP (QP)Restore earth surface by isolines series given as an input data.In collaboration with "Vernadsky Institute of Geochemistry and Analytical Chemistry RAS"

To find zy,x for the mesh (x,y) (heights for (x,y) coordinates) by "global" spline interpolation, i.e. minimization of

with the following constraints:zy,x = ck for (x,y) ∈ Ik (k=1:K) sets of isolines points with fixed heights Typical mesh dimensions Nx, Ny ~ 1000, 600

Thank you for your attention.

Questions?

To apply our experience and available softwarewe are looking for problems requiring optimization modeling.

And we are open for collaboration, http://dcs.isa.ru.

Instead of conclusion

GLPK, GNU LP Kit, www.gnu.org/software/glpkА.О. Махорин, МАИ, ~11 лет (v. 4.45). Симплекс-метод, ''управляемый'' B&B, Gomory-cat, Interior PointMathProg - “диалект” языка AMPL для LP и MILP.(AMPL, GAMS, MATLAB)

LP_SOLVE, lpsolve.sourceforge.net, ~11 лет (v. 5.5.2.0)изначально - в Эйндховенском университете, w3.tue.nl/en/, НидерландыСимплекс-метод, ''управляемый'' B&B.Разнообразное API: Java, Python, PHP (AMPL, GAMS, ZIMPL, MSF, XPRESS, MATLAB, Scilab, Octave, R ...)

Оба пакета: C++ API, MathProg, MPS, CPLEX- и LP-форматы.

Сведения об открытых LP, MILP пакетах

www.coin-or.org/projects/Ipopt.xml, более 6 лет, v. 3.9.3)Interior Point OPTimization, NLP (локальный оптимум)метод Ньютона+“логарифмический барьер”

f x minx∈ℝn

, s.t.

g L ≤ g x ≤ gU , g :ℝnℝm , g L∈ℝ∪{−∞}m , gU∈ℝ∪{∞}m ,xL ≤ x ≤ xU , xL∈ℝ∪{−∞}n , xU∈ℝ∪{∞}n .

A. Wachter (IBM Research Center), L. Biegler (Carnegie Mellon Un.)Построен по модульной схеме:ASL, Blas, Lapack, ...Взаимозаменяемые модули решения систем линейных уравнений (разреженные, с симметричной матрицей): многопоточные: MUMPS (MPI), PARDISO (MPI, GPU),однопоточные: HSL, Harwell Subroutine Library, MA27, MA57(!)

Пакет Ipopt