CONTROLS MIDDLEWARE RENOVATION – TECHNICAL OVERVIEW 26TH JUNE 2013 Wojciech Sliwinski BE-CO-IN for the BE-CO Middleware team: Felix Ehm, Kris Kostro, Joel Lauener, Radoslaw Orecki, Ilia Yastrebov, [Andrzej Dworak] Special thanks to: Vito Baggiolini and Pierre Charrue
28
Embed
Controls Middleware renovation – technical overview 26th June 201 3
Controls Middleware renovation – technical overview 26th June 201 3. Wojciech Sliwinski BE-CO-IN for the BE-CO Middleware team: Felix Ehm, Kris Kostro, Joel Lauener, Radoslaw Orecki, Ilia Yastrebov , [Andrzej Dworak] Special thanks to : Vito Baggiolini and Pierre Charrue. Agenda. - PowerPoint PPT Presentation
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Wojciech Sliwinski BE-CO-INfor the BE-CO Middleware team:
Felix Ehm, Kris Kostro, Joel Lauener, Radoslaw Orecki, Ilia Yastrebov, [Andrzej Dworak]
Special thanks to: Vito Baggiolini and Pierre Charrue
2
Agenda
Context & Motivation for Renovation
Middleware Review process
Technical evaluation of the transport layer
Changes in the MW Architecture in LS1
MW Upgrade milestones in 2013
Conclusions
3
Agenda
Context & Motivation for Renovation
4
Motivations for MW Renovation Current CORBA-based CMW-RDA
Integrated in the Control system Used to operate all CERN accelerators Provides widely accepted Device/Property model > 10 years old
Why to review & upgrade MW ? CORBA was choosen 15 years ago Technical limitations of CORBA-based transport Functional limitations of the current CMW-RDA Codebase with long history difficult to maintain, needs architecture review Major issue of long-term support & future evolution Evolution of technology over last 10 years: HW, OS, middleware, 3rd party libraries Human factor less & less CORBA expertise on the market
5
Technical limitations of CORBA transport Became legacy, not actively supported maintenance issue
Shrinking community, slow response time omniORB (C++) – 1 developer/maintainer, last release mid-2011 JacORB (Java) – few developers, small community
Major technical limitations Lack of fully asynchronous processing channel Blocking communication infamous JacORB blocking issue Lack of low-level control of IO resources (sockets, request queues)
Development issues Difficult to extend the wire protocol Backward compatibility issue Complex, error prone API Heavy in memory usage
6
Summary: Why change CORBA?
CORBA was choosen 15 years ago Not actively maintained big risk for the MW project Better solutions exist on the market Invest in future solution rather than maintaining old one
With current CORBA-based middleware we can’t solve the pending operational issues
We can’t provide better scalability & reliability CMW-RDA is difficult to evolve & extend
MW Review aims to provide the most appropriate technical solution satisfying the user requirements
MW Upgrade establishes the plan & strategy for introduction of the new MW Objective: LS1 the unique opportunity for the major MW upgrade
Middleware Review Process Gathering of users feedback and requirements (2010-11) Review of communication and serialization libraries (2011-12) Prototyping using selected communication products (2012) Design & impl. of new RDA3: Data, Client & Server (2012-13) Testing & validation of core MW infrastructure (summer’13) Upgrade of all dependent MW libraries & services (2013-14)
○ JAPC, Directory Service, Proxy, DIP Gateway
9
Review of users requirements 2010-11 – series of interviews with major users
Lars Jensen, Stephen Jackson (BI) Andy Butterworth, Frode Weierud, Roman Sorokoletov (RF) Brice Copy, Clara Gaspar (DIP, DIM) Frederic Bernard, Herve Milcent, Alexander Egorov (PVSS) Alexey Dubrovskiy (CTF), Kris Kostro (DIP gateways) Marine Gourber-Pace, Nicolas Hoibian (Logging) Nicolas De Metz-Noblat (Front-Ends), Alastair Bland (Infrastructure) Michel Arruat (FESA), Stephen Page (FGC) Niall Stapley, Mark Buttner, Marek Misiowiec (LASER & DIAMON) Nicolas Magnin, Christophe Chanavat (ABT) Stephane Deghaye, Jakub Wozniak (InCA, SIS) Vito Baggiolini, Roman Gorbonosov (JAPC & DA systems) + regular feedback from OP + internal team input
Java & C++ API, Win (64-bit) & Linux (SLC5 32-bit & SLC6 64-bit) Accelerator Device Model (i.e. Device/Property) Get, Set, Async-Get, Async-Set, Subscribe Early detection of communication failures Improve error reporting in all the layers: client, server, gateways Admin interface & runtime diagnostics & statistics
Data support Data object: primitives, n-dim arrays, data structures
Subscription mechanism Subscription behaviour the same regardless condition of the server (active, down) Several client subscription policies (default: continuous) Provide subscription notification ordering First-Update enforced via CMW on server-side
○ Provide callback to front-end framework for the server-side Get Drop support for on-change flag Standardise use of subscription filters and update flags (e.g. immediate update) Add header for acquired Data common metadata (e.g. acq. stamp, cycle name) All loss of data (dropped updates) must be notified to clients
New requirement
11
New RDA3: Accepted requirements Client side
RDA3 client API connects with both: RDA2 (old) & RDA3 (new) servers Efficient mechanism for: connection, disconnection & reconnection Must be able to recover from any interruption of communication with the server
○ Server restarts, IP address change, rename/move of a device to another server Improved semantics of Array Calls, i.e. handling of individual parameters Enhanced diagnostics & collection of statistics
Server side Policies for discarding notifications, i.e. deal with overflows and ’bad clients’
○ Instrument with counters & timings allowing to diagnose the notifications delivery Prioritisation of Get/Set requests for high-priority clients Server-side subscription tree fully managed by CMW
○ Server does not need to manage client subscriptions any more Manage the client connections, e.g. forced disconnect of a client Client lifetime callbacks (i.e. connected, disconnected)
New requirement
12
New RDA3: Accepted requirements Server side (cont.)
Client discovery for the diagnostics purposes (i.e. connected clients with payload) Enhanced diagnostics & collection of statistics
Ongoing discussions (not accepted yet) Prioritisation of subscription notifications for high-priority clients
Technical notes Invest in asynchronous & non-blocking communication Prefer 0-copy & lock-free data structures, message queues
New functionality Policies for subscription management (client & server)Client prioritiesServer-side subscription treeExtended Data supportStandardise First-Update concept
14
Agenda
Technical evaluation of thetransport layer
15
Middleware transport requirements
Desirable
Mandatory
Fundamental
Lightweight
Friendly API, documentation
Request/reply & pub/sub patterns
Open source license
Asynchronous
Active community
Stability, Maturity & Longevity
Performance & Scalability
C++/Java
Linux/Windows
Over TCP/IP LAN
16
Evaluated middleware products
Ice
Thrift
omniORB
YAMI
OpenSpliceDDS
OpenAMQCoreDXRTI DDS
ZeroMQ
QPid
MQtt RSMBJacORB
Mosquito
All opinions are based only on our knowledge and evaluation. Each of the products, depending on the requirements, may constitute a good solution.
RabbitMQ
Andrzej Dworak, ICALEPCS 2011
17
Products comparison (according to the criteria)
Sync, async & msg patterns
QoS
Dependencies & memory f-p
Performance
Look & feel, API, docs
Community & maturity
Score
ZeroMQ 6Ice 5
YAMI4 4RTI 3
Qpid 3CORBA 2
Thrift 2
Andrzej Dworak, ICALEPCS 2011
18
Conclusions Several good middleware solutions available The choice is dictated by the most critical requirements Not easy performance matters but also ease of use, community, … Prototyping was done with the most promising candidates:
ZeroMQ, Ice & YAMI
Finally we decided to choose ZeroMQ (http://www.zeromq.org/) Asynchronous & non-blocking communication 0-copy & lock-free data structures, message queues Nice API, good documentation & active community