1 www.d4science.eu Tutorial Outline 30’ •From Content Management Systems to VREs 50’ •Creating a VRE 80 •Using a VRE 20’ •Conclusions
Dec 27, 2015
1
www.d4science.eu
Tutorial Outline
30’ •From Content Management Systems to VREs
50’ •Creating a VRE
80 •Using a VRE
20’ •Conclusions
From Content Management Systems to VREs: requirements, challenges, and opportunities
Pasquale [email protected]
RCDL 2008
10 October 2007Dubna (Russia)
www.d4science.eu
3
www.d4science.eu
Session Outline
• Towards VREs A bit of history Delos DL Reference Model VREs, VOs, and e-Infrastructures D4Science Vision
• gCube System Characteristics Technological Complexity and Solution Architecture
• gCube Standards and Technology SOI, WSRF gCore
• gCube Enabling Services Features Opportunities
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
4
www.d4science.eu
The evolution
Virtual Research Environments
many virtual organizations
Repository +Catalogue +Search service
few large institutions
1996consumer
Digital Library
few small institutions
2001 consumer
2006 consumer and resource provider
Digital Library Management System
many small institutions
2003 consumer and data provider
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
5
www.d4science.eu
The DL Universe
DIENST
ADEPT
NSDL
NDLTD
DSPACE
FEDORATEL
DRIVER
BRICKS DILIGENT
ACM DL
ECHO
PERSEUS
OPENDLIB
EUROPEANA
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
6
www.d4science.eu
DELOS: Digital Library definition
A (potentially virtual) organization that comprehensively collects, manages, and preserves for the long term rich digital content and offers to its user communities specialized functionality on that content, of measurable quality, and according to prescribed policies
* DELOS Reference Model for Digital Libraries
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
7
www.d4science.eu
DL Reference Model: the 7 Domains
The Reference Model is founded on 6+1 domains:
Resource – captures generic characteristics (super-domain)
Content – information available
User – actors interacting with system
Functionality –operations supported
Policy – rules and conditions governing operation
Quality – qualitative & quantitative characterisations of system
Architecture –physical software (and hardware) constituents concretely realising the DL* DELOS Reference Model for Digital Libraries
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
8
www.d4science.eu
DL Reference Model: the 7 Domains [cont.]
* DELOS Reference Model for Digital LibrariesFrom Content Management Systems to VREs10 October 2008, Dubna (Russia)
9
www.d4science.eu
Reference Frameworks
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
10
www.d4science.eu
Virtual Research Environments (VRE)
• Distributed frameworks for carrying out cooperative activities like “in silico experiments”, data analysis and processing, production of new knowledge using specialized tools
• Largely based on retrieval and access of always updated knowledge from diverse heterogeneous content sources
• Produce knowledge that is preserved and made available for other usages inside and outside the VRE
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
11
www.d4science.eu
Virtual Research Environments: characteristics
Highly dynamic, created and dismissed on-demand
Based on specialised tools which support the generation of new knowledge
M26
0
0,2
0,4
0,6
0,8
1
1,2
Info
rmaio
n S
erv
ice
Bro
ker
& M
atc
hm
aker K
eep
er
DV
OS
VD
L G
en
era
tor
Con
ten
t M
an
ag
em
en
t
Wra
pp
er
& M
on
itor
Con
ten
t S
ecu
rity
Meta
data
Bro
ker
An
nota
tion
Meta
data
Man
ag
em
en
t
Data
Fu
sio
n
CS
DS
Pers
on
ali
zati
on
Ind
ex S
erv
ice
Searc
h S
erv
ice
Featu
re E
xtr
acti
on
Serv
ice
Pro
cess D
esig
n &
Veri
ficati
on
Pro
cess E
xecu
tion
& R
eli
ab
ilit
y
Pro
cess O
pti
miz
ati
on
Art
e P
ort
al
Imp
EC
t P
ort
al
PrototypeAvailableBuild
Operating on new information objects
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
12
www.d4science.eu
Virtual Research Environments: the vision
• A Virtual Research Environment (VRE) provides a framework of applications, services and data sources dynamically identified to support the underlying processes of research/collaboration/cooperation.
The purpose of a VRE is to help researchers* belonging to Virtual Organization by managing the increasingly complex range of tasks involved in carrying out their activities.
*Researcher has to be considered in the large, i.e. end-user, decision-makers, resource and data providers, etc.
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
13
www.d4science.eu
Virtual Organization (VO)
• A Virtual Organization (VO) models sets of users and resources belonging to a e-Infrastructure. It defines clearly and carefully
what is shared, who is allowed to share, and the conditions under which sharing occurs, usually
based on an authentication and authorization policies.
VOs may have a limited lifetime and they are dynamically created to satisfy transient needs of the constituent potentially heterogeneous actual Organizations.
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
14
www.d4science.eu
e-Infrastructure
• An infrastructure is the basic physical and organizational structures and facilities (roads, power supplies, ..) needed for the operation of a society or enterprise
• An e-Infrastructure provides support for effective consumption of shared resources:
hardware-bound resources (i.e. networks, storage, instruments, and computational resources),
system-level software resources (i.e. basic middleware services), and application-level software resources (i.e. data sources and services).
These e-Infrastructures offer mechanisms that concurrently exploit networks, grids and data in a seamless fashion, and will thus enable scientific communities to operate within a coherent model, regardless of the location of their research facilities.
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
15
www.d4science.eu
D4Science vision
D4Science visioncalls for the realization of scientific e-Infrastructures that will remove technical concerns from the minds of scientists, hide all related complexities from their perception, and enable users to focus on their science and collaborate on common research challenges
gCube isa framework to manage distributed e-infrastructures where it is possible to define, host, and maintain dynamic virtual environments capable to satisfy the collaboration needs of distributed Virtual Organizations (VOs)
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
16
www.d4science.eu
gCube System
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
17
www.d4science.eu
gCube Characteristics
A Software framework
to support ON-DEMAND virtual collaborations* among remote parties cost-effective, secure, dynamic, both short and long lived overcome ad-hoc systems alike
to make discoverable and accessible computing, storage, data and service resources
to promote and/or contribute to data and service integration
* Research EnvironmentFrom Content Management Systems to VREs10 October 2008, Dubna (Russia)
18
www.d4science.eu
gCube Technological Complexity
Software framework
needs a ‘middleware’ (typically distributed)
is open by definition new resource types and/or new resource instances can be de/registered at any
time
is powerful if it supports application scope the portion of the infrastructure in which a resource exists the portion of the infrastructure in which a resource can act, operate, or has
power or control
is powerful if it supports sharing scope (controlled resource sharing) machines, storage, data and services resources
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
19
www.d4science.eu
gCube Technological Solution
• By relying on gLite, gCube is an e-Infrastructure enabling system to share
computing, storage, data and service resources → g3
• gCube system allows collaborations in eScience strongly content-oriented, potentially data and processing intensive
within the sharing scope of Virtual Organizations (VOs) broader and longer lived may stretch across the whole infrastructure or else over significant subsets thereof
take place in Virtual Research Environments (VREs) scope interactively created, managed, defined, and used:
system administrators, application designers, researchers typically short to medium lifespan
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
20
www.d4science.eu
gCube Architecture
gCube framework is composed by 4 main subsystems:
Enabling Services definition and runtime management of VREs
Information Organization Services storage, management, description, and annotation of information in a VRE
Information Retrieval Services retrieval of information in a VRE
Presentation Services VRE users interface with system and application services
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
21
www.d4science.eu
gCube Standards and Technology
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
22
www.d4science.eu
Is a Virtualised IT Infrastructure which
1. exposes a catalog of services instead of running service instances,
2. supports Workflow definition and execution, and
3. includes infrastructure resources such as compute, storage, and networking hardware and software (middleware) to support the running of services.
Is a Virtualised IT Infrastructure which
1. exposes a catalog of WS instead of running service instances,
2. supports SOA Application, and
3. includes infrastructure resources such as compute, storage, and networking hardware and software (middleware) to support the running of services.
Service Oriented Infrastructure
gCube provides a production quality software framework to enable scientific e-Infrastructures empowered by a collaborative environment
SOI
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
23
www.d4science.eu
gCube & Standards for communication
• Java WSCore, Apache Axis, and GridForum specifications (and implementation if any):
WS-Notification, WS-Addressing, WS-Security
WSRFWS-ResourceProperties (WSRF-RP)WS-ResourceLifetime (WSRF-RL)WS-BaseFaults (WSRF-BF)WS-ServiceGroup (WSRF-SG)
WS-DAI, WS-DAIR, WS-DAIX
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
24
www.d4science.eu
Web Service Resource Framework (WSRF)
Web Services
Lifetime Properties Service
GroupsNotifi-cation
ErrorHandling
Unified way to model and interact with stateful web services
Logic
WS
Logic
Statefull WS
State
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
25
www.d4science.eu
Web Service Resource Framework (WSRF)
Lifetime (WS-ResourceLifetime) Factory based dynamic creation of
services Instances are created with a limited
lifetime Prevent services from consuming
resource indefinitely (“Garbage Collection”)
Properties (WS-ResourceProperties) Defines type and values of a resource
state
Web Services
Lifetime Properties Service
GroupsNotifi-cation
ErrorHandling
Service groups (WS-ServiceGroups) Describes an interface for operating on
collections of WS-Resources E.g. to distribute an action to a set of
services
Notification (WS-Notification) Notification about state changes Applies traditional publish/subscribe
paradigm
Error Handling (WS-BaseFaults) Defines base handling of communication
errorsFrom Content Management Systems to VREs10 October 2008, Dubna (Russia)
26
www.d4science.eu
WSRF: Advantages and Disadvantages
Advantages Clear Description of Resources and Interfaces Dynamic sharing of resources On-demand services exploitation Cross-organizations trusted environment Widely accepted Web Service standards
Disadvantages Reference implementations are still in development Several complementing specifications are in development Complex middleware requires maintenance and
administration overhead
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
27
www.d4science.eu
Towards the usability: gCore
To overcome the complexities of the design and implementation of SOI compliant services
an application framework for the consolidation / development of existing/new gCube services
the gCore Framework (gCF)
To meet the needs of system administrators, infrastructure managers, and resource providers
an easy to install self-contained service container for the participation to Service Oriented Infrastructures
the gCore Container (gCore)
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
28
www.d4science.eu
gCF facilities: partial overview
WS management
lifetime
activation update
re-deployement
failure
State
configuration
models
lifetime
scoping
publication
persistence
notification
Calls
scoping
faults
security
scoping
serviceport-types
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
29
www.d4science.eu
gCube & Standards for communication
• Mutual authentication based on GSI secure conversation (through delegation and renewal)
• Business Process Execution Language for Web Services (WS-BPEL)
• GridFTP and SRM support
• VOMS for users and groups management
• GWT and JSR168(JSR268 is coming)
https://quality.wiki.d4science.research-infrastructures.eu/quality/index.php/Standards
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
30
www.d4science.eu
gCube Enabling Services
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
31
www.d4science.eu
gCube Enabling Services
• VRE Management services definition of VREs the dynamic deployment of VRE resources across the infrastructure
• Software Repository service Storage and provision of deployable software components
• Information Service publication of resources profile discovery of VRE resources through xPath, XQuery real-time monitoring subscription/notification
Dynamic Virtual Organisation Support services robust and flexible security framework for managing VOs
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
32
www.d4science.eu
HWgHN
HWgHN
Service provision continuity
HWgHN
HW gHN
Balancing utilization with head room
Dynamic Load Balancing
WS
State
WS
State
CPU Usage
30%
CPU Usage
90%
Rapid deployment
Production
HWgHNSoftware
Repository
WS
…WS
State
WS
State
WS
State
WS
State
gCube Enabling Services
Soon available
Failure Recovery
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
33
www.d4science.eu
gCube goes beyond other management systems by adding Service Management capabilities
Provides solutions for system administrators to
eliminate manual deployment overheads, eliminate manual environment configuration overheads, ensure optimal placement of services within the infrastructure support user community services orchestration
Opens unique opportunities for outsourcing state-of-the-art service implementations
gCube Enabling Services
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
34
www.d4science.eu
Conclusions
• gCube infrastructures creates new opportunities to change the VRE development model used by distributed and dynamic organisations and communities
• gCube offers an application framework for the development of Stateful
Web Services (gCF) an easy to install self-contained service container (gCore) SOI middleware (Enabling services)
Rapid deployment Failure recovery Load balance (soon)
From Content Management Systems to VREs10 October 2008, Dubna (Russia)
35
www.d4science.eu
http://www.gcube-system.org/ http://www.d4science.org/
36
www.d4science.eu
gCube & Standardization Bodies
• ISO: data representation (e.g. ISO3166 for countries, ISO4217 for currencies) and metadata (ISO19115 for GIS)
• OGF: Standards related to Architecture (e.g. OGSA), Data (e.g. DAIS, GridFTP), Management (e.g. GLUE, Resources Usage), Applications (e.g. DRMAA), Compute (e.g. JSDL)
• OAI: Resources Exposure/Harvesting (OAI-PMH) Resources Exchange (OAI-ORE)
• OASIS: Standards related to stateful web services (e.g. WSRF), process management (BPEL), remote user interfaces (WSRP), A&A (SAML / XACML)
• W3C: All the standards related to Web Architecture (e.g., URI/URL, HTTP), Service Oriented Architectures (e.g. SOAP, WSDL, WS-Addressing) and data representation and manipulation( e.g. XML*)
• Others: Classification systems (e.g. ISSCAAP, ISSCFV, ISSCFG), features representation (e.g. GML for GIS), metadata (e.g. AgMES for Agricultural, SDMX for Statistics)
From Content Management Systems to VREs10 October 2008, Dubna (Russia)