& & Meta-cluster of Meta-cluster of distributed computing distributed computing “Dubna-Grid” “Dubna-Grid” Korenkov Korenkov Vladimir Vladimir JINR, Dubna JINR, Dubna
Jan 02, 2016
Grid activities in JINRGrid activities in JINR &&
Meta-cluster of distributed Meta-cluster of distributed computing “Dubna-Grid”computing “Dubna-Grid”
Korenkov VladimirKorenkov Vladimir
JINR, DubnaJINR, Dubna
In 2006 the Joint Institute for Nuclear Research (JINR) - an international intergovernmental scientific research organization
- celebrated its 50-th anniversary.
JINR Member StatesJINR Member StatesArmeniaArmenia
AzerbaijanAzerbaijan
BelarusBelarus
BulgariaBulgaria
CubaCuba
Czech RepublicCzech Republic
GeorgiaGeorgia
KazakhstanKazakhstan
Democratic People’s Democratic People’s Republic of KoreaRepublic of Korea
MoldovaMoldova
MongoliaMongolia
PolandPoland
RomaniaRomania
RussiaRussia
Slovak RepublicSlovak Republic
UkraineUkraine
UzbekistanUzbekistan
VietnamVietnam
JINR is a large multidisciplinary scientific centre incorporating:
– basic research in frontier particle, nuclear and condensed matter physics,
– development and application of high technologies, and
– university education in the relevant fields of knowledge.
International CooperationInternational Cooperation
JINR’s partners are about 700 institutions located in 60 countries
AUSTRIABELGIUM
CERNCROATIA
DENMARKFINLANDFRANCE
GREECE
IRELAND
NETHERLANDSNORWAY
PORTUGAL
SLOVENIASPAIN
SWEDENSWITZERLAND
YUGOSLAVIAUNITED KINGDOM
BULGARIA
CZECH REPUBLIC
GERMANY
HUNGARY
ITALY
EUROPE
DUBNA
AFRICAEGYPT
SOUTH AFRICA
AUSTRALIA AND
OCEANIAAUSTRALIA
REPUBLICSOF
FORMER USSR
AMERICA
BRAZILCANADA
UNITED STATES
CHINA
ISRAELJAPAN
SOUTH KOREATURKEY
INDIA
DEMOCRATIC PEOPLE'S REPUBLIC OF KOREA
MONGOLIA
VIETNAM
ASIA
BRAZILCANADA
CLAFCUBA
UNITED STATES
Science Bringing Nations TogetherScience Bringing Nations Together
CERN and JINR joint exhibitions:CERN and JINR joint exhibitions: 1997:1997: University of Oslo (Norway) University of Oslo (Norway) 1998:1998: UNESCO, ParisUNESCO, Paris 1999:1999: UN Office in GenevaUN Office in Geneva 2000:2000: European Parliament in BrusselsEuropean Parliament in Brussels 2001:2001: Russian State Duma, MoscowRussian State Duma, Moscow 2002:2002: Ministry of Education and Ministry of Education and
Sciences, Bucharest, RomaniaSciences, Bucharest, Romania 2003:2003: Yerevan University, ArmeniaYerevan University, Armenia
Diplomatic Academy, MoscowDiplomatic Academy, Moscow 2005:2005: University of University of Thessaloniki, GreeceThessaloniki, Greece
GenevaGeneva
BrusselsBrussels
MoscowMoscowBucharestBucharestThessalonikiThessaloniki
At present the JINR LAN comprises more 6000 computers and nodes (Gigabit Ethernet)
Local Area Network Backbone
- support and development support and development LCG/EGEE infrastructure;LCG/EGEE infrastructure;- participation in LCG middleware testing/evaluation, participation in LCG middleware testing/evaluation, - participation in Data and Service Challenges,participation in Data and Service Challenges,- CASTOR and dCache usage/development; CASTOR and dCache usage/development; - grid monitoring and accounting system development;grid monitoring and accounting system development;- EGEE-RDIG portal support and development, EGEE-RDIG portal support and development, - MCDB development; MCDB development; - HEP application; HEP application; - User & Administrator Training and Education
- support of JINR member states in the LCG/EGE activities.support of JINR member states in the LCG/EGE activities.
JINRJINR in the LCG/EGEE Project in the LCG/EGEE Project
RDIGRDIG monitoring&accountingmonitoring&accounting http://rocmon.jinr.ru:8080http://rocmon.jinr.ru:8080
The “Dubna-Grid” Project is aimed at the The “Dubna-Grid” Project is aimed at the creation of a distributed environment of meta-creation of a distributed environment of meta-computing on the basis of vacant computing computing on the basis of vacant computing resources of “office” computers. resources of “office” computers. Implementation of the project “Dubna-Grid” will allow:Implementation of the project “Dubna-Grid” will allow:
to create a unified computing environment of the city of Dubna (meta-cluster) on to create a unified computing environment of the city of Dubna (meta-cluster) on the basis of the resources of scientific and educational institutions, in particular, the basis of the resources of scientific and educational institutions, in particular, JINR subdivisions, University “Dubna”, secondary schools and other organizations JINR subdivisions, University “Dubna”, secondary schools and other organizations concerned; concerned; to create a segment of the international Grid-infrastructure, operating in frames of to create a segment of the international Grid-infrastructure, operating in frames of various Grid systems (LGG, NorduGrid and OSG). To perform research of control, various Grid systems (LGG, NorduGrid and OSG). To perform research of control, security and stability within the Grid environment.security and stability within the Grid environment. to provide execution of large-scale computing tasks of JINR and other enterprises to provide execution of large-scale computing tasks of JINR and other enterprises of Dubna scientific – industrial complex and other organizations concerned;of Dubna scientific – industrial complex and other organizations concerned; to distribute experience of creation of the city segment of the Grid-infrastructure to distribute experience of creation of the city segment of the Grid-infrastructure for creation of similar systems in other Russian cities.for creation of similar systems in other Russian cities.
Meta-cluster of distributed computing “Dubna-Grid”Meta-cluster of distributed computing “Dubna-Grid”
DUBNA-GRIDDUBNA-GRID
Main stages of realization Main stages of realization
development of technologies on realization of a development of technologies on realization of a separate administration of office computers and separate administration of office computers and nodes of the Grid infrastructure on the basis of nodes of the Grid infrastructure on the basis of software shells of virtual machinessoftware shells of virtual machines
creation of a meta-cluster prototype integrating creation of a meta-cluster prototype integrating resources of various organizationsresources of various organizations
development of mass installation technologies and development of mass installation technologies and spreading software to all accessible nodes of the city spreading software to all accessible nodes of the city infrastructureinfrastructure
installation of various Grid systems at the meta-installation of various Grid systems at the meta-cluster nodes and accordance of these resources to cluster nodes and accordance of these resources to the users of other Grid projects such as LCG, the users of other Grid projects such as LCG, NorduGrid, EGEE, OSG.NorduGrid, EGEE, OSG.
training of specialists in the field of Grid technologies. training of specialists in the field of Grid technologies. Holding seminars, giving lectures and Holding seminars, giving lectures and demonstrations.demonstrations.
City Informational-Educational Computer Network “Dubna”
Network resources of the Project
JINR Backbone
Construction of the Construction of the systemsystem
The Meta-cluster can be interpreted as a cluster, but the The Meta-cluster can be interpreted as a cluster, but the computing resources are distributed over different places. computing resources are distributed over different places. Basically meta-clusters are constructed from resources Basically meta-clusters are constructed from resources which are specially provided for it, but the “Dubna-Grid” which are specially provided for it, but the “Dubna-Grid” meta-cluster uses vacant computer resourcesmeta-cluster uses vacant computer resources of office of office computers in parallel with their main purpose. computers in parallel with their main purpose.
Dubna-Grid Meta-cluster includes a managed server, a Dubna-Grid Meta-cluster includes a managed server, a bridge and computational nodes. Computational nodes bridge and computational nodes. Computational nodes represent Virtual PCs emulated by virtual machine represent Virtual PCs emulated by virtual machine technology (VMware).technology (VMware).
By means of VMware we have two computers instead of By means of VMware we have two computers instead of one, there are no differences between real and virtual PCs, one, there are no differences between real and virtual PCs, they both have its own memory, IP address, processor and they both have its own memory, IP address, processor and etc, but physically they both use the same resources. Two etc, but physically they both use the same resources. Two machines share one processor, memory, Ethernet card and machines share one processor, memory, Ethernet card and all of PC devices. As all nodes in the cluster represent the all of PC devices. As all nodes in the cluster represent the virtual PCs, we have absolutely a homogenous virtual PCs, we have absolutely a homogenous environment for building the infrastructure. environment for building the infrastructure.
Construction of the systemConstruction of the system
In order to create the computing meta-cluster the following In order to create the computing meta-cluster the following approaches were used:approaches were used:
time-shared equipment with common access to the resources;time-shared equipment with common access to the resources; common network infrastructure for real and virtual resources common network infrastructure for real and virtual resources
and own additional network infrastructure for virtual resources and own additional network infrastructure for virtual resources virtual clustering of the equipment.virtual clustering of the equipment.
The following software tools and technologies were used:The following software tools and technologies were used: software support for virtual machines maintenance (VMware),software support for virtual machines maintenance (VMware), virtual network (VLAN),virtual network (VLAN), virtual access to the software and data (AFS),virtual access to the software and data (AFS), building and loading OS images to the Virtual PCs of the meta-building and loading OS images to the Virtual PCs of the meta-
cluster (Warewulf package).cluster (Warewulf package).
On the central server the following software is installed:On the central server the following software is installed: Scientific Linux CERN OS; Scientific Linux CERN OS; a package for support of cluster architecture Warewulf; a package for support of cluster architecture Warewulf; Ganglia monitoring system; Ganglia monitoring system; OpenAFS, OpenAFS, the batch system Torque with Maui scheduler. the batch system Torque with Maui scheduler.
ConclusionConclusion
A distributed meta-computing environment of A distributed meta-computing environment of Dubna city based on vacant computing Dubna city based on vacant computing resources (computational, storage) of office resources (computational, storage) of office computers of scientific and educational computers of scientific and educational institutions has been created.institutions has been created.
Mass installation technologies and spreading Mass installation technologies and spreading software to all accessible nodes of the city software to all accessible nodes of the city infrastructure have been developedinfrastructure have been developed
Virtual meta-clusters have been integrated Virtual meta-clusters have been integrated with JINR batch system with JINR batch system
Monitoring system of the meta-cluster has Monitoring system of the meta-cluster has been developedbeen developed
First real tasks have been runFirst real tasks have been run, , including including simulation of ttbar simulation of ttbar 4m in ATLAS detector 4m in ATLAS detector
Grid infrastructure in University Centre of JINRGrid infrastructure in University Centre of JINR
Grid infrastructure is a Grid infrastructure is a set of virtual machines set of virtual machines (VMs) running on physical (VMs) running on physical ones (hosts)ones (hosts) Virtualisation was made Virtualisation was made using User Mode Linuxusing User Mode Linux current number of VMs is current number of VMs is 36 (6 VMs on each of 6 36 (6 VMs on each of 6 hosts)hosts) all virtual resources are all virtual resources are grouped into independent grouped into independent testbeds which in turn can testbeds which in turn can be used for different aims: be used for different aims: system administrators and system administrators and users training in grid field, users training in grid field, debugging and testing debugging and testing custom grid services in custom grid services in desirable grid environmentdesirable grid environment
● Course for system administrators using Nordugrid ARC middleware, LCG and gLite was successfully conducted on that infrastructure