The Building of Thai Grid

Building a National Grid Infrastructure: A ThaiGrid Experience

Putchong Uthayopas,Department of Computer Engineering,Faculty of Engineering,Kasetsart University, [email protected]

The Need for National E-Infrastructure

• Efficient Resources Management for Government

• Information dissemination for people

• Emergency Response– Natural Disaster: Flooding, earth

quake, Tsunami• Better Quality of Live– Environmental monitoring,

protection

National E-Infrastructure• Fast computer

– massive computing power • Large and fast storage for data

intensive applications• Fast network

– Information sharing– Collaboration

• Middleware– Security, connectivity, access

• Portal– Hiding the complexity of access,

data movement, and execution from users

August 29,2008 TGCC 2008, Khon Khan University , Thailand

Introduction to ThaiGrid

• A National Project under Software Industry Promotion Agency (Public Organization) , Ministry of Information and Communication Technology

• Started in 2005 from 14 member organizations

• Expanded to 22 organizations in 2008

TGCC 2008, Khon Khan University , Thailand

ThaiGrid : an Enabling Platform for Innovation

August 29,2008

Thai Grid Infrastructure

2.5 G

bps

1 Gb

ps

1 G

bps 1

Gbp

s

1 Gbps

2.5 Gbps155 Mbps

310

Mbp

s15

5 M

bps

310

Mbp

s155

Mbps

155 Mbps

155

Mbp

s

155 Mbps

155

Mbp

s

19 sitesAbout 1000 CPU core.

August 29,2008 TGCC 2008, Khon Khan University , Thailand

ThaiGrid Usage• ThaiGrid provides about 290

years of computing time for members– 9 years on the grid– 280 years on tera

• 41 projects from 8 areas are being support on Teraflop machine

• More small projects on each machines

TGCC 2008, Khon Khan University , ThailandAugust 29,2008

Medicinal Herb Research• Partner

– Cheminormetics Center, Kasetsart Univesity (Chak Sangma and team)

• Objective– Using 3D-molecular databse and virtual

screening to verify the traditional medicinal herb

• Benefit– Scientific proof of the ancient

traditional drug – Benefit poor people that still rely on the

drug from medicinal herb – Potential benefit for local

pharmaceutical industry

TGCC 2008, Khon Khan University , Thailand

Virtual Screening

Infrastructure

Lab Test

August 29,2008

NanoGrid

• Objective– Platform that support computational Nano science

research• Technology used

– AccelRys Materials Studio– Cluster Scheduler: Sun Grid Engine and Torque

TGCC 2008, Khon Khan University , Thailand

AccelRys Materials Studio

Materials Studio Gateway

Cluster Scheduler Cluster Scheduler

Grid Infrastructure

ThaiGrid MS-Gateway

MS-Gateway

Computing ResourcesComputing Resources

1

2

3

August 29,2008

Building ThaiGrid

Challenges

• Size and Scale• Manageability– Deployment– Configuration– Operation

• Software and Hardware Compatibility

Principle Design Goal

• Openness• Usability• Functionality• Performance

Some statistics about Thai Grid Infrastructure

• 21 clusters from 16 sites around the country• More than than ~1,000 cores• A real super computer 4.6 Teraflops machines– 200 nodes of 800 cores of Intel Xeon 3.0GHz

• ThaiGrid provides more than 111 years of computing time for member– 7 years on the grid and more than 104 years on tera

server• More than 30-40 projects from 8 areas are being

support

Grid System Architecture

• Clusters– Satellite Sets

• 16 clusters delivered from ThaiGrid for initial members

• Composed of 5 nodes of IBM eServer xSeries 336 – Intel Xeon 2.8Ghz (Dual Processor)– x86_64 architecture– Memory: 4 GB (DDR2 SDRAM)

– Other sets• Various type of servers and

number of nodes • Provided by member institutes of

ThaiGrid

Grid Software Architecture

Cluster OS:Linux / Rocks 4.0 – 4.3 on x86/x86_64 machines

Grid Middleware:Globus Toolkit 4.0

Management/Monitoring:

SCMS

Grid Scheduler:GridWay

Programming:Ninf-G

MPICH-G2Globus WSRF

Grid Applications

Management/Monitoring:

SCMS,Ganglia

Grid Scheduler:SGE, PBS,

Condor, SCMS

Programming:MPICH

OpenMPI

Thai

Grid

Rol

l (P

rovi

de a

uto-

confi

gura

tion)

UNINET (10Gbps)

Production CA Infrastructure

CA RA OnlineRepository

Secure WebInterface

CA Operator RA Operator User

secure

secure

face-to-faceVTC or Skype

Experiences

• Building a quick deployment technology is a key!– NPACI Rocks is a key technology being used– Collaboration SDSC/ TNGC under PRAGMA– ThaiGrid Roll is built to facilitate the fast deployment

time• Building a good infrastructure monitoring and

management system– Know what your grid are doing.– Develop system monitoring, accounting, testing tools

Grid Monitoring and Scheduling

• Using SCMSWeb to monitor and probe for Grid status– Grid services probe is mainly used to categorize services

level in ThaiGrid– Used in resource certification system

• Grid Meta Scheduler– Meta scheduler is installed in central site– GridWay 5.2 is installed and experiment in some projects

• Drug Design Grid• Information is pulled from SCMSWeb

C C C C

H

C C C C

H C C C C

H

C C C C

HREN

GCC

Grid Scheduler

Grid as a Super Cluster

TGCC 2008, Khon Khan University , ThailandAugust 29,2008

Experiences

• Strong standard compliance is critical to make the systems interoperable

• Site certification program– Level 1: Network firewall

compliance– Level 2: Middleware

compliance (MPI/G2, Ninf-g)– Level 3: Job scheduling

compliance– Level 4: Data Grid compliance

TERA Cluster

August 29,2008 TGCC 2008, Khon Khan University , Thailand

Edge Switch 1Gbps Ethernet

FESunyata

FEAraya

WinHPC(FE)

TERA(FE)

SPARE1(FE)

SPARE2(FE)

FS1

FS2

FS3

FS4

4 nodes 4 nodes 64nodes

96 nodes +

16 sparenodes

200 Ports Gigabit Ethernet switch

Storage Tier 5TB Lustre FS

Anatta(FE)

15nodes

KU Fiber Backbone (1Gbps Fiber)

2.5Gbps to UninetStorage 48 TB

1 Gbps Ethernet/Fiber

• 1 Frontend (HP ProLiant DL360 G5 Server) and 192 computer nodes– Intel Xeon 3.2

GHz (Dual core, Dual processor)

– Memory 4 GB (8GB for Frontend & infiniband nodes)

– 70x4 GB SCSI HDD (RAID1)

• 4 Storage Servers– Lustre file

system for TERA cluster's storage

– Attached with Smart Array P400i Controller for 5TB space

Experience

• For a large user group, users will cluster them self based on their application interest. Create a divers set of requirement in term of Operating system, Scheduler and End user application– Single large cluster may not be used effectively

• Partition large cluster into several set of partitionthat can be modified seems to work well.– We have two set of infiniband for parallel job– Unix and windows for single batch/interactive

Future

• The world are moving toward Cloud– More cost effective– Better manage through centralized control– More elastic and on demand

• Thailand cover with cloud– E-government cloud– National Crisis Management Cloud– Research Cloud– Education Cloud

• Support mobile learning

• Nationwide broadband infrastructure is needed– 4G/4G , Wifi, ADSL , Fiber to Home

Acknowledgement• I would like to Thank you

– Dr. Vara Varavithya for being the co-founder of the project and spent numerous hours helping on laying out many technical detail.

– Dr. Arnon Rungsawang and ThaiGrid staffs for the hard work during the project to prove that we can do it!

– All the university contact persons for helping make our dream come true for a while

– Kasetsart University that allow me to do this for our country.

– MICT, SIPA, NECTEC, NANOTECH, NSTDA for their support

• Any honor and award should go to them for their contributions . I can not do this alone!

Thank you