Accounting Manager Taking resource usage into your own hands Scott Jackson [email protected] .

Accounting ManagerAccounting ManagerAccounting ManagerAccounting Manager

Taking resource usage into your own hands

Scott [email protected]

http://www.emsl.pnl.gov/docs/mscf/gold

2

Typical problems faced by HPC managersTypical problems faced by HPC managersTypical problems faced by HPC managersTypical problems faced by HPC managers

Problem SolutionResources are hogged by the projects or project members that submit jobs most aggressively

Fine-grained allocations

Projects start slow then simultaneously demand their entire allocations be fulfilled at end of project cycle

Expiring allocations (use-it-or-lose-it)

Users can exceed their allocations until accounting is synchronized with resource management system

Dynamic Charging, Reservations

Would like to delegate the management and distribution of credits to established project management hierarchy

Role-based AC, Nested accounts

Want to share resources with other organizations but face issues of security, autonomy and accountability

Strong security, Distrd accounting, Charge quotes

Each site wants to track and perform historical queries on site-specific accounting statistics

Custom accounting, Flexible charging, Journaling

You want the accounting system to be transparent to the user

Default projects, Credit accounts, Auto-generation

3

IntroductionIntroductionIntroductionIntroduction

Gold is an open source accounting system that allocates and tracks resource usage on High Performance Computers.

A resource bank (allocation management system) Much like a bank, it associates a cost to computing resources and allows resource

credits to be allocated to users and projects. As jobs complete or as resources are utilized, projects are dynamically charged and resource usage recorded.

An accounting system Can be dynamically customized to record any type of accounting data – pacct, sar,

node availability, performance data, etc.

An information service Also functions as a powerful generalized information service useful in a variety of

situations, such as providing meta-scheduling mappings of machines to resources, applications, accounts, users, etc.

4

BackgroundBackgroundBackgroundBackground

SciDAC Scientific Discovery through Advanced Computing – A

DOE initiative to improve the impact of scientific computing

Scalable Systems Software Project (SSS) Research, develop and support an integrated suite of

systems software and tools for the effective management and utilization of the highest scale computational resources. 5 yr project involving 12 sites

QBank Gold is based on a successful program, QBank, that

has been used for years on government and university computing systems.

5

Allocation ArchitectureAllocation ArchitectureAllocation ArchitectureAllocation Architecture

Time Period Allocation

F1Q04 1,000,000

F2Q04 1,000,000

F3Q04 1,000,000

F4Q04 1,000,000

Universeof

Projects

Universeof

Projects

Account: 1234Name: Chemistry

Account: 1234Name: Chemistry

Universeof

Users

Universeof

Users

Universeof

Machines

Universeof

Machines

Account: 1234Name: Chemistry

Account: 1234Name: Chemistry

Time Period Allocation Credit Limit

F1Q04 1,000,000 10,000

F2Q04 1,000,000 10,000

F3Q04 1,000,000 10,000

F4Q04 1,000,000 10,000

6

Dynamic ChargingDynamic ChargingDynamic ChargingDynamic Charging

Accounting Manager

(Gold)

0

2

1

4

3

5

6

Deposit

0 Deposits are made in Account

1 A Job is Submitted

2 A Quote is Requested

Quote

3 A Reservation is Made

Scheduler(Maui)

4 The Job is allowed to Start

Reservation

5 The Job Completes

Resource Manager(PBS, LL,

LSF)

6 The Reservation is Removed

ReservationCharge

and a Charge is Issued

7

Dynamic ChargingDynamic ChargingDynamic ChargingDynamic Charging

0 Make Deposits, etc. 1 Submit Job2 Obtain Quote3 Make Reservation4 Start Job5 Job Completes6 Issue Charge

Accounting Manager

(Gold)

0 2

1

4

3

5

6

Resource Manager(PBS, LL,

LSF)

8

Dynamic ChargingDynamic ChargingDynamic ChargingDynamic Charging

Resource Manager

(PBS, LL,LSF)

Allocation Manager

(Gold)

0

2

1

5

3

3

7

0 Make Deposits, etc. 1 Submit Job2 Locate Feasible Systems & Obtain Quote3 Stage Job4 Balance Check5 Make Reservation6 Start Job7 Job Completes8 Issue Charge

Meta-Scheduler

(Silver)Scheduler

(Maui)

68

4

9

Expiring AllocationsExpiring AllocationsExpiring AllocationsExpiring Allocations

Jan-99 Feb-99 Mar-99 Apr-99 May-99 Jun-99 Jul-99

Alloc6

Alloc5

Alloc4

Alloc3

Alloc2

Alloc1

.

10,000 NH

10,000 NH

10,000 NH

10,000 NH

10,000 NH

10,000 NH

Jan-99 Feb-99 Mar-99 Apr-99 May-99 Jun-99 Jul-99

Alloc1

.

60,000 NH

Jan-99 Feb-99 Mar-99 Apr-99 May-99 Jun-99 Jul-99

Alloc7

Alloc6

Alloc5

Alloc4

Alloc3

Alloc2

Alloc1

.

5,000 NH

10,000 NH

10,000 NH

10,000 NH

10,000 NH

10,000 NH

5,000 NH

Jan-99 Feb-99 Mar-99 Apr-99 May-99 Jun-99 Jul-99

Alloc6

Alloc5

Alloc4

Alloc3

Alloc2

Alloc1

.

10,000 NH

10,000 NH

10,000 NH

10,000 NH

10,000 NH

10,000 NH

10

Nested AccountsNested AccountsNested AccountsNested Accounts

UniversityUniversity

BiologyBiology

Actinides

Actinides

Chem 201Chem 201

PhysicsPhysics

Chem 101Chem 101

ChemistryChemistry

WorkshopWorkshop

252525

4040

5040

11

Web-based GUIWeb-based GUIWeb-based GUIWeb-based GUI

12

Other Gold FeaturesOther Gold FeaturesOther Gold FeaturesOther Gold Features

Strong Security

Historical JournalingTransparency Features

Dynamically Customizable

13

Other Gold FeaturesOther Gold FeaturesOther Gold FeaturesOther Gold Features

Flexible Charging

Credit and Debit ModelsGuaranteed Quotes

Powerful Querying

14

Enables Grid Computing

15

Distributed AccountingDistributed AccountingDistributed AccountingDistributed Accounting

Gold facilitates resource sharing across administrative domains (Computational Grids) by supporting distributed accounting and allocation while preserving local autonomyProblems: Security concerns (privacy, protection from fraud) Want to maintain autonomous control of own resources Need to know how much it is going to cost you (contract) You want your own local accounting record (a receipt)

Solutions: Traceback policy screening Distributed accounting Charge Quotes Strong authentication and encryption

16

Custom AccountingCustom AccountingCustom AccountingCustom Accounting

Sites can create or modify record types on the fly enabling them to meet their custom accounting needs. This capability turns Gold into a generalized information service.

[root@altix1 gold-1.0.a2.0]# gold -vgold> Object Create Name=Motorcyclegold> ObjectAttribute Create Object=Motorcycle Name=Name PrimaryKey=Truegold> ObjectAttribute Create Object=Motorcycle Name=Colorgold> ObjectAttribute Create Object=Motorcycle Name=HorsePower...gold> Motorcycle Create Name=Yamaha Color=Blue HorsePower=1200Color Name HorsePower----- ------ ---------- Blue Yamaha 1200Successfully created 1 Motorcycle...gold> Motorcycle Modify Name==Yamaha HorsePower=1500Color Name HorsePower----- ------ ----------Blue Yamaha 1500Successfully modified 1 Motorcycles

17

StatusStatusStatusStatus

Status: Currently in beta release (4Q04) General release targeting 2Q05

Download: http://www.emsl.pnl.gov/docs/mscf/gold

For more information: [email protected] http://www.emsl.pnl.gov/docs/mscf/gold http://sss.scl.ameslab.gov/home.shtml http://www.scidac.org/ScalableSystems